Automate conversion of GPFS policy outputs to parquet without Jupyter

Created a set of scripts to parse our standard GPFS policy outputs and save them as a parquet dataset without needing a Jupyter notebook. Iterated off of parquet-list-policy-data.ipynb.

Changes

• Simplified parsing algorithm
• Automatically extracts the "top-level directory" (tld). Can parse from /data/user, /data/user/home, /data/project, and /scratch.
  • Sets the tld as the index within each parquet file for faster aggregation later
• Variable output directory: defaults to log_dir/parquet but can be specified elsewhere
• Environment is controlled through a Singularity container (defaults to daskdev/dask:2024.8.0-py3.12) but is variable
  • If the container is not specified, the default is automatically downloaded and used
• Parallelization: Each part of a policy output is processed individually in an array task. Processing logs parts of 5 million lines apiece takes ~ 3 minutes
• Controlled via command line run-convert-to-parquet.sh

marked this merge request as draft

added 1 commit

• 1a9c7b10 - remove dask components for now

Compare with previous version

changed the description

marked this merge request as ready

Fixes #8 (closed)

added 1 commit

• 8dd5cb12 - remove SIF option for now. Only use the latest repo container

Compare with previous version

Let's have a directory structure like this:

|-- src
|   |-- legacy-notebooks
|   |   |-- *.ipynb (all the older notebooks)
|   |
|   |-- legacy-scripts
|   |   |-- *.sh (only the older scripts, not the new ones)
|   |
|   |-- (new directories/files here)
|
|-- (other things stay where they are here)

We can decide what to do with the policy definitions later in the policy directory. I'm content leaving them there for now.

mentioned in issue #10

added 1 commit

• 90b63609 - add instructions in README

Compare with previous version

added 1 commit

• e16641f8 - fix policy name

Compare with previous version

mentioned in merge request !10 (merged)

added 1 commit

• d29221d1 - move scripts into src directory

Compare with previous version

Addresses #10. Further reorganization will need to happen in a separate request

approved this merge request

mentioned in issue #8 (closed)

unapproved this merge request

approved this merge request

mentioned in commit f1b4bbcf

merged