Information Lifecycle Managment (ILM) via GPFS policy engine

The GPFS policy engine is well described in this white paper. A good presentation overview of the policy file is here. The relavent documentation is available from IBM.

This project focuses on scheduled execution of lifecyle policies to gather and process data about file system objects and issue actions against those objects based on policy.

Applying Policies

Applying a policy to filesets is done through the mmapplypolicy command at a base level. This repo contains wrapper scripts to call that command with a specified policy file on a given fileset where each wrapper has different levels of functionality meant for different groups of users in RC. All scripts are stored in src/run-policy

• run-mmpol: the main script that calls mmapplypolicy. Generally not invoked on its own
• submit-pol-job: general wrapper that sets up the Slurm job run-mmpol executes in. Admins can execute a policy run from this level using any policy file they have defined
• run-submit-pol-job.py: a Python wrapper for submit-pol-job meant specifically for running list policy jobs. This wrapper can be run by specific non-admins who have been given sudo permissions on this file alone. It can only run one of two policies: list-path-external and list-path-dirplus.

The production version of these scripts are kept in /data/rc/list-gpfs-dirs. Admins can run any one of these scripts from anywhere, but non-admins are only granted sudo privileges on the run-submit-pol-job.py file in that directory.

Note: The command is aligned to run on specific nodes by way of arguments to mmapplypolicy. The command is technically not run inside of the job reservation so the resource constraints are imperfect. The goal is to use the scheduler to ensure the policy run does not conflict with existing resource allocations on the cluster.

List Policies (non-admin)

A list policy can be executed using run-submit-pol-job.py using the following command:

run-submit-pol-job.py [-h] [-o OUTDIR] [-f LOG_PREFIX] [--with-dirs]
                      [-N NODES] [-c CORES] [-p PARTITION] [-t TIME]
                      [-m MEM_PER_CPU]
                      device

• outdir: specifies the directory the output log should be saved to. Defaults to /data/rc/gpfs-policy/data
• log-prefix: string to begin the name of the policy output with. Metadata containing the policy file name, slurm job ID, and time run will be appended to this prefix. Defaults to list-policy_<device>. See below for device
  • Note: this is currently non-functional
• --with-dirs: changes the policy file from list-path-external to list-path-dirplus. The only difference is that directories are included in the policy output.
• device: the fileset or directory to apply the policy to.

All other arguments are Slurm directives dictating resource requests. The default paramaters are as follows:

• nodes: 1
• cores: 16
• partition: amd-hdr100, medium
• time: 24:00:00
• mem-per-cpu: 8G

Run Any Policies (admins)

Any defined policy file can be run using the submit-pol-job by running the following:

sudo ./submit-pol-job [ -h ] [ -o | --outdir ] [ -f | --outfile ] [ -P | --policy ] 
                      [ -N | --nodes ] [ -c | --cores ] [ -p | --partition ] 
                      [ -t | --time ] [ -m | --mem ]
                      device

The only difference here is that a path to the policy file can be specified using -P or --policy. All other arguments are the same and have the same defaults

Output

The list-policy-external policy provides an efficient tool to gather file stat data into a URL-encoded ASCII text file. The output file can then be processed by down-stream to create reports on storage patterns and use. Make sure the output dir has sufficient space to hold the resulting file listing (It could be 100's of Gigabytes for a large collection of files.)

The slurm job output file will be local to the directory from which this command executed. It can be watched to observe progress in the generation of the file list. A listing of 100's of millions of files may take a couple of hours to generate and consume serveral hundred gigabytes for the output file.

List Policy Specific Outputs

The raw output file for list policies in outdir will be named list-<jobid>.list.gather-info.

The output file contains one line per file object stored under the device. No directories or non-file objects are included in this listing unless the list-path-dirplus policy is used. Each entry is a space-seperated set of file attributes selected by the SHOW command in the LIST rule. Entries are encoded according to RFC3986 URI percent encoding. This means all spaces and special characters will be encoded, making it easy to split lines into fields using the space separator.

Processing the output file

Split and compress

Policy outputs generated using list-path-external or list-path-dirplus can be split into multiple smaller log files to facilitate out-of-memory computation for very large filesets using tools such as dask. The policy output can be split and compressed using the src/split-info-file.sh script. See the following for usage:

./split-info-file.sh [ -h ] [ -l | --lines ] [ -o | --outdir ] 
                     [ -n | --ntasks ] [ -p | --partition] [ -t | --time ] [ -m | --mem ]  
                     log

• lines: the max number of lines to include in each split file. Defaults to 5000000
• outdir: directory to store the split files in. Defaults to ${log}.d in log's parent directory.
• log: path to the GPFS policy log. Can be either uncompressed or gzip compressed

All other options specify job resource parameters. Defaults are as follows:

• ntasks: 4
• partition: amd-hdr100
• time: 12:00:00
• mem: 16G

Split files will have the form ${outdir}/list-XXX.gz where XXX is an incrementing index. Files are automatically compressed.

Pre-parse output for Python

Processing GPFS log outputs is controlled by the run-convert-to-parquet.sh script and assumes the GPFS log has been split into a number of files of the form list-XXX.gz where XXX is an incrementing numeric index. This creates an array job where each task in the array reads the quoted text in one file, parses it into a dataframe, and exports it as a parquet file with the name list-XXX.parquet.

While the file is being parsed, the top-level-directory (tld) is extracted for each entry and added as a separate column to make common aggregations easier.

This script is written to parse the list-path-external policy format with quoted special characters.

Usage: ./run-convert-to-parquet.sh [ -h ] 
        [ -o | --outdir ] [ -n | --ntasks ] [ -p | --partition] 
        [ -t | --time ] [ -m | --mem ] 
        gpfs_logdir"

• outdir: Path to save parquet outputs. Defaults to ${gpfs_logdir}/parquet
• gpfs_logdir: Directory path containing the split log files as *.gz

All other options control the array job resources. The default resources can parse 5 million line files in approximately 3 minutes so should cover all common use cases.

Running reports

Disk usage by top level directies

A useful report is the top level directory (tld) report. This is akin to running a du -s * in a directory of interest, but much faster since there is no walking of directory trees. Only the list policy output file is used, reducing the operation to a parsing an summing of the data in the list policy output file.

Comparing directory similarity

Scheduling regular policy runs via cron

The policy run can be scheduled automatically with the cronwrapper script.

Simpley add append the above script and arguments to the crownwrapper in a crontab line.

For example to run it every morning at 4 am you would add:

0 4 * * * /path/to/cronwrapper submit-pol-job <outdir> <policy> <nodecount> <corespernode> <ram> <partition>