Skip to content

ENH: Change log storage organization and correctly apply log prefix and metadata to file name

Matthew K Defenderfer requested to merge enh-improve-default-log-prefix into rel_v0.1.1-1

This is a continuing revision for !17 (merged). It makes some more general enhancements to how the policy files are created and stored as well as adding a dry-run option for testing to all 3 scripts.

Major changes:

  • Implements changes suggested in #17 (closed). run-mmpol now creates a single directory to store the raw policy output, the split gz files, and the parquet files plus anything else associated with the run. The directory name matches the name of the log file now with an appended .d.

Minor changes:

  • Full device path is used in the prefix, changing / to -. All metadata entries in the file/directory names are separated by _ for clarity
  • Added dry-run option to all 3 scripts for testing. Dry runs submit no jobs and make no changes to any directory structure.
  • Changed methods for finding the raw log once it's created and moving/renaming it
  • Removed default device from run-mmpol.sh. Now the device must be specified regardless of which script is run.

Syntax Changes:

  • Encased all called variables in "{}" for clarity in variable substitution.
  • Replaced ```` syntax with $() when performing command substitution
  • Changed datestring from %Y-%m-%d-%H:%M:%S to %Y%m%dT%H%M%S to minimize special character use in file names

Issues Fixed:

Merge request reports