Improve project documentation with instructions on running policy

Provide details on running a policy script with the provided
workflow framework.
Information on the list-policy-external policy and generated output.
Outline of future docs for preprocessing and reports.

README.md

@@ -4,19 +4,89 @@ The GPFS policy engine is well described in this [white paper](https://www.ibm.c
 A good presentation overview of the policy file is [here](https://www.spectrumscaleug.org/event/ssugdigital-spectrum-scale-ilm-policy-engine/).
 The relavent [documentation is available from IBM](https://www.ibm.com/docs/en/spectrum-scale/4.2.0?topic=guide-information-lifecycle-management-spectrum-scale).
 
-This project focuses on scheduled execution of lifecyle policies.
+This project focuses on scheduled execution of lifecyle policies to gather and process data about
+file system objects and issue actions against those objects based on policy.
 
-A policy is executed via the run_job
+## Running a policy
 
-submit-pol-job <outdir> <policy> <nodecount> <corespernode> <ram> <partition>
+A policy is executed in the context of a SLURM batch job reservation using the submit-pol-job script:
+```
+submit-pol-job <outdir> <policy> <nodecount> <corespernode> <ram> <partition> <time>
+```
+Where the positional arguments are:
 
-Where:
-
-outdir - the directory for the policy and job output, should be global to cluster
-policy - path to the policy file
+outdir - the directory for the output files, should be global to cluster (e.g. /scratch of the user running the job)
+policy - path to the GPFS policy to execute (e.g. in ./policy directory) 
 nodecount - number of nodes in the cluster that will run the policy
 corespernode - number of cores on each node to reserve
-partition - the partition used to submit the job
+ram - ram per core, can use "G" for gigabytes
+partition - the partition to submit the job
+time - the time in minutes to reserve for the job
+
+Note: the resource reservation is imperfect.  The job wrapper calls a script `run-mmpol.sh` which 
+is responsible for executing the `mmapplypolicy` command.  
+The command is aligned to run on specific nodes by way of arguments to 
+mmapplypolicy.  The command is technically not run inside of the job reservation so the resource
+constraints are imperfect.  The goal is to use the scheduler to ensure the policy run does not conflict
+with existing resource allocations on the cluster.
+
+## Running the policy "list-policy-external"
+
+The list-policy-external policy provides an efficient tool to gather file stat data into a URL-encoded
+ASCII text file.  The output file can then be processed by down-stream to create reports on storage
+patterns and use.
+
+An example invocation would be:
+```
+submit-pol-job /path/to/output/dir \
+     /absolute/path/policy/list-path-external \
+	 4 24 4G partition_name \
+	 /path/to/listed/dir \
+	 180
+```
+
+Some things to keep in mind:
+- the `submit-pol-job` script may need a `./` prefix if it is not in your path.
+- use absolute paths for all directory arguments to avoid potential confusion
+- make sure the output dir has sufficient space to hold the resulting file listing (It could be 100's of Gigabytes for a large collection of files.)
+
+The slurm job output file will be local to the directory from which this command executed.  It can be 
+watched to observe progress in the generation of the file list.  A listing of 100's of millions of files
+may take a couple of hours to generate and consume serveral hundred gigabytes for the output file.
+
+The output file in `/path/to/output/dir` is named as follows
+- a prefix of "list-${SLURM_JOBID}"
+- ".list" for the name of the policy rule type of "list"
+- a tag for the list name name defined in the policy file,  "list-gather" for `list-path-external` policy
+
+The output file contains one line per file object stored under the `/path/to/listed/dir`.  No directories
+or non-file objects are included in this listing.  Each entry is a space-seperated set of file attributes
+selected by the SHOW command in the LIST rule.  Entries are encoded according to RFC3986 URI percent 
+encoding.  This means all spaces and special characters will be encoded, making it easy to split lines
+into fields using the space separator.
+
+The ouput file is an unsorted list of files in uncompressed ASCII.  Further processing is desireble
+to use less space for storage and provide organized collections of data.
+
+## Processing the output file
+
+### Split and compress
+
+### Pre-parse output for Python
+
+## Running reports
+
+### Disk usage by top level directies
+
+A useful report is the top level directory (tld) report.  This is akin to running a `du -s *` in a
+directory of interest, but much faster since there is no walking of directory trees.  Only the list
+policy output file is used, reducing the operation to a parsing an summing of the data in the list
+policy output file.
+
+### Comparing directory similarity
+
+
+## Scheduling regular policy runs via cron
 
 The policy run can be scheduled automatically with the cronwrapper script.
 Simpley add append the above script and arguments to the crownwrapper in a crontab line.