GitLab

Split the cluster into gpu and non-gpu nodes with two partitions
Add QOS for each partation defined with:

sudo sacctmgr add qos maxcpu set maxtresperuser=cpu=4
sudo sacctmgr add qos maxgpucpu set maxtresperuser=cpu=2

Update the gpu config to have a type flag and cpu affinity
This feature doesn't work unless select/cons_res is active.

The Type parameter for GPUs in the gres.conf and NodeName lines in
slurm.conf don't work unless select/cons_res is active.  In other
works, they don't work if select/linear (nodes are alocated as hole
units).  The --gres=gpu flag works for linear but gpu allocation
fails completely for type-based gpus --gres=gpu:p100:1.

slurmctld logs a warning that pgid is a slow method and recommends
linuxproc.

The dedicate state dir needs to be created owned by slurm exclusively
so that the slurmctld don't complain about security error at start.

Note gpu type and cpu affinities doen't work with this basic slurm
config.  If set, slurm will report that the resources are unavailable.

This is the default slurm.conf from the OpenHPC repo slurm packages
with localization of cluster name and nodename list.