README.md 2.94 KB
Newer Older
Ryan Randles Jones's avatar
Ryan Randles Jones committed
1
A yml and a set of instructions to build a functioning environment for the Research Computing
Saisri Vegesna's avatar
Saisri Vegesna committed
2

Ryan Randles Jones's avatar
Ryan Randles Jones committed
3
clone this repo and update with the job composer
Saisri Vegesna's avatar
Saisri Vegesna committed
4
5
Copy and paste the following job script into a job composer job on rc.uab.edu
#!/bin/bash
Ryan Randles Jones's avatar
Ryan Randles Jones committed
6
#SBATCH --partition=express
Saisri Vegesna's avatar
Saisri Vegesna committed
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#SBATCH --mem-per-cpu=4000
module load cuda10.0/toolkit
module load Anaconda3

FOLDER=/data/user/$USER/slurm-ds
URL=https://gitlab.rc.uab.edu/rc-data-science/createandparsesacct.git
if [ ! -d "$FOLDER" ] ; then
    git clone "$URL" "$FOLDER"
conda env create -f /data/user/$USER/slurm-ds/environment-slurm-ds.yml
else
    cd $FOLDER
    git pull "$URL"
    conda env update -f /data/user/$USER/slurm-ds/environment-slurm-ds.yml
fi
21
22
<<<<<<< HEAD
=======
Saisri Vegesna's avatar
Saisri Vegesna committed
23
```
Ryan Randles Jones's avatar
Ryan Randles Jones committed
24
25

# Configuring the environment
26
>>>>>>> ab6ec8536fc1daf2070f8550943b3f28acd0b56f
Saisri Vegesna's avatar
Saisri Vegesna committed
27

Ryan Randles Jones's avatar
Ryan Randles Jones committed
28
Configuring the environment
Saisri Vegesna's avatar
Saisri Vegesna committed
29
30
31
32
After the environment is created, you can start up an interactive Jupyter notebook session through rc.uab.edu to check if the environment works.
Under environment setup, specify
# Load required modules
module load cuda10.0/toolkit
Ryan Randles Jones's avatar
Ryan Randles Jones committed
33
module load Anaconda3/2019.10
34
35
<<<<<<< HEAD
=======
Saisri Vegesna's avatar
Saisri Vegesna committed
36
37
```

38
>>>>>>> ab6ec8536fc1daf2070f8550943b3f28acd0b56f
Saisri Vegesna's avatar
Saisri Vegesna committed
39
40
Under Extra jupyter arguments, specify
--notebook-dir=/data/user/$USER/slurm-ds
41
<<<<<<< HEAD
Ryan Randles Jones's avatar
Ryan Randles Jones committed
42
43
44
45
46
47
48
49
50
51
52
53
For partition, set partition to
express
for time up to 2 hours. Also make sure to specify the number of hours field to match. For up to 12 hours, the
short
partition can be used.
After the Jupyter notebook is started, click on the blue "Connect to Jupyter" button.
Once the Jupyter session is active, select the slurm-2sql notebook. Then change the kernel, via Kernel->Change kernel->Python [conda env:.conda-slurm-ds]
Verify the environment loaded correctly by running the first cell of the slurm-2sql notebook (with the library imports)

Creating a text version of sacct output
If we have to create a database from sacct
directoryToUse="/data/user/$USER/group"
54
55
sacct -P -u $USER --starttime=2019-01-01 --format user,start,jobid,jobname,state,partition,maxrss,reqmem,reqcpus,node,nnodes,elapsed >> "$directoryToUse"group.txt
=======
Saisri Vegesna's avatar
Saisri Vegesna committed
56
57
```

Ryan Randles Jones's avatar
Ryan Randles Jones committed
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
For partition, set partition to 
```
express
```
for time up to 2 hours. Also make sure to specify the number of hours field to match. For up to 12 hours, the  
```
short
```
partition can be used.

After the Jupyter notebook is started, click on the blue "Connect to Jupyter" button.

Once the Jupyter session is active, select the `slurm-2sql` notebook. Then change the kernel, via `Kernel->Change kernel->Python [conda env:.conda-slurm-ds]`

Verify the environment loaded correctly by running the first cell of the `slurm-2sql` notebook (with the library imports)
Saisri Vegesna's avatar
Saisri Vegesna committed
73

Ryan Randles Jones's avatar
Ryan Randles Jones committed
74
75
76
77
78
79
# Creating a text version of sacct output

If we have to create a database from sacct
```
directoryToUse="/data/user/$USER/group"
sacct -P -u $USER --starttime=2019-01-01 --format user,start,jobid,jobname,state,partition,maxrss,reqmem,reqcpus,node,nnodes,elapsed >> "$directoryToUse"group.txt
Saisri Vegesna's avatar
Saisri Vegesna committed
80
```
81
>>>>>>> ab6ec8536fc1daf2070f8550943b3f28acd0b56f