README.md 1.92 KB
Newer Older
Ryan Randles Jones's avatar
Ryan Randles Jones committed
1
2
3
A yml and a set of instructions to build a functioning environment for the Research Computing 

# clone this repo and update with the job composer
Saisri Vegesna's avatar
Saisri Vegesna committed
4
5

Copy and paste the following job script into a job composer job on rc.uab.edu
Ryan Randles Jones's avatar
Ryan Randles Jones committed
6
7

```
Saisri Vegesna's avatar
Saisri Vegesna committed
8
#!/bin/bash
Ryan Randles Jones's avatar
Ryan Randles Jones committed
9
#SBATCH --partition=express
Saisri Vegesna's avatar
Saisri Vegesna committed
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#SBATCH --mem-per-cpu=4000
module load cuda10.0/toolkit
module load Anaconda3

FOLDER=/data/user/$USER/slurm-ds
URL=https://gitlab.rc.uab.edu/rc-data-science/createandparsesacct.git
if [ ! -d "$FOLDER" ] ; then
    git clone "$URL" "$FOLDER"
conda env create -f /data/user/$USER/slurm-ds/environment-slurm-ds.yml
else
    cd $FOLDER
    git pull "$URL"
    conda env update -f /data/user/$USER/slurm-ds/environment-slurm-ds.yml
fi
Saisri Vegesna's avatar
Saisri Vegesna committed
24
```
Ryan Randles Jones's avatar
Ryan Randles Jones committed
25
26

# Configuring the environment
Saisri Vegesna's avatar
Saisri Vegesna committed
27
28

After the environment is created, you can start up an interactive Jupyter notebook session through rc.uab.edu to check if the environment works.
Ryan Randles Jones's avatar
Ryan Randles Jones committed
29

Saisri Vegesna's avatar
Saisri Vegesna committed
30
Under environment setup, specify
Ryan Randles Jones's avatar
Ryan Randles Jones committed
31
32

```
Saisri Vegesna's avatar
Saisri Vegesna committed
33
34
# Load required modules
module load cuda10.0/toolkit
Ryan Randles Jones's avatar
Ryan Randles Jones committed
35
module load Anaconda3/2019.10
Saisri Vegesna's avatar
Saisri Vegesna committed
36
37
38
```

Under Extra jupyter arguments, specify
Ryan Randles Jones's avatar
Ryan Randles Jones committed
39

Ryan Randles Jones's avatar
Ryan Randles Jones committed
40
41
```
--notebook-dir=/data/user/$USER/slurm-ds
Saisri Vegesna's avatar
Saisri Vegesna committed
42
43
```

Ryan Randles Jones's avatar
Ryan Randles Jones committed
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
For partition, set partition to 
```
express
```
for time up to 2 hours. Also make sure to specify the number of hours field to match. For up to 12 hours, the  
```
short
```
partition can be used.

After the Jupyter notebook is started, click on the blue "Connect to Jupyter" button.

Once the Jupyter session is active, select the `slurm-2sql` notebook. Then change the kernel, via `Kernel->Change kernel->Python [conda env:.conda-slurm-ds]`

Verify the environment loaded correctly by running the first cell of the `slurm-2sql` notebook (with the library imports)
Saisri Vegesna's avatar
Saisri Vegesna committed
59

Ryan Randles Jones's avatar
Ryan Randles Jones committed
60
61
62
63
64
65
# Creating a text version of sacct output

If we have to create a database from sacct
```
directoryToUse="/data/user/$USER/group"
sacct -P -u $USER --starttime=2019-01-01 --format user,start,jobid,jobname,state,partition,maxrss,reqmem,reqcpus,node,nnodes,elapsed >> "$directoryToUse"group.txt
Saisri Vegesna's avatar
Saisri Vegesna committed
66
```