Commit f16856f3 authored by Chirag Chandrahas Shetty's avatar Chirag Chandrahas Shetty
Browse files

Getting slurm data using slurm2sql

parent eac8f3d5
astroid==2.3.3
astropy==4.0
attr==0.3.1
attrs==19.3.0
backcall==0.1.0
bleach==3.1.3
certifi==2019.11.28
chardet==3.0.4
confuse==1.0.0
cycler==0.10.0
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
htmlmin==0.1.12
idna==2.8
ipykernel==5.1.4
ipython==7.13.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
isort==4.3.21
jedi==0.16.0
Jinja2==2.11.1
joblib==0.14.1
jsonschema==3.2.0
jupyter-client==6.0.0
jupyter-core==4.6.1
kaggle==1.5.6
kiwisolver==1.1.0
lazy-object-proxy==1.4.3
llvmlite==0.31.0
MarkupSafe==1.1.1
matplotlib==3.2.0
mccabe==0.6.1
missingno==0.4.2
mistune==0.8.4
more-itertools==8.2.0
nbconvert==5.6.1
nbformat==5.0.4
networkx==2.4
notebook==6.0.3
numba==0.48.0
numpy==1.18.2
packaging==20.3
pandas==0.25.3
pandas-profiling==2.5.0
pandocfilters==1.4.2
parso==0.6.2
pexpect==4.8.0
phik==0.9.9
pickleshare==0.7.5
pluggy==0.13.1
prometheus-client==0.7.1
prompt-toolkit==3.0.3
ptyprocess==0.6.0
py==1.8.1
Pygments==2.6.1
pylint==2.4.4
pyparsing==2.4.6
pyrsistent==0.15.7
pytest==5.4.1
pytest-pylint==0.15.1
python-dateutil==2.8.1
python-slugify==4.0.0
pytz==2019.3
PyYAML==5.3
pyzmq==18.1.1
requests==2.22.0
scipy==1.4.1
seaborn==0.10.0
Send2Trash==1.5.0
six==1.14.0
slurm2sql==0.9.0
tangled-up-in-unicode==0.0.3
terminado==0.8.3
testpath==0.4.4
text-unidecode==1.3
tornado==6.0.4
tqdm==4.42.0
traitlets==4.3.3
urllib3==1.25.8
visions==0.2.2
wcwidth==0.1.8
webencodings==0.5.1
widgetsnbextension==3.5.1
wrapt==1.11.2
%% Cell type:code id: tags:
``` python
import sqlite3
import slurm2sql
import pandas as pd
```
%% Cell type:code id: tags:
``` python
db = sqlite3.connect('test.db')
slurm2sql.slurm2sql(db, ['-S', '2020-03-18', '-a'])
```
%% Output
0
%% Cell type:code id: tags:
``` python
# For example, you can then convert to a dataframe:
df1 = pd.read_sql('SELECT * FROM slurm', db)
```
%% Cell type:code id: tags:
``` python
df1.head(5)
```
%% Output
JobID ArrayJobID ArrayTaskID JobStep JobIDSlurm \
0 3319116 3319116 NaN None 3319116_[43-45,47%5]
1 3927198 3887451 30.0 None 3887451_30
2 3927198 3887451 30.0 batch 3887451_30.batch
3 3927198 3887451 30.0 extern 3887451_30.extern
4 3927199 3887451 31.0 None 3887451_31
JobName User Group Account State ... \
0 1mUD1MPa gpekmezi gpekmezi gpekmezi PENDING ...
1 100kCrC20MPa gpekmezi gpekmezi gpekmezi COMPLETED ...
2 batch gpekmezi COMPLETED ...
3 extern gpekmezi COMPLETED ...
4 100kCrC20MPa gpekmezi gpekmezi gpekmezi COMPLETED ...
MaxDiskReadNode MaxDiskReadTask MaxDiskWrite MaxDiskWriteNode \
0 NaN
1 NaN
2 c0088 0 1.222336e+10 c0088
3 c0088 0 0.000000e+00 c0088
4 NaN
MaxDiskWriteTask ReqGPUS Comment GPUMem GPUEff NGPU
0 NaN None None None None
1 NaN None None None None
2 0 NaN None None None None
3 0 NaN None None None None
4 NaN None None None None
[5 rows x 63 columns]
%% Cell type:code id: tags:
``` python
```
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment