error from qstat in jlab-ch

[klindsay28]

When I try to run jlab-ch, I get the following (truncated) output:

Launching notebook server
  queue = share
  account = P93300670
  nodes = 1
  ncpus = 1
  memory = 8GB
  walltime = 06:00:00
  port = 8888

submitted job: 7186475.chadmin1.ib0.cheyenne.ucar to queue share
waiting for job to runqstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
qstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
..qstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
qstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
..qstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar

It looks like something is going wrong with extracting the job id from the output generated by qsub. It is quite possible that the format of the output from qsub has changed with the updates to cheyenne. That said, the bash for extracting the job id is sjob=${s%.*}. I don't know bash well enough to know what this is doing, or why it appears to no longer work.

[matt-long]

I suspect that the string returned from PBS has changed following machine downtime. It now returns something like

$JOBID.chadmin1.ib0.cheyenne.ucar

and sjob=${s%.*} is everything before the last ".". I changed this to sjob=${s%%.*} and it works.

However, I am not getting another error from Jupyter

WARNING: The conda.compat module is deprecated and will be removed in a future release.
[I 07:05:11.183 LabApp] [nb_conda_kernels] enabled, 5 kernels found
Traceback (most recent call last):
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/glade/work/mclong/miniconda3/bin/jupyter-lab", line 11, in <module>
    sys.exit(main())
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "</glade/work/mclong/miniconda3/lib/python3.7/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1628, in initialize
    self.init_configurables()
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1319, in init_configurables
    connection_dir=self.runtime_dir,
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/glade/work/mclong/miniconda3/lib/python3.7/os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/glade/work/mclong/miniconda3/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/run/user/4267'

I haven't used this script in a while, since JupyterHub. jlab-dav I use all the time without issue.

[klindsay28]

I've generally not been using jupyterhub.ucar.edu because the git hash based provenance tracking that I'm using in my notebooks doesn't work there (even though it works fine with jlab-ch and the same conda environment). I usually use jlab-dav. I tried using jlab-ch this morning because I've been getting multiple socket timeout errors with slurm this morning that cause dask to fail within jlab-dav.

…

On Wed, Jul 17, 2019 at 7:14 AM Matthew Long ***@***.***> wrote: I suspect that the string returned from PBS has changed following machine downtime. It now returns something like $JOBID.chadmin1.ib0.cheyenne.ucar and sjob=${s%.*} is everything before the last ".". I changed this to sjob=${s%%.*} and it works. However, I am not getting another error from Jupyter WARNING: The conda.compat module is deprecated and will be removed in a future release. [I 07:05:11.183 LabApp] [nb_conda_kernels] enabled, 5 kernels found Traceback (most recent call last): File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 528, in get value = obj._trait_values[self.name] KeyError: 'runtime_dir' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/glade/work/mclong/miniconda3/bin/jupyter-lab", line 11, in <module> sys.exit(main()) File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/jupyter_core/application.py", line 266, in launch_instance return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs) File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/config/application.py", line 657, in launch_instance app.initialize(argv) File "</glade/work/mclong/miniconda3/lib/python3.7/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error return method(app, *args, **kwargs) File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1628, in initialize self.init_configurables() File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1319, in init_configurables connection_dir=self.runtime_dir, File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 556, in __get__ return self.get(obj, cls) File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get value = self._validate(obj, dynamic_default()) File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default ensure_dir_exists(rd, mode=0o700) File "/glade/work/mclong/miniconda3/lib/python3.7/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists os.makedirs(path, mode=mode) File "/glade/work/mclong/miniconda3/lib/python3.7/os.py", line 211, in makedirs makedirs(head, exist_ok=exist_ok) File "/glade/work/mclong/miniconda3/lib/python3.7/os.py", line 221, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/run/user/4267' I haven't used this script in a while, since JupyterHub. jlab-dav I use all the time without issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#20>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADWZPO6TFDFZ6BF4L66U6UDP74LKJANCNFSM4IEQCL3Q> .

[matt-long]

This looks similar to the issue I am encountering
jupyter/docker-stacks#718

[klindsay28]

After replacing sjob=${s%.*} with sjob=${s%%.*}, I'm getting the same error that you're getting.

[matt-long]

Ok, I have a fix. I added

export JUPYTER_RUNTIME_DIR=${WORKDIR}

to the job here-document. This is documented here
https://jupyter.readthedocs.io/en/latest/projects/jupyter-directories.html#runtime-files