Sockeye-3.0.4: AttributeError: module 'distutils' has no attribute 'version'

[SamuelLarkin]

Hi,
I'm trying out the new Sockeye-3.0.4 and I'm getting the following error

Traceback (most recent call last):
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.0.4/bin/sockeye-train", line 8, in <module>
    sys.exit(main())
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.0.4/lib/python3.8/site-packages/sockeye/train_pt.py", line 848, in main
    train(args)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.0.4/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 3
    return f(*args, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.0.4/lib/python3.8/site-packages/sockeye/train_pt.py", line 1039, in train
    trainer = training_pt.PyTorchEarlyStoppingTrainer(
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.0.4/lib/python3.8/site-packages/sockeye/training_pt.py", line 174, in __init__
    self._tflogger = TensorboardLogger(logdir=os.path.join(self.config.output_dir, C.TENSORBOARD_NAME))
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.0.4/lib/python3.8/site-packages/sockeye/training_pt.py", line 779, in __init__
    from torch.utils.tensorboard import SummaryWriter
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.0.4/lib/python3.8/site-packages/torch/utils/tensorboard/__init__.py", line 4, in <module>
    LooseVersion = distutils.version.LooseVersion
AttributeError: module 'distutils' has no attribute 'version'

distutils is from setuptools and my installed versionis setuptools=59.6.0=py38h578d9bd_0

I guess my first question would be which version of setuptools should I install?

I've installed Sockeye in a conda environment with the following export (conda env export)

name: sockeye-3.0.4
channels:
  - pytorch
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_llvm
  - absl-py=1.0.0=pyhd8ed1ab_0
  - aiohttp=3.8.1=py38h497a2fe_0
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - alabaster=0.7.12=py_0
  - alsa-lib=1.2.3=h516909a_0
  - async-timeout=4.0.1=pyhd8ed1ab_0
  - attrs=21.2.0=pyhd8ed1ab_0
  - babel=2.9.1=pyh44b312d_0
  - blas=1.0=mkl
  - blinker=1.4=py_1
  - brotli=1.0.9=h7f98852_6
  - brotli-bin=1.0.9=h7f98852_6
  - brotlipy=0.7.0=py38h497a2fe_1003
  - c-ares=1.18.1=h7f98852_0
  - ca-certificates=2021.10.8=ha878542_0
  - cachetools=4.2.4=pyhd8ed1ab_0
  - certifi=2021.10.8=py38h578d9bd_1
  - cffi=1.15.0=py38h3931269_0
  - charset-normalizer=2.0.9=pyhd8ed1ab_0
  - click=8.0.3=py38h578d9bd_1
  - colorama=0.4.4=pyh9f0ad1d_0
  - conda=4.11.0=py38h578d9bd_0
  - conda-package-handling=1.7.3=py38h497a2fe_1
  - conda-tree=1.0.4=pyhd8ed1ab_0
  - cryptography=36.0.0=py38h3e25421_0
  - cudatoolkit=10.2.89=h8f6ccaa_9
  - cudnn=8.2.1.32=h2c0ae14_0
  - cxxfilt=0.3.0=py38h709712a_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - dataclasses=0.8=pyhc8e2a94_3
  - dbus=1.13.6=h48d8840_2
  - docutils=0.16=py38h578d9bd_3
  - expat=2.4.1=h9c3ff4c_0
  - fontconfig=2.13.1=hba837de_1005
  - fonttools=4.28.3=py38h497a2fe_0
  - freetype=2.10.4=h0708190_1
  - frozenlist=1.2.0=py38h497a2fe_1
  - gettext=0.19.8.1=h73d1719_1008
  - glib=2.70.2=h780b84a_0
  - glib-tools=2.70.2=h780b84a_0
  - google-auth=2.3.3=pyh6c4a22f_0
  - google-auth-oauthlib=0.4.6=pyhd8ed1ab_0
  - grpcio=1.42.0=py38hdd6454d_0
  - gst-plugins-base=1.18.5=hf529b03_2
  - gstreamer=1.18.5=h9f60fe5_2
  - icu=68.2=h9c3ff4c_0
  - idna=3.1=pyhd3deb0d_0
  - imagesize=1.3.0=pyhd8ed1ab_0
  - importlib-metadata=4.8.2=py38h578d9bd_0
  - iniconfig=1.1.1=pyh9f0ad1d_0
  - jbig=2.1=h7f98852_2003
  - jinja2=3.0.3=pyhd8ed1ab_0
  - jpeg=9d=h36c2ea0_0
  - kiwisolver=1.3.2=py38h1fd1430_1
  - krb5=1.19.2=hcc1bbae_3
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=3.0=h9c3ff4c_0
  - libbrotlicommon=1.0.9=h7f98852_6
  - libbrotlidec=1.0.9=h7f98852_6
  - libbrotlienc=1.0.9=h7f98852_6
  - libclang=11.1.0=default_ha53f305_1
  - libdeflate=1.8=h7f98852_0
  - libedit=3.1.20191231=he28a2e2_2
  - libevent=2.1.10=h9b69904_4
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=11.2.0=h1d223b6_11
  - libglib=2.70.2=h174f98d_0
  - libiconv=1.16=h516909a_0
  - libllvm11=11.1.0=hf817b99_2
  - libnsl=2.0.0=h7f98852_0
  - libogg=1.3.4=h7f98852_1
  - libopus=1.3.1=h7f98852_1
  - libpng=1.6.37=h21135ba_2
  - libpq=13.5=hd57d9b9_1
  - libprotobuf=3.19.1=h780b84a_0
  - libstdcxx-ng=11.2.0=he4da1e4_11
  - libtiff=4.3.0=h6f004c6_2
  - libuuid=2.32.1=h7f98852_1000
  - libuv=1.42.0=h7f98852_0
  - libvorbis=1.3.7=h9c3ff4c_0
  - libwebp-base=1.2.1=h7f98852_0
  - libxcb=1.13=h7f98852_1004
  - libxkbcommon=1.0.3=he3ba5ed_0
  - libxml2=2.9.12=h72842e0_0
  - libzlib=1.2.11=h36c2ea0_1013
  - llvm-openmp=12.0.1=h4bd325d_1
  - lz4-c=1.9.3=h9c3ff4c_1
  - markdown=3.3.6=pyhd8ed1ab_0
  - markupsafe=2.0.1=py38h497a2fe_1
  - matplotlib=3.5.1=py38h578d9bd_0
  - matplotlib-base=3.5.1=py38hf4fb855_0
  - mkl=2021.4.0=h8d4b97c_729
  - mkl-service=2.4.0=py38h95df7f1_0
  - mkl_fft=1.3.1=py38h8666266_1
  - mkl_random=1.2.2=py38h1abd341_0
  - more-itertools=8.12.0=pyhd8ed1ab_0
  - multidict=5.2.0=py38h497a2fe_1
  - munkres=1.1.4=pyh9f0ad1d_0
  - mysql-common=8.0.27=ha770c72_2
  - mysql-libs=8.0.27=hfa10184_2
  - nccl=2.11.4.1=h1a5f58c_0
  - ncurses=6.2=h58526e2_4
  - networkx=2.5.1=pyhd8ed1ab_0
  - nspr=4.32=h9c3ff4c_1
  - nss=3.73=hb5efdd6_0
  - numpy=1.21.2=py38h20f2e39_0
  - numpy-base=1.21.2=py38h79a1101_0
  - nvidia-apex=0.1=py38h3af061c_4
  - oauthlib=3.1.1=pyhd8ed1ab_0
  - olefile=0.46=pyh9f0ad1d_1
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1l=h7f98852_0
  - packaging=21.3=pyhd8ed1ab_0
  - pcre=8.45=h9c3ff4c_0
  - pillow=8.4.0=py38h8e6f84c_0
  - pip=21.3.1=pyhd8ed1ab_0
  - pluggy=1.0.0=py38h578d9bd_2
  - protobuf=3.19.1=py38h709712a_1
  - pthread-stubs=0.4=h36c2ea0_1001
  - py=1.11.0=pyh6c4a22f_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycosat=0.6.3=py38h497a2fe_1009
  - pycparser=2.21=pyhd8ed1ab_0
  - pygments=2.10.0=pyhd8ed1ab_0
  - pyjwt=2.3.0=pyhd8ed1ab_1
  - pyopenssl=21.0.0=pyhd8ed1ab_0
  - pyparsing=3.0.6=pyhd8ed1ab_0
  - pyqt=5.12.3=py38h578d9bd_8
  - pyqt-impl=5.12.3=py38h0ffb2e6_8
  - pyqt5-sip=4.19.18=py38h709712a_8
  - pyqtchart=5.12=py38h7400c14_8
  - pyqtwebengine=5.12.1=py38h7400c14_8
  - pysocks=1.7.1=py38h578d9bd_4
  - pytest=6.2.5=py38h578d9bd_1
  - python=3.8.12=hb7a2778_2_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.8=2_cp38
  - pytorch=1.10.0=py3.8_cuda10.2_cudnn7.6.5_0
  - pytorch-mutex=1.0=cuda
  - pytz=2021.3=pyhd8ed1ab_0
  - pyu2f=0.1.5=pyhd8ed1ab_0
  - pyyaml=6.0=py38h497a2fe_3
  - qt=5.12.9=hda022c4_4
  - readline=8.1=h46c0cb4_0
  - requests=2.26.0=pyhd8ed1ab_1
  - requests-oauthlib=1.3.0=pyh9f0ad1d_0
  - rsa=4.8=pyhd8ed1ab_0
  - ruamel_yaml=0.15.80=py38h497a2fe_1006
  - setuptools=59.6.0=py38h578d9bd_0
  - six=1.16.0=pyh6c4a22f_0
  - snowballstemmer=2.2.0=pyhd8ed1ab_0
  - sphinx=4.3.1=pyh6c4a22f_0
  - sphinxcontrib-applehelp=1.0.2=py_0
  - sphinxcontrib-devhelp=1.0.2=py_0
  - sphinxcontrib-htmlhelp=2.0.0=pyhd8ed1ab_0
  - sphinxcontrib-jsmath=1.0.1=py_0
  - sphinxcontrib-qthelp=1.0.3=py_0
  - sphinxcontrib-serializinghtml=1.1.5=pyhd8ed1ab_1
  - sqlite=3.37.0=h9cd32fc_0
  - tbb=2021.4.0=h4bd325d_1
  - tensorboard=2.7.0=pyhd8ed1ab_0
  - tensorboard-data-server=0.6.0=py38h3e25421_1
  - tensorboard-plugin-wit=1.8.0=pyh44b312d_0
  - tk=8.6.11=h27826a3_1
  - toml=0.10.2=pyhd8ed1ab_0
  - tornado=6.1=py38h497a2fe_2
  - tqdm=4.62.3=pyhd8ed1ab_0
  - typing-extensions=4.0.1=hd8ed1ab_0
  - typing_extensions=4.0.1=pyha770c72_0
  - unicodedata2=13.0.0.post2=py38h497a2fe_4
  - urllib3=1.26.7=pyhd8ed1ab_0
  - werkzeug=2.0.1=pyhd8ed1ab_0
  - wheel=0.37.0=pyhd8ed1ab_1
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xz=5.2.5=h516909a_1
  - yaml=0.2.5=h516909a_0
  - yarl=1.7.2=py38h497a2fe_1
  - zipp=3.6.0=pyhd8ed1ab_0
  - zlib=1.2.11=h36c2ea0_1013
  - zstd=1.5.0=ha95c52a_0
  - pip:
    - backcall==0.2.0+computecanada
    - decorator==5.1.0+computecanada
    - ipython==7.30.1
    - jedi==0.18.1+computecanada
    - matplotlib-inline==0.1.3+computecanada
    - mock==4.0.3+computecanada
    - parso==0.8.3
    - pexpect==4.8.0+computecanada
    - pickleshare==0.7.5+computecanada
    - portalocker==2.3.2
    - prompt-toolkit==3.0.24
    - ptyprocess==0.7.0+computecanada
    - pudb==2014.1
    - sacrebleu==1.4.14
    - sentencepiece==0.1.96
    - sockeye==3.0.4
    - subword-nmt==0.3.8
    - tabulate==0.8.9+computecanada
    - tokenizers==0.10.3
    - traitlets==5.1.1
    - urwid==2.1.2
    - wcwidth==0.2.5+computecanada
prefix: $HOME/miniconda3/envs/sockeye-3.0.4

[SamuelLarkin]

Looks like it is related to:

• AttributeError: module 'distutils' has no attribute 'version' : with setuptools 59.6.0 pytorch/pytorch#69894
• Use packaging.version instead of distutils.version pytorch/pytorch#69823

[SamuelLarkin]

Solution for now:
I downgraded setuptools to setuptools=58.0.4=py38h06a4308_0 doing conda install -c anaconda setuptools.

[tdomhan]

Thanks for reporting your workaround and likely related pytorch issues!