auto-download function

[rickvanhoof]

First of all, thanks for the nice work you have done with neuropythy!

I am having an issue with the automatic downloads of the HCP dataset. The auto_download function you mention in your tutorial https://nben.net/HCP-and-Neuropythy/#auto-download was removed at some point.

I set up neuropythy as you instructed in your wiki with a valid HCP key/pw. When I call 'ny.hcp.download(100307)' I get the following error:

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/hcp/core.py", line 315, in download
return ny.data['hcp'].download(sid)
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/datasets/hcp.py", line 283, in download
sub = self.subjects[sid]
File "/usr/local/lib/python3.8/dist-packages/pimms-0.3.18-py3.8.egg/pimms/util.py", line 789, in getitem
File "/usr/local/lib/python3.8/dist-packages/pimms-0.3.18-py3.8.egg/pimms/util.py", line 785, in _examine_val
File "/usr/local/lib/python3.8/dist-packages/pimms-0.3.18-py3.8.egg/pimms/util.py", line 689, in curried_f
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/datasets/hcp.py", line 273, in _add_retino
else: return _subjects[sid]
File "/usr/local/lib/python3.8/dist-packages/pimms-0.3.18-py3.8.egg/pimms/util.py", line 789, in getitem
File "/usr/local/lib/python3.8/dist-packages/pimms-0.3.18-py3.8.egg/pimms/util.py", line 785, in _examine_val
File "/usr/local/lib/python3.8/dist-packages/pimms-0.3.18-py3.8.egg/pimms/util.py", line 689, in curried_f
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/datasets/hcp.py", line 218, in _load_subject
return subject(ppath, default_alignment=df, name=str(sid))
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/hcp/core.py", line 262, in subject
sub = subject_from_filemap(fmap, name=name, check_path=check_path, meta_data=meta_data,
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/hcp/core.py", line 124, in subject_from_filemap
if check_path and not is_hcp_subject_path(fmap.pseudo_paths[None]):
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/hcp/files.py", line 87, in is_hcp_subject_path
if is_pseudo_path(path): return all(path.find(k) is not None for k in needed)
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/hcp/files.py", line 87, in
if is_pseudo_path(path): return all(path.find(k) is not None for k in needed)
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/util/filemap.py", line 526, in find
return pmod.find(*args)
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/util/filemap.py", line 236, in find
return self.base_find(rpath)
File "/usr/local/lib/python3.8/dist-packages/neuropythy-0.12.0-py3.8.egg/neuropythy/util/filemap.py", line 310, in base_find
if self.s3fs.exists(fpath): return rpath
File "/usr/local/lib/python3.8/dist-packages/fsspec-0.9.0-py3.8.egg/fsspec/asyn.py", line 72, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/fsspec-0.9.0-py3.8.egg/fsspec/asyn.py", line 53, in sync
raise result[0]
File "/usr/local/lib/python3.8/dist-packages/fsspec-0.9.0-py3.8.egg/fsspec/asyn.py", line 20, in _runner
result[0] = await coro
File "/usr/local/lib/python3.8/dist-packages/s3fs-0.6.0-py3.8.egg/s3fs/core.py", line 724, in _exists
await self._info(path, bucket, key, version_id=version_id)
File "/usr/local/lib/python3.8/dist-packages/s3fs-0.6.0-py3.8.egg/s3fs/core.py", line 910, in _info
out = await self._call_s3(
File "/usr/local/lib/python3.8/dist-packages/s3fs-0.6.0-py3.8.egg/s3fs/core.py", line 228, in _call_s3
await self._connect()
File "/usr/local/lib/python3.8/dist-packages/s3fs-0.6.0-py3.8.egg/s3fs/core.py", line 370, in _connect
self._s3 = await s3creator.aenter()
File "/usr/local/lib/python3.8/dist-packages/aiobotocore-1.2.2-py3.8.egg/aiobotocore/session.py", line 20, in aenter
File "/usr/local/lib/python3.8/dist-packages/aiobotocore-1.2.2-py3.8.egg/aiobotocore/session.py", line 104, in _create_client
File "/usr/local/lib/python3.8/dist-packages/aiobotocore-1.2.2-py3.8.egg/aiobotocore/client.py", line 32, in create_client
File "/usr/local/lib/python3.8/dist-packages/aiobotocore-1.2.2-py3.8.egg/aiobotocore/client.py", line 87, in _get_client_args
File "/usr/local/lib/python3.8/dist-packages/aiobotocore-1.2.2-py3.8.egg/aiobotocore/args.py", line 53, in get_client_args
File "/usr/local/lib/python3.8/dist-packages/aiobotocore-1.2.2-py3.8.egg/aiobotocore/endpoint.py", line 307, in create_endpoint
TypeError: init() got an unexpected keyword argument 'verify_ssl'

I am running Ubuntu 18.04.5 with Python 3.8.9. Have you encountered this issue before?

[noahbenson]

The auto_download function no longer needs to be called (that tutorial is a few years old and somewhat out of date). Instead, you configure with the neuropythy.config structure, ideally using a config file in ~/.npythyrc (see this documentation).

I haven't seen this error before, nor do I get it when I run the download function using the current version of neuropythy on my desktop. Offhand, my guess is that it's an issue related to the s3fs library, which was just recently updated to version 0.6.0, which you are using. I have version 0.4.0 of s3fs installed, and I don't even have the aiobotocore library installed, which apparently is being used by s3fs versoin 0.6.0. So, initial guess: you have a new version of s3fs installed that includes some bug in the aiobotocore dependency (which is a new dependency in a recent version update of s3fs).

Looking at the source code for aiobotocore, Line 307, which generates the error, it does indeed have a verify_ssl option that is not supported by the current up-to-date version of the aoihttp library (here). Nor does it look to me like that option was supported in any recent version of aiohttp going back to at least 2019. So probably the aiobotocore version that was recently pushed has a bug in its dependencies such that's out of sync with the current version of the aiohttp library. In any case, this error seems to be occurring in a library that is used by a library that is used by a library that is used by neuropythy, so this is all I can do to help. You'll need to file a bug report with the aiobotocore library (I would just link to this issue so that they can see this explanation). You'll probably also want to provide them with the version of aiohttp you have installed.

For a quick fix in the meantime, I would try using the Dockerized version of Neuropythy---this will avoid dependency issues altogether---or you could try downgrading your s3fs version to 0.4.0.

[noahbenson]

Actually, I just went ahead and submitted the issue (link)---if it ends up being more complicated than my initial guess, however, they will likely need you to provide more details about your environment and the failure.

[rickvanhoof]

Thanks for the quick solution, reverting to 0.4.0 fixed the issue.