ValueError: tuple is not allowed for map key

[KabitaBaral1]

Hi I am using pyscenic via Jupyter Notebook and when I run the following code for grn:

!pyscenic grn {f_loom_path_scenic} {f_tfs} -o adj.csv --num_workers 20

I get the following error:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}

2020-04-09 14:18:38,967 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2020-04-09 14:18:47,333 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
preparing dask client
parsing input
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/arboreto/algo.py:214: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
expression_matrix = expression_data.as_matrix()
creating dask graph
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 108, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
distributed.core - ERROR - tuple is not allowed for map key
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/core.py", line 347, in handle_comm
msg = yield comm.read()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/tcp.py", line 218, in read
frames, deserialize=self.deserialize, deserializers=deserializers
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py", line 85, in from_frames
res = _from_frames()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py", line 71, in _from_frames
frames, deserialize=deserialize, deserializers=deserializers
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 108, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 108, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
not shutting down client, client was created externally
finished
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/bin/pyscenic", line 8, in
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 420, in main
args.func(args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 72, in find_adjacencies_command
network = method(expression_data=ex_mtx, tf_names=tf_names, verbose=True, client_or_address=client, seed=args.seed)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/arboreto/algo.py", line 41, in grnboost2
early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/arboreto/algo.py", line 128, in diy
seed=seed)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/arboreto/core.py", line 403, in create_graph
future_tf_matrix = client.scatter(tf_matrix, broadcast=True)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/client.py", line 2071, in scatter
hash=hash,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/client.py", line 753, in sync
return sync(self.loop, func, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/utils.py", line 331, in sync
six.reraise(*error[0])
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/six.py", line 696, in reraise
raise value
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/utils.py", line 316, in f
result[0] = yield future
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/client.py", line 1916, in _scatter
timeout=timeout,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/core.py", line 739, in send_recv_from_rpc
result = yield send_recv(comm=comm, op=key, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/core.py", line 533, in send_recv
response = yield comm.read(deserializers=deserializers)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/tcp.py", line 218, in read
frames, deserialize=self.deserialize, deserializers=deserializers
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py", line 85, in from_frames
res = _from_frames()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py", line 71, in _from_frames
frames, deserialize=deserialize, deserializers=deserializers
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py", line 108, in loads
header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key

I know that someone else posted the same error but that solution did not fix my problem.

Thank you

[liekevandehaar]

Hi! I am having the same problem running the tutorial of pySCENIC (not from jupyter notebook). Also for me the solutions mentioned in issue#147 are not working.

adjacencies = grnboost2(expression_data=ex_matrix, tf_names=tf_names, verbose=True)
ValueError: tuple is not allowed for map key

arboreto                0.1.5              
asn1crypto              1.3.0              
attrs                   19.3.0             
boltons                 20.1.0             
certifi                 2019.11.28         
cffi                    1.14.0             
chardet                 3.0.4              
click                   7.1.1              
cloudpickle             1.3.0              
conda                   4.8.2              
conda-package-handling  1.6.0              
cryptography            2.8                
cytoolz                 0.10.1             
dask                    2.14.0             
decorator               4.4.2              
dill                    0.3.1.1            
distributed             2.14.0             
frozendict              1.2                
fsspec                  0.7.2              
h5py                    2.10.0             
HeapDict                1.0.1              
idna                    2.8                
interlap                0.2.6              
joblib                  0.14.1             
llvmlite                0.31.0             
locket                  0.2.0              
loompy                  2.0.2              
msgpack                 1.0.0              
multiprocessing-on-dill 3.5.0a4            
networkx                2.4                
numba                   0.48.0             
numpy                   1.18.2             
numpy-groupies          0+unknown          
pandas                  0.25.3             
partd                   1.1.0              
pip                     20.0.2             
psutil                  5.7.0              
pyarrow                 0.16.0             
pycosat                 0.6.3              
pycparser               2.19               
pyOpenSSL               19.1.0             
pyscenic                0.10.0             
PySocks                 1.7.1              
python-dateutil         2.8.1              
pytz                    2019.3             
PyYAML                  5.3.1              
requests                2.22.0             
ruamel-yaml             0.15.87            
scikit-learn            0.22.2.post1       
scipy                   1.4.1              
setuptools              45.2.0.post20200210
six                     1.14.0             
sortedcontainers        2.1.0              
tbb                     2020.0.133         
tblib                   1.6.0              
toolz                   0.10.0             
tornado                 6.0.4              
tqdm                    4.42.1             
typing                  3.7.4.1            
umap-learn              0.4.0              
urllib3                 1.25.8             
wheel                   0.34.2             
zict                    2.0.0   

[cflerin]

Hi, I would recommend downgrading dask/distrubuted to an older version as described here. You can also find in that link an alternative script that should be more stable than the dask implementation. It seems like others have gotten later versions of dask to run successfully in #147, but I have had the most success with the multiprocessing script.

[liekevandehaar]

@cflerin Thanks for the answer, I will give it a try.

[KabitaBaral1]

It worked for me. Thank you very much

[linanzhang]

It does not work for me :(