Switch tests to use HDF reader instead of kerchunk-based HDF5 reader #374
Labels
dependencies
Updates a dependency
HDF reader
Non-kerchunk-based HDF reader backend
Kerchunk
Relating to the kerchunk library / specification itself
testing
Zarr v3 was officially released today, but we still can't just explicitly depend on it because kerchunk doesn't work with zarr v3. There is a compatibility branch of kerchunk, which is sufficient to get kerchunk's readers working again, but development of that branch has stalled due to errors that arise in kerchunk's
MultiZarrToZarr
code.We can't just depend directly on this compatibility branch because that's janky AF.
If we try to change our dependency to zarr v3 these things will break:
Breaking the kerchunk-based readers wouldn't be that bad - we have @sharkinsspatial 's HDF reader as an alternative for opening HDF/netCDF4 files, and the other two are much less complex/commonly used.
Breaking our integration tests isn't really acceptable, so what we should try is swapping all those tests to use Sean's HDF reader instead of the kerchunk-based HDF5 reader. If that works then we can move forward, and the only downside is that we would have to raise a warning in the HDF5, netCDF3, and FITS readers that you need to install the weird compatibility branch of kerchunk. That's better than the current situation, where you need the weird compatibility branch if you want to use zarr v3 (and therefore icechunk!) at all.
cc @norlandrhagen @mpiannucci
The text was updated successfully, but these errors were encountered: