Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch tests to use HDF reader instead of kerchunk-based HDF5 reader #374

Open
TomNicholas opened this issue Jan 9, 2025 · 0 comments
Open
Labels
dependencies Updates a dependency HDF reader Non-kerchunk-based HDF reader backend Kerchunk Relating to the kerchunk library / specification itself testing

Comments

@TomNicholas
Copy link
Member

Zarr v3 was officially released today, but we still can't just explicitly depend on it because kerchunk doesn't work with zarr v3. There is a compatibility branch of kerchunk, which is sufficient to get kerchunk's readers working again, but development of that branch has stalled due to errors that arise in kerchunk's MultiZarrToZarr code.

We can't just depend directly on this compatibility branch because that's janky AF.

If we try to change our dependency to zarr v3 these things will break:

  • kerchunk-based readers (HDF5, netCDF3, and FITS)
  • half our integration tests, which round-trip using the HDF5 reader.

Breaking the kerchunk-based readers wouldn't be that bad - we have @sharkinsspatial 's HDF reader as an alternative for opening HDF/netCDF4 files, and the other two are much less complex/commonly used.

Breaking our integration tests isn't really acceptable, so what we should try is swapping all those tests to use Sean's HDF reader instead of the kerchunk-based HDF5 reader. If that works then we can move forward, and the only downside is that we would have to raise a warning in the HDF5, netCDF3, and FITS readers that you need to install the weird compatibility branch of kerchunk. That's better than the current situation, where you need the weird compatibility branch if you want to use zarr v3 (and therefore icechunk!) at all.

cc @norlandrhagen @mpiannucci

@TomNicholas TomNicholas added dependencies Updates a dependency HDF reader Non-kerchunk-based HDF reader backend Kerchunk Relating to the kerchunk library / specification itself testing labels Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Updates a dependency HDF reader Non-kerchunk-based HDF reader backend Kerchunk Relating to the kerchunk library / specification itself testing
Projects
None yet
Development

No branches or pull requests

1 participant