Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write zarr stores with manifest.json files to non-local storage #46

Closed
Tracked by #132
TomNicholas opened this issue Mar 25, 2024 · 3 comments
Closed
Tracked by #132
Labels
enhancement New feature or request

Comments

@TomNicholas
Copy link
Member

TomNicholas commented Mar 25, 2024

Awesome to see this develop, Tom!

Is there a goal of being able to write to non-local storage devices? If so we'd need a different (fsspec?) opener for the context manager?

Originally posted by @cisaacstern in #45 (comment)

@TomNicholas
Copy link
Member Author

@cisaacstern I think now that Icechunk exists this is probably out of date? Or is there still something to do if someone wanted to dump kerchunk references into a bucket?

@cisaacstern
Copy link
Collaborator

👋 @TomNicholas, I guess it depends if you want these methods

@classmethod
def from_zarr_json(cls, filepath: str) -> "ChunkManifest":
"""Create a ChunkManifest from a Zarr manifest.json file."""
with open(filepath, "r") as manifest_file:
entries = json.load(manifest_file)
return cls(entries=entries)
def to_zarr_json(self, filepath: str) -> None:
"""Write the manifest to a Zarr manifest.json file."""
entries = self.dict()
with open(filepath, "w") as json_file:
json.dump(entries, json_file, indent=4, separators=(", ", ": "))

to be able to read json from a cloud bucket or any other non-local device.

If so, then I would say this issue is still open. If not, then fine to close!

Not sure if/how the existence of Icechunk relates to this, but at the level of those specific methods it appears to my naive eye as unrelated?

In terms of my own requirements (none), I am personally fine if this is out of scope / wontfix.

@TomNicholas
Copy link
Member Author

Thanks for the clarification @cisaacstern - I think this can be closed, see #359 for the reasoning.

@TomNicholas TomNicholas closed this as not planned Won't fix, can't repro, duplicate, stale Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants