What file formats should we support for the VFS?

[RaisinTen]

According to the discussion that took place in #17, there are suggestions on using the following file formats for the embedded VFS:

• tgz produced by npm pack - Bundling of non-binary assets within the binaries #17 (reply in thread)
• asar file format - Bundling of non-binary assets within the binaries #17 (reply in thread)
• zip file format - Bundling of non-binary assets within the binaries #17 (reply in thread)

Questions:

• Should we have a single blessed file format? If so, which one?
• Or should we keep that flexible and allow any custom file format by having users input the matching parser? If so:
  • How do we expect users to input the parser? It can't be a part of the embedded VFS itself because we would need the parser to parse out the parser from the VFS in the first place and that feels like a chicken and egg situation. The solution I can think of is - have a way to embed the VFS and the parser (bundled into a single file) separately (I tried to use Postject for this approach but it crashes currently - Adding multiple sections to the __POSTJECT segment causes a crash postject#1). Are there any better ways y'all can think of?
  • How do we expect users to select a file format for their application?

[jviotti]

If a specific VFS implementation checks all the boxes, that I don't see why we would want the additional overhead of supporting arbitrary ones.

But before we make such a decision, we need to know exactly what is needed from a VFS implementation to be able to "check all the boxes".

Can we take https://github.com/nodejs/single-executable/blob/main/blog/2022-08-05-an-overview-of-the-current-state.md#virtual-file-system as a starting point and commit on this repo a complete list of mandatory and optional requirements for the VFS that based on everybody's input, it does seem to check all the boxes?

Once we have clear requirements, we can run some experiments to categorize how each of the available VFS implementations rank according to those requirements.

[RaisinTen]

If a specific VFS implementation checks all the boxes, that I don't see why we would want the additional overhead of supporting arbitrary ones.

My fear is that multiple filesystems might check the box. If we hardcode a specific filesystem, there might be another existing filesystem which might better suite an application based on its needs and then folks might get annoyed if they are forced to use a particular filesystem. But I've opened a PR to list out the requirements nonetheless - #28, let's see what everyone thinks!

[jviotti]

Maybe! Let's do some experimentation and then we'll know better

[ovflowd]

It might be just me, but what I consider a higher priority level:

• Whose are more native and require fewer requirements/dependencies across the table?
• Whose has a faster decompressing rate, as a vfs we would determine if data gets decompressed on bootstrap. on-demand?
• While fetching data, is there any active garbage collecting mechanism we can use?
• Which one would produce the smallest bundle sizes? Can we use compression algorithms like LZ4/LZMGF/GZIP et cetera
• Which one has higher stability across the chart?

[jviotti]

Very good points! To expand on some of them:

Whose has a faster decompressing rate, as a vfs we would determine if data gets decompressed on bootstrap. on-demand?

This assumes we want compression. I personally agree, but it is a question we do need to bring to the table and make sure we all agree.

Which one would produce the smallest bundle sizes? Can we use compression algorithms like LZ4/LZMGF/GZIP et cetera

One problem here is that different compressors are best for different types of data. Which brings me to the point that we should probably support different file entries on the VFS being compressed using different formats. And potentially different compression levels of the same format.

Which one has higher stability across the chart?

@ovflowd How would we measure this, though? Simple formats tend to yield simple implementations with less errors, so that might be one way of looking at it. Also whether it is used in production already?

[ovflowd]

This assumes we want compression. I personally agree, but it is a question we do need to bring to the table and make sure we all agree.

What about... optional compression? (By default maybe disabling it, as you mentioned yourself, different kinds of compressions can... have different results across the charts for different files). Still today I see plain zip having some benefits over rar or 7zip depending what's inside the archive 😅

One problem here is that different compressors are best for different types of data. Which brings me to the point that we should probably support different file entries on the VFS being compressed using different formats. And potentially different compression levels of the same format.

Hehe, I thought about this! WDYT? #22

@ovflowd How would we measure this, though? Simple formats tend to yield simple implementations with less errors, so that might be one way of looking at it. Also whether it is used in production already?

Benchmarking prototypes and tests? I would believe we could support a few formats, and then have public benchmarks and tests. In the end I would love to support more formats rather than just one for the VFS, and let the user in the control. Of course, that's on them if things get.. nasty.

Of course in the end after initial tests, we would, possibly, determine a default one. I can for sure see how the first alpha versions of SEA will have bugs regarding this 😅

[RaisinTen]

These are some great points. Feel free to suggest those in #28!

[ovflowd]

Thanks, @RaisinTen!