Jetstream file storage internals

[maxless]

Hi. I'm evaluating using Jetstream for the case of user to user and system to user messaging in our system (the messages are rare and no greater than 4kb but there are potentially millions of users that we need to store the messages for up to a week when they are offline) and I can't find any documentation that describes what actually happens with stream messages once the memory limit of a NATS Jetstream node is reached and the file storage kicks in. What's the offload strategy and algorithm? What happens if a consumer asks for messages that are on disk but not in memory? What happens on NATS node restart?

[neilalexander]

Streams are either memory-backed or file-backed. There's no offload from memory to file after a certain threshold is reached, on a file-based stream the messages are persisted straight to disk.

[maxless]

So in the file store case Jetstream essentiallly works like a file-based database? Is there some indexing and such to optimize pulling messages with a specific subject of a file-based stream or is the full stream loaded? Basically, will the performance of the consumer fetch call be comparable to "SELECT msg FROM Messages WHERE UserID = xxx" in an SQL database. And the same in case of multiple subjects.

[neilalexander]

We have per-subject indexing, so we can do optimised lookups for messages by subject.

[Jarema]

The best way to measure performance for your use case is to use our CLI and nats bench command. It will give you, in a convinient way, a good idea of how NATS can perform, as it of course depends on the subject cardinality, stream characteristics, what exactly you look up etc.

[maxless]

Ok, thanks for the answers. Are you talking about the --multisubject benchmark option, @Jarema?