I hope ZoneTree can provide a better out-of-the-box performance experience

[LeoYang06]

I am happy that the .NET ecosystem has an excellent work like ZoneTree, and I am very grateful for the author's efforts. Compared to LiteDB and DBreeze, ZoneTree can be a bit challenging to use out of the box, or it requires a good understanding to leverage its best performance; otherwise, unexpected issues may arise. I understand this is the price of flexibility, and at times, it can also be an advantage. I have two questions hoping to get answers, thank you!

1.After completing data write operations, I want them to be immediately persisted to disk. I use maintainer.EvictToDisk(); maintainer.WaitForBackgroundThreads(); but it takes a long time. In my benchmark tests, it is significantly slower than LiteDB and DBreeze, which is clearly not as expected.

2.The client-side application is very sensitive to memory usage. I want to minimize memory consumption while still maintaining performance advantages. I had to go through a lot of parameter tuning to achieve a balance between memory usage and performance. It would be great if there were several built-in modes, allowing for a better out-of-the-box experience.

This is just my superficial understanding, and it might be due to my insufficient grasp of the ZoneTree design, but this is from a user's perspective.

[koculu]

Hi, thank you for your kind words and feedback!

Let me clarify a few points that I believe may help address your concerns:

ZoneTree uses a Write-Ahead Log (WAL), meaning data is persisted to disk immediately during each write operation. This ensures durability without the need to call EvictToDisk after every write.

The EvictToDisk operation is only necessary if you specifically want to clear data from memory to free up space. If you’re okay with retaining some data in the WAL, calling EvictToDisk isn’t required.

Additionally, ZoneTree automatically handles data eviction to disk in the background. This process is managed by the MutableSegmentMaxItemCount parameter in ZoneTreeOptions. Once this item count limit is reached, automatic eviction (or compaction) begins without any manual action, allowing you to balance memory usage and performance based on your needs.

Thank you for your suggestions to improve the library! Contributions towards preset configurations are always welcome.

[koculu]

I have executed some benchmarks using LiteDB and DBreeze, and neither of them is directly competitive in terms of performance with ZoneTree.

In short, you have misused ZoneTree. What you are doing is similar to pressing the gas pedal and brake pedal over and over again and then complaining about the speed of the car.

ZoneTree's Upsert and Insert methods are thread-safe, and the data is persisted to the disk immediately.

Never call EvictToDisk manually; it is called automatically when needed. If you really want to do it, always let it run in the background. Don't block your main thread using WaitForBackgroundThreads.

[LeoYang06]

Thank you for your detailed explanation. I realize that I have been abusing the system, which has prevented me from achieving the expected performance. I used EvictToDisk because some data was not persisted in a timely manner when the program was closed, which may have been related to me disabling WAL at the time.

I have also conducted extensive benchmark tests, and I can confirm that ZoneTree is definitely on par with Rocksdb and LMDB.