Ideas

[koculu]

Ideas page is locked.

Please create new discussion pages when you want to share new ideas.

[dodyg]

Are you planning to build a higher abstraction on top of ZoneTree? e.g. a Document Store or something similar

[koculu]

Hi @dodyg,
I am working on a separate GraphDB project that uses ZoneTree.
It will also be open-source with an MIT license.

ZoneTree is a core database engine with several tuning options. Thus its primary target audience is advanced developers.

People can create scalable or non-scalable document databases, graph databases, SQL databases, or other kinds of databases by using ZoneTree.

ZoneTree works in a single instance. However, like any other core database engine, it is possible to use clustering and consensus algorithms to create scalable databases on top of ZoneTree.

As far as I know, @KSemenenko started to develop a scalable time-series database for analytics using ZoneTree. He builds his database on Azure Blob Storage with the help of the FileStream abstraction of ZoneTree.

I believe ZoneTree is an excellent opportunity for the NET ecosystem.

Today, there are several popular NoSQL or Graph databases written in C++, Java, and Go, but only a few in the NET. Yet, I am hopeful for tomorrow, and I don't doubt that C#'s power, flexibility, and elegance will be the winning factor in the long term.

Thank you for listening to my long answer to a short question. I have more to say, but I reserve that for documentation.

[dodyg]

Nice. Yup. I am really excited that you are working on this. Indeed we need more C# based infrastructure projects.

[harrison314]

@koculu Will there be an example posted on how to use ZoneTree to create a secondary index (in context create custom database on top of ZoneTree)?

[koculu]

This article might help you.
https://misfra.me/2017/01/18/how-to-implement-secondary-indexes/

There are multiple ways of creating a secondary index. Above is a blog post that I found using google. I believe you would find much better resources to guide you about secondary-indexes.
There isn't any plan, for now, to support a built-in secondary index at ZoneTree.

[harrison314]

Thank you.
It didn't even occur to me that it could be so simple.

[ljcollins25]

Any thoughts on adding merge operator support (https://github.com/facebook/rocksdb/wiki/Merge-Operator)?

[koculu]

Hi @ljcollins25

The good news! It is already supported.
There are atomic methods to modify records atomically. It is a first class citizen in ZoneTree.
The following methods are available through IZoneTree interface.

• AtomicUpsert
• TryAtomicAdd
• TryAtomicAddOrUpdate
• TryAtomicGetAndUpdate
• TryAtomicUpdate

https://tenray.io/docs/ZoneTree/api/Tenray.ZoneTree.IZoneTree-2.html

There is also a unit test case that demonstrates atomic counters using ZoneTree:
https://github.com/koculu/ZoneTree/blob/main/src/ZoneTree.UnitTests/AtomicUpdateTests.cs

[ljcollins25]

Looking at the implementation of the operations you mentioned, it seems they are not equivalent to merge operators. Notably, merge operators allow writing updates without reading the value first. When looking up the value, the merge operators for every level of the LSM tree are read and merged into the final value. This is ideal for scenarios where you have lots of updates to values but fewer reads. When merging segments, the merge operators in those segments are merged. Another concerning aspect about the TryAtomic* operations is that they take a global lock. This would introduce a significant amount of lock contention in high volume, high concurrency scenarios.

[koculu]

Thanks for the pointers. I have no idea how RocksDB merge operation works at the moment.
I have to read more to understand how it works before making a comment on this topic.

[koculu]

I confirm @ljcollins25 that the Merge Operator feature in RocksDb is not equivalent to the atomic methods in ZoneTree.

The feature is specific (many writes, few reads scenarios), and I don't need this feature for my business. Therefore I will not work on that, at least for now.

It might be nice if you could develop the feature without impacting the current feature set and performance.

Regarding the atomic update methods, global lock contention:

All updates are written into the memory first. As a result, the lock contention will be small in most cases.
For example: if the mutable region size is 1M, parallel updates of 1M keys remain in memory.

[koculu]

I think I found a better approach to achieving the goal of merge operators in RocksDB.

The goal of the merge operator is to add parts of the value of a key in separate calls.

The value associated with the key could be either a list or a JSON-like document.

The merge operator feature in RocksDB utilizes levels in the LSM tree and makes it possible to merge the parts of the value from different levels. As far as I understand, the merge happens at the reading stage.

One can achieve the same thing without adding the complexity to the LSM Tree.

The proposal:

• Create a key that is paired with a timestamp or an auto-incremented id.
• Add different parts of the values using the same key but different timestamp/id.
• Use the iterators to fetch a record value and do the merge manually. (prefix search)
• Optionally, use the first record to keep the checkpoint id and insert the merged records in a background thread to optimize reads.

I think this approach achieves the same thing.
It is better, because the LSM Tree code stays simpler.

[koculu]

Ideas page is locked.
Please create new discussion pages when you want to share new ideas.