Performance: Blockstore

[bw-solana]

Some initial (raw) commentary on Blockstore perf

solWinInsert spends ~10% of the time deserializing bincode columns from rocks.

What's in there? why is it bincode? If it must be bincode, we should write hand crafted parsers like the one Alessandro wrote for VoteState which is ~3000x faster than the bincode crate.

~5% of shred recv is actually writing shreds inside rocksdb, half of which is pagefaulting the memtable (we probably have configured the memtable to be too small, so it's getting grown). The other half is performing write syscalls to the filesystem. We probably need to configure much larger buffers.

Completed Items

• blockstore: Move rocksdb keys from heap to stack #3603
  • Refactors to enable this: 3650, 3687, 3712, 3935, 3934. 4030
• blockstore: Use Vec::from() instead of slice::to_vec() #4047
• blockstore: Reduce intermediate collects from multi_get() #4312

Inflight Items

• Blockstore: Migrate ShredIndex type to more efficient data structure #3570
  • Current PR: Blockstore: Migrate ShredIndex type to more efficient data structure #3900

Potential Future Work

• Blockstore: Avoid extra heap allocation in LedgerColumn::iter() #3850
• Shreds incur extra copies in several places #1142
• [DRAFT] Blockstore: Reduce WAL Usage #4132