Add ensemble check to over-replicated ledger GC #2813

shustsud · 2021-10-05T23:53:34Z

Motivation

Issue is as described in PR#2797.

In one day, zookeepers became high cpu usage and disk full.
The cause of this is bookie's gc of overreplicated ledgers.
Gc created/deleted zk nodes under /ledgers/underreplication/locks very frequently and some bookies ran gc at same time.
As a result, zookeepers created a lot of snapshots and became disk full.

I want to reduce the number of lock node creations and deletions in ZK.

Changes

Add an ensemble check before creating the lock node.
This is to reduce the number of lock node creations and deletions in ZK.
~~If PR#2797 was merged, this PR needs to be fixed.~~

nicoloboschi

LGTM, I left a non blocker comment

nicoloboschi · 2021-10-06T07:03:59Z

...keeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java

+            }
+        }
+        latch.await();
+        LOG.info("Finished pre-check over-replicated ledgers. candidateOverReplicatedLedgersSize={}",


Maybe it can be interesting to log how many ledgers are over replicated over the total number of active ledgers (bkActiveLedgers)
something like

Over-replicated ledgers pre-check count: {}/{}, candidateOverReplicatedLedgers.size(), bkActiveLedgers.size()

nicoloboschi

LGTM

dlg99 · 2021-10-08T15:50:13Z

zookeepers created a lot of snapshots and became disk full

take a look at autopurge.snapRetainCount and autopurge.purgeInterval at http://zookeeper.apache.org/doc/r3.6.3/zookeeperAdmin.html#sc_advancedConfiguration

dlg99

Left couple of comments.

...keeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java

shustsud · 2021-10-11T01:51:56Z

take a look at autopurge.snapRetainCount and autopurge.purgeInterval at http://zookeeper.apache.org/doc/r3.6.3/zookeeperAdmin.html#sc_advancedConfiguration

Thank you for your comment.
I take a look at the URL above.

dlg99

LGTM. Thank you.
@eolivelli please take a look

shustsud · 2021-10-12T01:30:26Z

rerun failure checks

shustsud · 2021-10-12T02:13:18Z

rerun failure checks

zymap · 2021-10-15T00:20:04Z

@eolivelli Would you like to review this PR? @dlg99 want to include this PR into the release/4.14.3 so I want to merge this PR

zymap · 2021-10-18T02:00:21Z

ping @eolivelli

eolivelli

Lgtm
I left one request, PTAL

Happy to merge

eolivelli · 2021-10-18T06:25:28Z

...keeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java

+                // this is to reduce the number of lock node creations and deletions in ZK.
+                // the ensemble check is done again after the lock node is created.
+                // also, check if the ledger is being replicated already by the replication worker
+                if (!isNotBookieIncludedInLedgerEnsembles(ledgerManager.readLedgerMetadata(ledgerId).get())


readLedgerMetadata is a costly operation,
Can you please move it in a dedicated line, out of the 'if' statement?
This way is it more evident while reading the code?

@eolivelli
Thank you for your comment.
I fixed it as above.

zymap · 2021-10-20T00:38:59Z

@shustsud Could you please resolve the conflicts?

shustsud · 2021-10-21T02:43:39Z

@zymap
I resolved this PR conflict.

shustsud · 2021-10-22T02:14:58Z

rerun failure checks

shustsud · 2021-10-23T05:49:13Z

@zymap
I'm sorry. I have created the following PR.
　#2844

I think this PR should be included in the 4.14.3 release.

…#2844) ### Motivation For each ledger whose metadata is not in ZK, following stack trace will be output: ``` 15:30:17.925 [GarbageCollectorThread-11-1] ERROR o.a.b.b.ScanAndCompareGarbageCollector - Exception when iterating through the ledgers to check for over-replication java.util.concurrent.ExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.removeOverReplicatedledgers(ScanAndCompareGarbageCollector.java:199) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.gc(ScanAndCompareGarbageCollector.java:120) at org.apache.bookkeeper.bookie.GarbageCollectorThread.doGcLedgers(GarbageCollectorThread.java:372) at org.apache.bookkeeper.bookie.GarbageCollectorThread.runWithFlags(GarbageCollectorThread.java:323) at org.apache.bookkeeper.bookie.GarbageCollectorThread.safeRun(GarbageCollectorThread.java:301) at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:397) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:575) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` It is noisy, makes the size of log files large and finally causes OOM during log rotation. So we should suppress the stacktrace. (This problem is due to [#2813](#2813).) ### Changes Add error handling to readLedgerMetadata in over-replicated ledger GC in order to suppress the stacktrace.

### Motivation - Issue is as described in [PR#2797](#2797). > In one day, zookeepers became high cpu usage and disk full. > The cause of this is bookie's gc of overreplicated ledgers. > Gc created/deleted zk nodes under /ledgers/underreplication/locks very frequently and some bookies ran gc at same time. > As a result, zookeepers created a lot of snapshots and became disk full. - I want to reduce the number of lock node creations and deletions in ZK. ### Changes - Add an ensemble check before creating the lock node. This is to reduce the number of lock node creations and deletions in ZK. - ~~If [PR#2797](#2797) was merged, this PR needs to be fixed.~~ (cherry picked from commit 53954ca)

…#2844) ### Motivation For each ledger whose metadata is not in ZK, following stack trace will be output: ``` 15:30:17.925 [GarbageCollectorThread-11-1] ERROR o.a.b.b.ScanAndCompareGarbageCollector - Exception when iterating through the ledgers to check for over-replication java.util.concurrent.ExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.removeOverReplicatedledgers(ScanAndCompareGarbageCollector.java:199) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.gc(ScanAndCompareGarbageCollector.java:120) at org.apache.bookkeeper.bookie.GarbageCollectorThread.doGcLedgers(GarbageCollectorThread.java:372) at org.apache.bookkeeper.bookie.GarbageCollectorThread.runWithFlags(GarbageCollectorThread.java:323) at org.apache.bookkeeper.bookie.GarbageCollectorThread.safeRun(GarbageCollectorThread.java:301) at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:397) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:575) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` It is noisy, makes the size of log files large and finally causes OOM during log rotation. So we should suppress the stacktrace. (This problem is due to [#2813](#2813).) ### Changes Add error handling to readLedgerMetadata in over-replicated ledger GC in order to suppress the stacktrace. (cherry picked from commit bd5c50b)

### Motivation In order to complete migration to Gradle we must build all the subprojects. ### Changes - Enabled `sh` integration tests with gradle, located in `tests/scripts/src/test/bash/gradle` - Added these modules to the build - `bookkeeper-http:servlet-http-server` - `metadata-drivers:etcd` - `tests:backward-compat:*` - `tests:shaded:*` - `stream:bk-grpc-name-resolver` - DL shading process is now performed (before it didn't build any jar) - Groovy tests (`tests:backward-compat:*`) now are triggered by the build/tests itself; with Maven, there is a "runner" project (`tests/integration-tests-base-groovy`); in Gradle is useless so it is skipped ### Test - Both `bin/bookkeper standalone` and `bin/bookkeper_gradle standalone` work locally - Tests are passing locally Master Issue: #2849 Reviewers: Henry Saputra <[email protected]>, Prashant Kumar <None> This closes #2850 from nicoloboschi/fix/2849/gradle and squashes the following commits: 00b49f4 [Nicolò Boschi] Fix common_gradle.sh regex bd739fd [Nicolò Boschi] fix sh tests 43230ba [Nicolò Boschi] revert sh files. Avoid to modify maven files, create gradle versions to faciltate migration d1f95e4 [Nicolò Boschi] fix shaded deps bcab40d [Nicolò Boschi] fix build 5fd0341 [Nicolò Boschi] fix build 0082e0e [Nicolò Boschi] fix build 2c32ac1 [Nicolò Boschi] fixes 3bc0b26 [Nicolò Boschi] bookkeeper-server-shaded-tests ba89132 [Nicolò Boschi] shaded tests 6d39e33 [Nicolò Boschi] sh tests e0032bc [Nicolò Boschi] actually run arquillian groovy tests 08dcc39 [Nicolò Boschi] backwards 2361f79 [Nicolò Boschi] hierarchical-ledger-manager 8388e11 [Nicolò Boschi] current-server-old-clients 6a24344 [Nicolò Boschi] bc-non-fips 2faca01 [Nicolò Boschi] bk-grpc-name-resolver 991bc11 [Nicolò Boschi] servlet-http-server 675ef7b [Nicolò Boschi] etcd b1d5e14 [ZhangJian He] A empty implement in EtcdLedgerManagerFactory to let the project can compile (#2845) bd5c50b [shustsud] Add error handling to readLedgerMetadata in over-replicated ledger GC (#2844) 746f9f6 [Matteo Merli] Remove direct ZK access for Auditor (#2842) 4117200 [ZhangJian He] the compare should be >= instead of > (#2782) 14ef56f [Prashant Kumar] BookieId can not be cast to BookieSocketAddress (#2843) e10f3fe [ZhangJian He] Forget to close preAllocator log on shutdown (#2819) 53954ca [shustsud] Add ensemble check to over-replicated ledger GC (#2813) 919fdd3 [Prashant Kumar] Issue:2840 Create bookie shellscript for gradle (#2841) 031d168 [gaozhangmin] fix-npe-when-pulsar-ZkBookieRackAffinityMapping-getBookieAddressResolver (#2788) 3dd671c [Prashant Kumar] Migrate bookkeepr-server:test to gradle run unit tests excepts org.apache.bookkeeper.bookie. org.apache.bookkeeper.client org.apache.bookkeeper.replication org.apache.bookkeeper.tls. (#2812) f6903b8 [Jack Vanlightly] BP-44 USE metrics a4afaa4 [Matteo Merli] Eliminate direct ZK access in ScanAndCompareGarbageCollector (#2833) a9b576d [Yunze Xu] Release semaphore when addEntry accepts the same entries (#2832) 148bf22 [Yun Tang] Ensure to release cache during KeyValueStorageRocksDB#closec (#2821) 4dc4260 [gaozhangmin] Heap memory leak problem when ledger replication failed (#2794) a522fa3 [Raúl Gracia] Issue 2815: Upgrade to log4j2 to get rid of CVE-2019-17571 (#2816) 0465052 [Nicolò Boschi] Upgrade httpclient from 4.5.5 to 4.5.13 (#2793) 594a056 [Raúl Gracia] Issue 2795: Bookkeeper upgrade using Bookie ID may fail due to cookie mismatch (#2796) 354cf37 [Raúl Gracia] Upgraded dependencies with CVEs (#2792) e413c70 [Raúl Gracia] Issue 2728: Entry Log GC may get blocked when using entryLogPerLedgerEnabled option (#2779) 883231e [pradeepbn] Building bookkeeper with gradle on java11

### Motivation - Issue is as described in [PR#2797](apache#2797). > In one day, zookeepers became high cpu usage and disk full. > The cause of this is bookie's gc of overreplicated ledgers. > Gc created/deleted zk nodes under /ledgers/underreplication/locks very frequently and some bookies ran gc at same time. > As a result, zookeepers created a lot of snapshots and became disk full. - I want to reduce the number of lock node creations and deletions in ZK. ### Changes - Add an ensemble check before creating the lock node. This is to reduce the number of lock node creations and deletions in ZK. - ~~If [PR#2797](apache#2797) was merged, this PR needs to be fixed.~~

…apache#2844) ### Motivation For each ledger whose metadata is not in ZK, following stack trace will be output: ``` 15:30:17.925 [GarbageCollectorThread-11-1] ERROR o.a.b.b.ScanAndCompareGarbageCollector - Exception when iterating through the ledgers to check for over-replication java.util.concurrent.ExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.removeOverReplicatedledgers(ScanAndCompareGarbageCollector.java:199) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.gc(ScanAndCompareGarbageCollector.java:120) at org.apache.bookkeeper.bookie.GarbageCollectorThread.doGcLedgers(GarbageCollectorThread.java:372) at org.apache.bookkeeper.bookie.GarbageCollectorThread.runWithFlags(GarbageCollectorThread.java:323) at org.apache.bookkeeper.bookie.GarbageCollectorThread.safeRun(GarbageCollectorThread.java:301) at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:397) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:575) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ``` It is noisy, makes the size of log files large and finally causes OOM during log rotation. So we should suppress the stacktrace. (This problem is due to [apache#2813](apache#2813).) ### Changes Add error handling to readLedgerMetadata in over-replicated ledger GC in order to suppress the stacktrace.

### Motivation In order to complete migration to Gradle we must build all the subprojects. ### Changes - Enabled `sh` integration tests with gradle, located in `tests/scripts/src/test/bash/gradle` - Added these modules to the build - `bookkeeper-http:servlet-http-server` - `metadata-drivers:etcd` - `tests:backward-compat:*` - `tests:shaded:*` - `stream:bk-grpc-name-resolver` - DL shading process is now performed (before it didn't build any jar) - Groovy tests (`tests:backward-compat:*`) now are triggered by the build/tests itself; with Maven, there is a "runner" project (`tests/integration-tests-base-groovy`); in Gradle is useless so it is skipped ### Test - Both `bin/bookkeper standalone` and `bin/bookkeper_gradle standalone` work locally - Tests are passing locally Master Issue: apache#2849 Reviewers: Henry Saputra <[email protected]>, Prashant Kumar <None> This closes apache#2850 from nicoloboschi/fix/2849/gradle and squashes the following commits: 00b49f4 [Nicolò Boschi] Fix common_gradle.sh regex bd739fd [Nicolò Boschi] fix sh tests 43230ba [Nicolò Boschi] revert sh files. Avoid to modify maven files, create gradle versions to faciltate migration d1f95e4 [Nicolò Boschi] fix shaded deps bcab40d [Nicolò Boschi] fix build 5fd0341 [Nicolò Boschi] fix build 0082e0e [Nicolò Boschi] fix build 2c32ac1 [Nicolò Boschi] fixes 3bc0b26 [Nicolò Boschi] bookkeeper-server-shaded-tests ba89132 [Nicolò Boschi] shaded tests 6d39e33 [Nicolò Boschi] sh tests e0032bc [Nicolò Boschi] actually run arquillian groovy tests 08dcc39 [Nicolò Boschi] backwards 2361f79 [Nicolò Boschi] hierarchical-ledger-manager 8388e11 [Nicolò Boschi] current-server-old-clients 6a24344 [Nicolò Boschi] bc-non-fips 2faca01 [Nicolò Boschi] bk-grpc-name-resolver 991bc11 [Nicolò Boschi] servlet-http-server 675ef7b [Nicolò Boschi] etcd b1d5e14 [ZhangJian He] A empty implement in EtcdLedgerManagerFactory to let the project can compile (apache#2845) bd5c50b [shustsud] Add error handling to readLedgerMetadata in over-replicated ledger GC (apache#2844) 746f9f6 [Matteo Merli] Remove direct ZK access for Auditor (apache#2842) 4117200 [ZhangJian He] the compare should be >= instead of > (apache#2782) 14ef56f [Prashant Kumar] BookieId can not be cast to BookieSocketAddress (apache#2843) e10f3fe [ZhangJian He] Forget to close preAllocator log on shutdown (apache#2819) 53954ca [shustsud] Add ensemble check to over-replicated ledger GC (apache#2813) 919fdd3 [Prashant Kumar] Issue:2840 Create bookie shellscript for gradle (apache#2841) 031d168 [gaozhangmin] fix-npe-when-pulsar-ZkBookieRackAffinityMapping-getBookieAddressResolver (apache#2788) 3dd671c [Prashant Kumar] Migrate bookkeepr-server:test to gradle run unit tests excepts org.apache.bookkeeper.bookie. org.apache.bookkeeper.client org.apache.bookkeeper.replication org.apache.bookkeeper.tls. (apache#2812) f6903b8 [Jack Vanlightly] BP-44 USE metrics a4afaa4 [Matteo Merli] Eliminate direct ZK access in ScanAndCompareGarbageCollector (apache#2833) a9b576d [Yunze Xu] Release semaphore when addEntry accepts the same entries (apache#2832) 148bf22 [Yun Tang] Ensure to release cache during KeyValueStorageRocksDB#closec (apache#2821) 4dc4260 [gaozhangmin] Heap memory leak problem when ledger replication failed (apache#2794) a522fa3 [Raúl Gracia] Issue 2815: Upgrade to log4j2 to get rid of CVE-2019-17571 (apache#2816) 0465052 [Nicolò Boschi] Upgrade httpclient from 4.5.5 to 4.5.13 (apache#2793) 594a056 [Raúl Gracia] Issue 2795: Bookkeeper upgrade using Bookie ID may fail due to cookie mismatch (apache#2796) 354cf37 [Raúl Gracia] Upgraded dependencies with CVEs (apache#2792) e413c70 [Raúl Gracia] Issue 2728: Entry Log GC may get blocked when using entryLogPerLedgerEnabled option (apache#2779) 883231e [pradeepbn] Building bookkeeper with gradle on java11

nicoloboschi approved these changes Oct 6, 2021

View reviewed changes

nicoloboschi approved these changes Oct 7, 2021

View reviewed changes

dlg99 requested a review from eolivelli October 8, 2021 15:52

dlg99 requested changes Oct 8, 2021

View reviewed changes

...keeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java Outdated Show resolved Hide resolved

...keeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java Outdated Show resolved Hide resolved

dlg99 approved these changes Oct 11, 2021

View reviewed changes

shustsud changed the title ~~Add pre-check to over-replicated ledger GC~~ Add ensemble check to over-replicated ledger GC Oct 12, 2021

zymap assigned shustsud Oct 15, 2021

zymap added the release/4.14.3 label Oct 15, 2021

eolivelli approved these changes Oct 18, 2021

View reviewed changes

shustsud added 4 commits October 21, 2021 10:17

Add pre-check to over-replicated ledger GC

8f5f600

Fixed log

c316900

Fixed pre-check

cb3452b

Moved readLedgerMetadata

728fbba

shustsud force-pushed the add_pre-check_to_over-replicated_ledger_gc branch from 766eb36 to 728fbba Compare October 21, 2021 02:34

zymap merged commit 53954ca into apache:master Oct 22, 2021

shustsud mentioned this pull request Oct 23, 2021

Add error handling to readLedgerMetadata in over-replicated ledger GC #2844

Merged

rdhabalia assigned shustsud and unassigned shustsud Feb 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ensemble check to over-replicated ledger GC #2813

Add ensemble check to over-replicated ledger GC #2813

shustsud commented Oct 5, 2021 •

edited

Loading

nicoloboschi left a comment

nicoloboschi Oct 6, 2021

shustsud Oct 7, 2021

nicoloboschi left a comment

dlg99 commented Oct 8, 2021

dlg99 left a comment

shustsud commented Oct 11, 2021

dlg99 left a comment

shustsud commented Oct 12, 2021

shustsud commented Oct 12, 2021

zymap commented Oct 15, 2021

zymap commented Oct 18, 2021

eolivelli left a comment

eolivelli Oct 18, 2021

shustsud Oct 18, 2021

zymap commented Oct 20, 2021

shustsud commented Oct 21, 2021

shustsud commented Oct 22, 2021

shustsud commented Oct 23, 2021

Add ensemble check to over-replicated ledger GC #2813

Add ensemble check to over-replicated ledger GC #2813

Conversation

shustsud commented Oct 5, 2021 • edited Loading

Motivation

Changes

nicoloboschi left a comment

Choose a reason for hiding this comment

nicoloboschi Oct 6, 2021

Choose a reason for hiding this comment

shustsud Oct 7, 2021

Choose a reason for hiding this comment

nicoloboschi left a comment

Choose a reason for hiding this comment

dlg99 commented Oct 8, 2021

dlg99 left a comment

Choose a reason for hiding this comment

shustsud commented Oct 11, 2021

dlg99 left a comment

Choose a reason for hiding this comment

shustsud commented Oct 12, 2021

shustsud commented Oct 12, 2021

zymap commented Oct 15, 2021

zymap commented Oct 18, 2021

eolivelli left a comment

Choose a reason for hiding this comment

eolivelli Oct 18, 2021

Choose a reason for hiding this comment

shustsud Oct 18, 2021

Choose a reason for hiding this comment

zymap commented Oct 20, 2021

shustsud commented Oct 21, 2021

shustsud commented Oct 22, 2021

shustsud commented Oct 23, 2021

shustsud commented Oct 5, 2021 •

edited

Loading