[move-prover][cli] Add benchmark functionality

[wrwg]

Description

This adds a new option aptos move prove --benchmark which lets verification run in the benchmark framework of the move-prover/lab tool. This tool verifies function by function and detects timeouts as well as measure verification time.

The result of the benchmark will be stored in <move-package>/prover_benchmark.fun_data. A graphic representation will be stored as well in prover_benchmark.svg.

The PR fixes also some other things on the way which came apparent when working on it, namely trying to identify the reasons for #15605:

• Adds new debug print for which verification targets are generated. This is reachable via move prove -v debug
• Reduces unnecessary verification of symmetric type instantiations f<#1, #2> and f<#2, #1>
• Adds an option --skip-instance-check to completely turn off verification of type instantiations.
• Removed legacy verification_analysis_v2 and global_invariant_instrumentation_v2. These are dead code since long and confusing when trying to understand what functions are verified

How Has This Been Tested?

Existing tests for not breaking something, and manually for the new feature

[trunk-io]

⏱️ 8h 23m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
execution-performance / single-node-performance	3h 13m	🟩 🟩 🟩 🟩 🟩 (+5 more)
execution-performance / test-target-determinator	34m	🟩 🟩 🟩 🟩 🟩 (+5 more)
test-target-determinator	29m	🟩 🟩 🟩 🟩 🟩 (+5 more)
check	22m	🟥 🟩 🟩 🟩 🟩 (+6 more)
rust-cargo-deny	19m	🟩 🟩 🟩 🟩 🟩 (+7 more)
rust-move-tests	14m	🟩
check-dynamic-deps	14m	🟩 🟩 🟩 🟩 🟩 (+7 more)
fetch-last-released-docker-image-tag	14m	🟩 🟩 🟩 🟩 🟩 (+5 more)
rust-move-tests	13m	🟩
rust-move-tests	13m	🟩
rust-move-tests	13m	🟩
rust-move-tests	13m	🟩
rust-move-tests	13m	🟩
rust-move-tests	10m	⬜
forge-framework-upgrade-test / forge	10m	🟥

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / test-target-determinator	3m	5m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[wrwg]

• [move-prover][cli] Add benchmark functionality #15636 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[wrwg]

Here is an example of generated SVG output file for an example from econia

[rahxephon89]

LGTM, thanks!

[github-actions]

❌ Forge suite framework_upgrade failure on 6593fb81261f25490ffddc2252a861c994234c2a ==> a17f57db2d0f5d2e107a87011211d10a06761cf2

Compatibility test results for 6593fb81261f25490ffddc2252a861c994234c2a ==> a17f57db2d0f5d2e107a87011211d10a06761cf2 (PR)
Upgrade the nodes to version: a17f57db2d0f5d2e107a87011211d10a06761cf2
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1394.74 txn/s, submitted: 1398.34 txn/s, failed submission: 3.60 txn/s, expired: 3.60 txn/s, latency: 2227.98 ms, (p50: 2100 ms, p70: 2400, p90: 3000 ms, p99: 4200 ms), latency samples: 124060
Test Failed: API error: Unknown error Transaction committed on chain, but failed execution: Move abort in 0x1::gas_schedule: EINVALID_GAS_FEATURE_VERSION(0x10002): 

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:564:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:1964:27
   2: aptos_release_builder::validate::NetworkConfig::execute_proposal::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:342:9
   3: aptos_release_builder::validate::NetworkConfig::submit_and_execute_multi_step_proposal::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:116:64
   4: aptos_release_builder::validate::execute_release::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:412:22
   5: aptos_release_builder::validate::validate_config_and_generate_release::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:475:6
   6: aptos_release_builder::validate::validate_config::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:461:80
   7: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:144:10
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  17: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  18: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  19: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:428:24
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:331:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  24: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start
Trailing Log Lines:
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="05f5849d-fee0-4647-bbe4-e105949cdb2b">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="framework_upgrade::framework-upgrade">
            <failure message="API error: Unknown error Transaction committed on chain, but failed execution: Move abort in 0x1::gas_schedule: EINVALID_GAS_FEATURE_VERSION(0x10002): 

Stack backtrace:
   0: anyhow::error::&lt;impl core::convert::From&lt;E&gt; for anyhow::Error&gt;::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:564:25
   1: &lt;core::result::Result&lt;T,F&gt; as core::ops::try_trait::FromResidual&lt;core::result::Result&lt;core::convert::Infallible,E&gt;&gt;&gt;::from_residual
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:1964:27
   2: aptos_release_builder::validate::NetworkConfig::execute_proposal::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:342:9
   3: aptos_release_builder::validate::NetworkConfig::submit_and_execute_multi_step_proposal::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:116:64
   4: aptos_release_builder::validate::execute_release::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:412:22
   5: aptos_release_builder::validate::validate_config_and_generate_release::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:475:6
   6: aptos_release_builder::validate::validate_config::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:461:80
   7: &lt;aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:144:10
   8: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  17: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  18: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  19: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:428:24
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:331:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  24: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:381"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-15636-1734646021-6593fb81261f25490ff","timestamp":"2024-12-19T22:14:12.503098Z","message":"Deleting namespace forge-framework-upgrade-pr-15636: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:398"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-15636-1734646021-6593fb81261f25490ff","timestamp":"2024-12-19T22:14:12.503139Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-15636"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:428:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:331:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          5m1s
aptos-node-1-validator-0               1/1     Running     0          4m23s
aptos-node-2-validator-0               1/1     Running     0          7m3s
aptos-node-3-validator-0               1/1     Running     0          7m3s
forge-testnet-deployer-wdmf4           0/1     Completed   0          7m10s
genesis-aptos-genesis-eforge66-5hjkt   0/1     Completed   0          7m4s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite realistic_env_max_load success on f43e9a9f846de31364d52134284a1ccf24f2b945

two traffics test: inner traffic : committed: 14879.87 txn/s, latency: 2671.84 ms, (p50: 2700 ms, p70: 2700, p90: 2900 ms, p99: 3300 ms), latency samples: 5657680
two traffics test : committed: 99.97 txn/s, latency: 1362.93 ms, (p50: 1300 ms, p70: 1400, p90: 1500 ms, p99: 2500 ms), latency samples: 1760
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.599, avg: 1.550", "ConsensusProposalToOrdered: max: 0.293, avg: 0.288", "ConsensusOrderedToCommit: max: 0.310, avg: 0.296", "ConsensusProposalToCommit: max: 0.597, avg: 0.584"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.93s no progress at version 61624 (avg 0.19s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.51s no progress at version 2453894 (avg 0.51s) [limit 16].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 6593fb81261f25490ffddc2252a861c994234c2a ==> f43e9a9f846de31364d52134284a1ccf24f2b945

Compatibility test results for 6593fb81261f25490ffddc2252a861c994234c2a ==> f43e9a9f846de31364d52134284a1ccf24f2b945 (PR)
1. Check liveness of validators at old version: 6593fb81261f25490ffddc2252a861c994234c2a
compatibility::simple-validator-upgrade::liveness-check : committed: 14194.11 txn/s, latency: 2322.17 ms, (p50: 1800 ms, p70: 1900, p90: 2500 ms, p99: 24500 ms), latency samples: 491320
2. Upgrading first Validator to new version: f43e9a9f846de31364d52134284a1ccf24f2b945
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6947.32 txn/s, latency: 4280.43 ms, (p50: 4900 ms, p70: 5300, p90: 5400 ms, p99: 5600 ms), latency samples: 131600
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6910.16 txn/s, latency: 4958.84 ms, (p50: 5400 ms, p70: 5500, p90: 5600 ms, p99: 5700 ms), latency samples: 233300
3. Upgrading rest of first batch to new version: f43e9a9f846de31364d52134284a1ccf24f2b945
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7983.70 txn/s, latency: 3757.54 ms, (p50: 4300 ms, p70: 4400, p90: 4600 ms, p99: 4700 ms), latency samples: 148540
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 8115.83 txn/s, latency: 4122.41 ms, (p50: 4500 ms, p70: 4600, p90: 4700 ms, p99: 5000 ms), latency samples: 271880
4. upgrading second batch to new version: f43e9a9f846de31364d52134284a1ccf24f2b945
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 1556.11 txn/s, latency: 17826.76 ms, (p50: 19400 ms, p70: 25000, p90: 29200 ms, p99: 29900 ms), latency samples: 59220
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 12128.63 txn/s, latency: 2721.86 ms, (p50: 2800 ms, p70: 2900, p90: 3000 ms, p99: 3300 ms), latency samples: 394080
5. check swarm health
Compatibility test for 6593fb81261f25490ffddc2252a861c994234c2a ==> f43e9a9f846de31364d52134284a1ccf24f2b945 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking