TiFlash crashes when multiple regions of the same table received concurrently #8217

JaySon-Huang · 2023-10-18T10:32:42Z

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/14745/pipeline/89

tiflash crashes with error logging as below

[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.568 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=439]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.570 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=278]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.580 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=279]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.581 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=280]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.583 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=546]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.585 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=547]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.587 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=318]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.588 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=328]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.589 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=621]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.591 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=672]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.592 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=673]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.594 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=356]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.595 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=374]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.597 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=429]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.598 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=384]
[2023-10-18T07:22:07.332Z] [2023/10/18 15:21:50.798 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is 
       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0xc72e3c6    DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
       0xc73af23    std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
       0xc72e542    DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
       0xc72e6e4    DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
       0xc9c6580    DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
       0xc9c7d37    DB::InterpreterCreateQuery::execute() [tiflash+211582263]
       0xdcf52d0    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
       0xdcf31e5    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
  0x7efeaa70fd6d    std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
  0x7efeaa6e7061    _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
  0x7efeaa70ff96    std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
  0x7efeaa72323b    __rust_try [libtiflash_proxy.so+166867515]
  0x7efeaa70febf    std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
  0x7efeaa70e371    std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=439]
[2023-10-18T07:22:07.332Z] [2023/10/18 15:21:50.799 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is 
       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0xc72e3c6    DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
       0xc73af23    std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
       0xc72e542    DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
       0xc72e6e4    DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
       0xc9c6580    DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
       0xc9c7d37    DB::InterpreterCreateQuery::execute() [tiflash+211582263]
       0xdcf52d0    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
       0xdcf31e5    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
  0x7efeaa70fd6d    std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
  0x7efeaa6e7061    _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
  0x7efeaa70ff96    std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
  0x7efeaa72323b    __rust_try [libtiflash_proxy.so+166867515]
  0x7efeaa70febf    std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
  0x7efeaa70e371    std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=278]
[2023-10-18T07:22:07.333Z] [2023/10/18 15:21:50.799 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is 
       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0xc72e3c6    DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
       0xc73af23    std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
       0xc72e542    DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
       0xc72e6e4    DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
       0xc9c6580    DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
       0xc9c7d37    DB::InterpreterCreateQuery::execute() [tiflash+211582263]
       0xdcf52d0    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
       0xdcf31e5    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
  0x7efeaa70fd6d    std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
  0x7efeaa6e7061    _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
  0x7efeaa70ff96    std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
  0x7efeaa72323b    __rust_try [libtiflash_proxy.so+166867515]
  0x7efeaa70febf    std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
  0x7efeaa70e371    std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=279]
[2023-10-18T07:22:07.334Z] [2023/10/18 15:21:50.800 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=604, index=7, term=6), e.what() = DB::Exception, Stack trace:


       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0x41cd3ba    DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
       0x41c921e    DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
       0xc6e72dc    DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
       0xc6dbb8b    DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
       0xc6db91b    void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
       0xc6db61e    std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
       0xc6db444    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
       0xc6da9e7    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
       0xc6d688f    DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
       0xc70342c    DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
       0xc6b2be9    DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
       0xd4eca4f    DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
       0xd4ed475    DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
       0xdcf3799    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=547]
[2023-10-18T07:22:07.334Z] [2023/10/18 15:21:50.800 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=582, index=7, term=6), e.what() = DB::Exception, Stack trace:


       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0x41cd3ba    DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
       0x41c921e    DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
       0xc6e72dc    DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
       0xc6dbb8b    DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
       0xc6db91b    void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
       0xc6db61e    std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
       0xc6db444    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
       0xc6da9e7    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
       0xc6d688f    DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
       0xc70342c    DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
       0xc6b2be9    DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
       0xd4eca4f    DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
       0xd4ed475    DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
       0xdcf3799    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=328]
[2023-10-18T07:22:07.335Z] [2023/10/18 15:21:50.801 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=602, index=7, term=6), e.what() = DB::Exception, Stack trace:


       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0x41cd3ba    DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
       0x41c921e    DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
       0xc6e72dc    DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
       0xc6dbb8b    DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
       0xc6db91b    void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
       0xc6db61e    std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
       0xc6db444    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
       0xc6da9e7    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
       0xc6d688f    DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
       0xc70342c    DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
       0xc6b2be9    DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
       0xd4eca4f    DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
       0xd4ed475    DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
       0xdcf3799    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=318]
[2023-10-18T07:22:07.336Z] [2023/10/18 15:21:50.808 +08:00] [ERROR] [BaseDaemon.cpp:370] [########################################] [source=BaseDaemon] [thread_id=762]

4. What is your TiFlash version? (Required)

d8e8eff

The text was updated successfully, but these errors were encountered:

…ously (#8229) close #8217

…ously (pingcap#8229) close pingcap#8217

…ously (#8229) (#8238) close #8217

JaySon-Huang added type/bug The issue is confirmed as a bug. severity/major component/storage affects-7.5 This bug affects the 7.5.x(LTS) versions. labels Oct 18, 2023

ti-chi-bot bot added may-affects-5.3 may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Oct 18, 2023

JaySon-Huang removed may-affects-5.3 may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Oct 18, 2023

hongyunyan mentioned this issue Oct 23, 2023

ddl: Fix corner case when multiple region creates one table simultaneously #8229

Merged

12 tasks

ti-chi-bot bot closed this as completed in #8229 Oct 25, 2023

ti-chi-bot bot pushed a commit that referenced this issue Oct 25, 2023

ddl: Fix corner case when multiple region creates one table simultane…

d4edc74

…ously (#8229) close #8217

ti-chi-bot mentioned this issue Oct 25, 2023

ddl: Fix corner case when multiple region creates one table simultaneously (#8229) #8238

Merged

12 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this issue Oct 25, 2023

ddl: Fix corner case when multiple region creates one table simultane…

aa903d4

…ously (pingcap#8229) close pingcap#8217

ti-chi-bot bot pushed a commit that referenced this issue Oct 27, 2023

ddl: Fix corner case when multiple region creates one table simultane…

b6b2d1e

…ously (#8229) (#8238) close #8217

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TiFlash crashes when multiple regions of the same table received concurrently #8217

TiFlash crashes when multiple regions of the same table received concurrently #8217

JaySon-Huang commented Oct 18, 2023

TiFlash crashes when multiple regions of the same table received concurrently #8217

TiFlash crashes when multiple regions of the same table received concurrently #8217

Comments

JaySon-Huang commented Oct 18, 2023

Bug Report

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)