Bugcheck when replicator state is changed concurrently

[hvlad]

Crashed thread stack

#0 0x00007f216046058c in __pthread_kill_implementation () from /lib64/libc.so.6
#1 0x00007f2160413d06 in raise () from /lib64/libc.so.6
#2 0x00007f21603e77f3 in abort () from /lib64/libc.so.6
#3 0x00007f21518248c6 in fb_utils::logAndDie (text=0x7f0f3fe1ef10 "Fatal lock manager error: invalid lock id (0), errno: 2\n--No such file or directory") at /usr/home/firebird/v5.0-release/src/common/utils.cpp:1687
#4 0x00007f2151738bb5 in Jrd::LockManager::bug (this=0x7f214e8b4d70, statusVector=0x0, string=0x7f0f3fe20f50 "invalid lock id (0)") at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:1639
#5 0x00007f2151738ead in Jrd::LockManager::get_request (this=this@entry=0x7f214e8b4d70, offset=offset@entry=0) at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:2110
#6 0x00007f215173ab3d in Jrd::LockManager::dequeue (this=0x7f214e8b4d70, request_offset=0) at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:715
#7 0x00007f2151487361 in DEQUEUE (lock=0x7f1e77201990, tdbb=0x7f0f3fe21598) at /usr/home/firebird/v5.0-release/src/jrd/lck.cpp:169
#8 LCK_release (tdbb=0x7f0f3fe21598, lock=0x7f1e77201990) at /usr/home/firebird/v5.0-release/src/jrd/lck.cpp:807
#9 0x00007f215129c7e9 in Jrd::Attachment::invalidateReplSet (this=, tdbb=, broadcast=broadcast@entry=true) at /usr/home/firebird/v5.0-release/src/jrd/Attachment.cpp:1158
#10 0x00007f21513e58a9 in change_repl_state (tdbb=, phase=, work=0x7f132766d000) at /usr/home/firebird/v5.0-release/temp/Release/jrd/dfw.cpp:11597
#11 0x00007f21513fc6df in DFW_perform_work (tdbb=tdbb@entry=0x7f0f3fe21598, transaction=transaction@entry=0x7f1cdac1b0c0) at /usr/home/firebird/v5.0-release/temp/Release/jrd/dfw.cpp:5016
#12 0x00007f21514ef66b in TRA_commit (tdbb=0x7f0f3fe21598, transaction=0x7f1cdac1b0c0, retaining_flag=) at /usr/home/firebird/v5.0-release/src/jrd/tra.cpp:487

Concurrent thread stack

#0 0x00007f216045b570 in __lll_lock_wait () from /lib64/libc.so.6
#1 0x00007f2160461cfd in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6
#2 0x00007f215173b8a6 in Firebird::Mutex::enter (aReason=0x0, this=) at /usr/home/firebird/v5.0-release/src/include/../common/classes/locks.h:209
#3 Jrd::LockManager::LockTableCheckout::~LockTableCheckout (this=0x7f0ff0a83ca0, __in_chrg=) at /usr/home/firebird/v5.0-release/src/lock/../lock/lock_proto.h:366
#4 Jrd::LockManager::blocking_action (this=0x7f214e8b4d70, tdbb=0x7f0ff0a84598, blocking_owner_offset=17179912) at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:1399
#5 0x00007f215173bd12 in Jrd::LockManager::signal_owner (this=this@entry=0x7f214e8b4d70, tdbb=tdbb@entry=0x7f0ff0a84598, blocking_owner=blocking_owner@entry=0x7f12a4c01508) at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:3205
#6 0x00007f215173bf95 in Jrd::LockManager::post_blockage (this=0x7f214e8b4d70, tdbb=0x7f0ff0a84598, request=, lock=) at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:2676
#7 0x00007f215173c20d in Jrd::LockManager::wait_for_request (this=0x7f214e8b4d70, tdbb=0x7f0ff0a84598, request=0x7f12a4565750, lck_wait=) at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:3766
#8 0x00007f215173ce22 in Jrd::LockManager::internal_convert (this=this@entry=0x7f214e8b4d70, tdbb=tdbb@entry=0x7f0ff0a84598, statusVector=statusVector@entry=0x7f0ff0a840e0, request_offset=request_offset@entry=10250064, type=type@entry=6 '\006', lck_wait=lck_wait@entry=1, ast_routine=0x7f215129db70 Jrd::Attachment::blockingAstReplSet(void*), ast_argument=0x7f16916d5b40) at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:2523
#9 0x00007f215173cf8c in Jrd::LockManager::convert (this=0x7f214e8b4d70, tdbb=tdbb@entry=0x7f0ff0a84598, statusVector=statusVector@entry=0x7f0ff0a840e0, request_offset=10250064, type=type@entry=6 '\006', lck_wait=lck_wait@entry=1, ast_routine=0x7f215129db70 Jrd::Attachment::blockingAstReplSet(void*), ast_argument=0x7f16916d5b40) at /usr/home/firebird/v5.0-release/src/lock/lock.cpp:625
#10 0x00007f21514866e5 in CONVERT (wait=1, level=6, lock=0x7f1fc6aee990, statusVector=0x7f0ff0a840e0, tdbb=0x7f0ff0a84598) at /usr/home/firebird/v5.0-release/src/jrd/lck.cpp:158
#11 LCK_convert (tdbb=0x7f0ff0a84598, lock=0x7f1fc6aee990, level=, wait=) at /usr/home/firebird/v5.0-release/src/jrd/lck.cpp:346
#12 0x00007f215129c7f5 in Jrd::Attachment::invalidateReplSet (this=0x7f16916d5b40, tdbb=0x7f0ff0a84598, broadcast=broadcast@entry=true) at /usr/home/firebird/v5.0-release/src/jrd/Attachment.cpp:1155
#13 0x00007f21513e58a9 in change_repl_state (tdbb=, phase=, work=0x7f1810d722a0) at /usr/home/firebird/v5.0-release/temp/Release/jrd/dfw.cpp:11597
#14 0x00007f21513fc6df in DFW_perform_work (tdbb=tdbb@entry=0x7f0ff0a84598, transaction=transaction@entry=0x7f1c3ba79ec0) at /usr/home/firebird/v5.0-release/temp/Release/jrd/dfw.cpp:5016

[hvlad]

The bug was reported privately (FB5, Linux), no more details is known.

[hvlad]

Looks like att_repl_lock could be double released in Attachment::invalidateReplSet()