daemon crashed with SIGABRT during XRouter service call

[walkjivefly]

daemon wallet crashed with SIGABRT, possibly while making an XRouter service call. No clues in the debug.log but a coredump was produced.

Core was generated by `blocknetd'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f55856197bb in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f54e9ffb700 (LWP 12765))]
(gdb) bt
#0  0x00007f55856197bb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f5585604535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x0000562c0a360bf1 in potential_deadlock_detected (mismatch={...}, 
    mismatch={...}, s2=..., s1=...) at sync.cpp:109
#3  push_lock (locklocation=..., c=<optimized out>) at sync.cpp:133
#4  EnterCritical (pszName=<optimized out>, pszFile=<optimized out>, 
    nLine=<optimized out>, cs=<optimized out>, fTry=fTry@entry=false)
    at sync.cpp:144
#5  0x0000562c0a3da5ce in UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::Enter (nLine=<optimized out>, 
    pszFile=<optimized out>, pszName=<optimized out>, this=0x7f54e9ff7220)
    at ./sync.h:122
#6  UniqueLock<AnnotatedMixin<std::recursive_mutex>, std::unique_lock<std::recursive_mutex> >::UniqueLock (this=0x7f54e9ff7220, mutexIn=..., 
    pszName=<optimized out>, pszFile=<optimized out>, nLine=<optimized out>, 
    fTry=<optimized out>) at ./sync.h:150
#7  0x0000562c0a9f4733 in xrouter::App::<lambda(CNode*)>::operator()(CNode *) const (__closure=<optimized out>, pnode=0x7f5504026970)
    at xrouter/xrouterapp.cpp:1921
#8  0x0000562c0a9f48c0 in CConnman::ForEachNode<xrouter::App::checkSnodeBan(const NodeAddr&, int)::<lambda(CNode*)> > (func=..., this=<optimized out>)
    at /usr/include/c++/8/bits/stl_iterator.h:783
#9  xrouter::App::checkSnodeBan (score=<optimized out>, 
    node="144.91.116.146:41412", this=<optimized out>)
    at xrouter/xrouterapp.cpp:1917
#10 xrouter::App::checkSnodeBan (this=<optimized out>, 
    node="144.91.116.146:41412", score=<optimized out>)
    at xrouter/xrouterapp.cpp:1914
#11 0x0000562c0a9f96b0 in xrouter::App::xrouterCall (
    this=0x562c0b0c6180 <xrouter::App::instance()::app>, 
    command=command@entry=xrouter::xrService, 
    uuidRet="0acd337a-52aa-4a99-ae6b-150eadd0fad4", 
    fqServiceName="cpfairprice", confirmations=@0x7f54e9ff9830: 0, params=...)
    at xrouter/xrouterapp.cpp:1371
#12 0x0000562c0a9c9f52 in xrService (request=...)
    at xrouter/rpcxrouter.cpp:1029
#13 0x0000562c0a5144f5 in CRPCTable::execute (this=<optimized out>, 
    request=...) at rpc/server.cpp:581
#14 0x0000562c0a62a4d3 in HTTPReq_JSONRPC (req=0x7f5508202310)
    at httprpc.cpp:192
#15 0x0000562c0a62b2a9 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1="", __args#0=<optimized out>, 
    this=0x7f5508078230) at /usr/include/c++/8/bits/std_function.h:682
#16 HTTPWorkItem::operator() (this=0x7f5508078200) at httpserver.cpp:54
#17 WorkQueue<HTTPClosure>::Run (this=0x562c0c914cc0) at httpserver.cpp:113
#18 HTTPWorkQueueRun (queue=0x562c0c914cc0) at httpserver.cpp:347
#19 0x00007f55859fdb2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#20 0x00007f5585adcfa3 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007f55856db4cf in clone () from /lib/x86_64-linux-gnu/libc.so.6

The backtrace is different than that in #604 although it ends up in the same place.

[walkjivefly]

Had a second crash shortly after restarting the daemon, with the same call stack but this time with some messages in the debug.log:

2022-01-26T08:26:10Z POTENTIAL DEADLOCK DETECTED
2022-01-26T08:26:10Z Previous lock order was:
2022-01-26T08:26:10Z  (1) cs_vNodes  ./net.h:209
2022-01-26T08:26:10Z  (2) cs_main  xrouter/xrouterapp.cpp:1921
2022-01-26T08:26:10Z Current lock order is:
2022-01-26T08:26:10Z  (2) cs_main  net_processing.cpp:3284
2022-01-26T08:26:10Z  (1) cs_vNodes  net.cpp:1649

[walkjivefly]

It also seems like there may be a memory leak because the coredump this time was 8GB. blocknetd coredumps in the past have generally been around 2GB.

What changed since yesterday? I'm running a lot more XCloud calls, on a timer. "Lot" is relative, 3 calls every 15 seconds is not really a lot, but in the past it was just a handful of ad-hoc calls each day.

[walkjivefly]

First crash today happened with wallet unlocked for staking, in the second it was locked so that's not a factor.

[walkjivefly]

And again, 12GB coredump this time.

[walkjivefly]

I wonder if the issue is related to attempting a number of calls in rapid succession? I turned down the frequency to 3 calls every 60 seconds prior to the last crash. Will do some more testing to explore this possibility.

[walkjivefly]

Crashed again after running for about 3.5 hours making a single XC service call every 20 seconds (717 calls).

[walkjivefly]

And again, after only 260 calls.