I have an python app as unix socket server, and several C++ apps as unix socket client. Most of time, it function well. Last week, they ran into an issue. For this issue, what I observed is like:
- ss shows several the remote socket's inode number as 0 and rxq number is a little high for one of the sockets:
# ss -x | grep -iE "7830816|111729901|112222563" u_str ESTAB 22928 0 /var/run/uds_sock 7830816 * 0 u_str ESTAB 0 0 /var/run/uds_sock 111729901 * 0 u_str ESTAB 0 0 /var/run/uds_sock 112222563 * 0
- RX thread in the server is stuck with calling recv(). Actually this is nonblocking socket.
The bt shows like:Thread 401873 (active): "receive_thread" recv_data (metricsReceiver/internal/client.py:160) Arguments: self: <ClientConns at 0x78aa5c1f7460> conn: <socket at 0x78aa567d6740> total_len: 753666 Locals: rcv_len: 255795 res: <bytes at 0x78a8c002bdd0> READ_LEN: 1000 count: 5 cur: <bytes at 0x78aa607d0030> recv (/metricsReceiver/internal/client.py:109) Arguments: self: <ClientConns at 0x78aa5c1f7460> conn: <socket at 0x78aa567d6740> Locals: msg: <ShaClientV2Msg at 0x78aa5cb71780> delimiter: <bytes at 0x78aa567c4120> length: 753666 _wait_for_data (metricsReceiver/internal/connection.py:115) Arguments: self: <Connection at 0x78aa5c30e0b0> Locals: trace: "Traceback (most recent call last):\n File "/metricsReceiver/internal/..." rlist: [96] elist: [] _: [] sock: 96 conn: <socket at 0x78aa567d6740> addr: "" run (threading.py:953) Arguments: self: <Thread at 0x78aa5c300a60> _bootstrap_inner (threading.py:1016) Arguments: self: <Thread at 0x78aa5c300a60> _bootstrap (threading.py:973) Arguments: self: <Thread at 0x78aa5c300a60>
The fd causing rx being stuck is one of those sockets' inode as 0:
# lsof -U | grep -iE "7830816|111729901|112222563"python3 401661 sha 96u unix 0xffff888107badd80 0t0 112222563 /var/run/uds_sock type=STREAM
I do not know what concretely happened cause these, and could not figure out if there is relations between the two phenomenons. Seeking some help here.
DEV os: Ubuntu 22.04.4 LTS
Thanks for your time in advance.