Ok, so for the curious, a similar issue occured today near 9:30PM as well. And I found out that the system ran out of resources, because I used a `ssh xxx@x.x.x.x "cd x; exec 4</var/lock/x.lock; flock -x 4; echo x > x; kill -USR1 \$(cat x); flock -u 4; exec 4<&-;";` command in a script and it was called every few seconds. The ssh connection would be kept alive indefinitely if the remote lock can't be obtained. This resulted in thousands of ssh sessions being active at the same time which made the system run out of resources. Moral: at least use timeouts via "timeout x command args" on commands like ssh and scp when they are used in script if you want to save yourself from oom issues like this. However, why did the system reboot so many times and become a remote brick so quickly is still a mystery which I would love to resolve.
Statistics: Posted by adma03 — Wed Sep 04, 2024 10:25 pm