I noticed there might be a way to systematically reproduce the error by doing a port scan on the machine in question. One of my machines was port scanned, possibly with nmap, over a larger number of ports (more than just services, ie > 1024). Upon discovering open ports 22, 80, 443, the scanner tried to
- log in via ssh with ssh1
- issued a large number of requests to nginx, which began with
"TRACE / HTTP/1.1" 405 157 "-" "Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)"
and then continued to a large number of things like
"\x16\x03\x01\x02\x00\x01\x00\x01\xFC\x03\x03\xF8n\xC2\xCCy\xB3O0\x1D\xA3\xE0h\xCBE\x1F\xE39d)\xE7\xF3\x9B\xA6W\xEFg0A=\xEE\xBAk \x9E8w\xC57\xB8\xEF\xCC\x01&\x92\xCE\xF9\x06\xDF\xDC\xCF\xC1t\xCFZN\xB1\xFD\xB0\x157\x91\xBF\x03y\x1F\x00\x9C\x13\x02\x13\x03\x13\x01\x003\x009\x005\x00/\xC0,\xC00\x00\xA3\x00\x9F\xCC\xA9\xCC\xA8\xCC\xAA\xC0\xAF\xC0\xAD\xC0\xA3\xC0\x9F\xC0]\xC0a\xC0W\xC0S\xC0+\xC0/\x00\xA2\x00\x9E\xC0\xAE\xC0\xAC\xC0\xA2\xC0\x9E\xC0\x5C\xC0`\xC0V\xC0R\xC0$\xC0(\x00k\x00j\xC0s\xC0w\x00\xC4\x00\xC3\xC0#\xC0'\x00g\x00@\xC0r\xC0v\x00\xBE\x00\xBD\xC0" 400 157 "-" "-"
Over one minute pf issued about 20 messages
"pfi_kif_unref: state refcount <= 0"
If such behavior indeed reproduces the message, it might help track down the bug.
Updated by