Bug #3032

IPFW3: memory leakage? objcache(xxx): Exhausted!

Added by liweitianux 3 months ago. Updated about 1 month ago.

Target version:
Start date:
Due date:
% Done:



Recently, I setup a small VPS (512MB RAM) with DFly(v4.8)+Nginx+Postfix+Dovecot services as my personal email server, and configured IPFW3 as the firewall. However, the system hang/freeze with these warnings after some time, and can only be reset:

Warning, objcache(mbuf pkt hdr): Exhausted!
Warning, objcache(mbuf pkt hdr + cluster): Exhausted!

The VPS can run rather smoothly without IPFW3 enabled. So I tried to monitor the mbuf usages reported by "netstat -m", and found the "mbufs in use" was continuously increasing, and therefore, the mbufs are exhausted finally which freeze the whole system!!


In addition, I tried to reproduce the IPFW3 problem in my home NAS (DFly master@2017-02-24, 8GB RAM, Nginx+ownCloud+PostgreSQL+transmission bt), and the system just paniced with following messages:

stack pointer = 0x10:0xffffff81db763720
frame pointer = 0x10:0xffffff81db763770
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = Idle
current thread = pri 12
trap number = 9
panic: general protection fault
cpuid = 2
Trace beginning at frame 0xffffff81db7634e8
panic() at panic+0x236 0xffffffff805e5d56
panic() at panic+0x236 0xffffffff805e5d56
trap_fatal() at trap_fatal+0x2c3 0xffffffff80a58203
trap() at trap+0x19a 0xffffffff80a58a8a
calltrap() at calltrap+0x9 0xffffffff80a4125f
--- trap 0000000000000009, rip = ffffffff84840dd4, rsp = ffffff81db763710, rbp = ffffff81db763770 ---
ipfw_sync_send_state() at ipfw_sync_send_state+0x54 0xffffffff84840dd4
check_keep_state() at check_keep_state+0x196 0xffffffff84821d76
boot() called on cpu#2
Uptime: 12d1h18m34s
Physical memory: 8113 MB
Dumping 3889 MB:Warning, objcache(cluster mbuf): Exhausted!

For unknown reason, I failed to get a core dump. I set "debug.debugger_on_panic=0" and have a 16GB swap partition (dm_crypt).

An example "netstat -m" report looks like:
202336/293376 mbufs in use (current/max):
522/17248 mbuf clusters in use (current/max)
0/8368 mbuf jumbo clusters in use (current/max)
202839 mbufs and mbuf clusters allocated to data
19 mbufs and mbuf clusters allocated to packet headers
102212 Kbytes allocated to network (56% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

With IPFW3 enabled, the above "mbufs in use" keeps increasing! Flushing the firewall rules does not help either.
I also tried to *disable* the IPFW3 by setting "net.inet.ip.fw3.enable=1", and the mbuf usages *stop increase* (but don't decrease either).

Therefore, I suspect there may be *memory leakages* within IPFW3??

What's additional information can I provide to help solve this problem?


P.S., my IPFW3 rules (example output of "ipfw3 show"):
00010 0 0 allow via lo0
00100 14446903 15349529336 check-state
00200 168513 106435518 allow tcp from me out via em0 keep-state
00201 5853336 6205262652 allow udp from me out via em0 keep-state
00202 1 56 allow icmp from me out via em0 keep-state
00301 3 168 deny from in via em0
00303 0 0 deny from in via em0
00304 3 984 deny from in via em0
00305 1 328 deny from in via em0
00306 0 0 deny from in via em0
00307 0 0 deny from in via em0
00308 0 0 deny from in via em0
00310 2012 239927 allow icmp in via em0 keep-state
00315 0 0 deny tcp dst-port 113 in via em0
00320 0 0 deny tcp dst-port 137 in via em0
00321 0 0 deny tcp dst-port 138 in via em0
00322 0 0 deny tcp dst-port 139 in via em0
00323 0 0 deny tcp dst-port 81 in via em0
00332 7033 1814028 deny tcp established in via em0
00500 68 11257 allow tcp dst-port 8860 in via em0 keep-state
00510 0 0 allow tcp dst-port 80 in via em0 keep-state
00512 0 0 allow tcp dst-port 8800 in via em0 keep-state
00513 9281 3765242 allow tcp dst-port 8801 in via em0 keep-state
00700 0 0 allow tcp dst-port 22000 in via em0 keep-state
00701 737 86229 allow udp dst-port 21027 in via em0 keep-state
00800 8563201 9051221125 allow dst-port 51413 in via em0 keep-state
00801 515 263178 allow tcp from dst-port 9091 in via em0 keep-state
60000 1253 185397 deny in via em0
65535 414 59978 deny


#1 Updated by liweitianux about 1 month ago

  • Status changed from New to Resolved

There is a memory leakage bug with IPFW3 sync feature, which keeps consuming the mbuf, and finally causes this "objcache(xxx): Exhausted" issue.

The bug has been fixed by bycn82@ in commit 2032f72fa4bc9355fef4594376299c9aeb707530 on 2017-06-10. I already did some tests and things are OK.

Generally speaking, the IPFW3 "sync" shouldn't be used (which is also useless) on a single isolated host. But IPFW3 originally kept complaining that "ipfw3sync: centre not running", so I hacked to get rid of these annoying messages by "ipfw3 sync start centre". Now, this complaining issue has also been fixed.

Also available in: Atom PDF