Project

General

Profile

Actions
Copy link

Bug #2118

closed

swap filled on machine with heavy activity+swapcache, swap doesn't drain

Added by ftigeot over 12 years ago. Updated over 12 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
dillon
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:

Description

pkgbox64.dragonflybsd.org uses 64GB swapcache + 8GB RAM

When under heavy load (Opengrok indexing, Hammer daily cleanup and bulk build
running at the same time), its swap filled entirely

This kernel message was logged on the console:
swap_pager_getswapspace: failed

And the system behaved erratically:
dmesg: sysctl kern.msgbuf: Cannot allocate memory

Killing processes had no lasting effect; as soon as a little bit of swap
emptied, it was filled again.

Even though swap was 99% filled and vm.swapcache.maxswappct much lower, the
kernel still tried to add data to swap.
Reducing vm.swapcache.maxswappct further (40%, 20%, etc...) had no effect.

The only way to force swap to drain was to set vm.swapcache.maxswappct to 0

Actions
Copy link
 #1

Updated by dillon over 12 years ago

:
:When under heavy load (Opengrok indexing, Hammer daily cleanup and bulk build
:running at the same time), its swap filled entirely
:
:This kernel message was logged on the console:
: swap_pager_getswapspace: failed

Ok, I believe I have fixed the problem.  fork() was causing a
    vm_object leak where old vm_object's were not being properly terminated,
    leaving vm_page's intact and slowly causing swap to fill up.  pkgbox64
    is now running a new kernel.
-Matt
                    Matthew Dillon 
                    <>
Actions
Copy link
 #2

Updated by y0n3t4n1 over 12 years ago

On 2011/8/21 05:43 AM, Matthew Dillon wrote:

:
:When under heavy load (Opengrok indexing, Hammer daily cleanup and bulk build
:running at the same time), its swap filled entirely
:
:This kernel message was logged on the console:
: swap_pager_getswapspace: failed

Ok, I believe I have fixed the problem. fork() was causing a
vm_object leak where old vm_object's were not being properly terminated,
leaving vm_page's intact and slowly causing swap to fill up. pkgbox64
is now running a new kernel.

The fix (1f8fc82a) seems to fix my problem I was talking about
on kernel@ the other day, so I jumped in here :), but I started
seeing this occasional warning message on the console under load:

vm_prefault: Warning, backing object race averted lobject 0xffffffe062f5bf20

I also caught a panic in vm_object_deallocate_locked() afterwards:

#15 0xffffffff8052df79 in vm_object_deallocate_locked (
object=0xffffffe06298f420) at /usr/src/sys/vm/vm_object.c:601
601 LIST_REMOVE(object, shadow_list);
(kgdb) l
596 if (temp == object->backing_object)
597 break;
598 vm_object_unlock(temp);
599 }
600 if (temp) {
601 LIST_REMOVE(object, shadow_list);
602 temp->shadow_count--;
603 temp->generation++;
604 object->backing_object = NULL;
605 vm_object_lock_swap();

Actions
Copy link
 #3

Updated by ftigeot over 12 years ago

For my part, I consider the issue resolved

The machine memory consumption has not grown unreasonably even though it has
been under moderate to heavy I/O load for a few days

Swap usage stays just under the vm.swapcache.maxswappct limit

Thanks for the prompt fix!

Actions
Copy link

Also available in: Atom PDF