Bug #2118

swap filled on machine with heavy activity+swapcache, swap doesn't drain

Added by ftigeot almost 3 years ago. Updated almost 3 years ago.

Status:ClosedStart date:
Priority:UrgentDue date:
Assignee:dillon% Done:

0%

Category:-
Target version:-

Description

pkgbox64.dragonflybsd.org uses 64GB swapcache + 8GB RAM

When under heavy load (Opengrok indexing, Hammer daily cleanup and bulk build
running at the same time), its swap filled entirely

This kernel message was logged on the console:
swap_pager_getswapspace: failed

And the system behaved erratically:
dmesg: sysctl kern.msgbuf: Cannot allocate memory

Killing processes had no lasting effect; as soon as a little bit of swap
emptied, it was filled again.

Even though swap was 99% filled and vm.swapcache.maxswappct much lower, the
kernel still tried to add data to swap.
Reducing vm.swapcache.maxswappct further (40%, 20%, etc...) had no effect.

The only way to force swap to drain was to set vm.swapcache.maxswappct to 0

History

#1 Updated by dillon almost 3 years ago

:
:When under heavy load (Opengrok indexing, Hammer daily cleanup and bulk build
:running at the same time), its swap filled entirely
:
:This kernel message was logged on the console:
: swap_pager_getswapspace: failed

Ok, I believe I have fixed the problem. fork() was causing a
vm_object leak where old vm_object's were not being properly terminated,
leaving vm_page's intact and slowly causing swap to fill up. pkgbox64
is now running a new kernel.

-Matt
Matthew Dillon
<>

#2 Updated by y0n3t4n1 almost 3 years ago

On 2011/8/21 05:43 AM, Matthew Dillon wrote:
> :
> :When under heavy load (Opengrok indexing, Hammer daily cleanup and bulk build
> :running at the same time), its swap filled entirely
> :
> :This kernel message was logged on the console:
> : swap_pager_getswapspace: failed
>
> Ok, I believe I have fixed the problem. fork() was causing a
> vm_object leak where old vm_object's were not being properly terminated,
> leaving vm_page's intact and slowly causing swap to fill up. pkgbox64
> is now running a new kernel.

The fix (1f8fc82a) seems to fix my problem I was talking about
on kernel@ the other day, so I jumped in here :), but I started
seeing this occasional warning message on the console under load:

vm_prefault: Warning, backing object race averted lobject 0xffffffe062f5bf20

I also caught a panic in vm_object_deallocate_locked() afterwards:

#15 0xffffffff8052df79 in vm_object_deallocate_locked (
object=0xffffffe06298f420) at /usr/src/sys/vm/vm_object.c:601
601 LIST_REMOVE(object, shadow_list);
(kgdb) l
596 if (temp == object->backing_object)
597 break;
598 vm_object_unlock(temp);
599 }
600 if (temp) {
601 LIST_REMOVE(object, shadow_list);
602 temp->shadow_count--;
603 temp->generation++;
604 object->backing_object = NULL;
605 vm_object_lock_swap();

#3 Updated by ftigeot almost 3 years ago

For my part, I consider the issue resolved

The machine memory consumption has not grown unreasonably even though it has
been under moderate to heavy I/O load for a few days

Swap usage stays just under the vm.swapcache.maxswappct limit

Thanks for the prompt fix!

Also available in: Atom PDF