Bug #911

kldload/kernel linker can exceed malloc reserve and panic system

Added by corecode over 6 years ago. Updated about 1 month ago.

Status:FeedbackStart date:
Priority:NormalDue date:
Assignee:tuxillo% Done:

0%

Category:Kernel
Target version:3.8.0

Description

hey,

I just booted with hw.physmem=64m and got a panic when trying to load a module:

panic: kld: malloc limit exceeded

(kgdb) bt
#0 dumpsys () at thread.h:83
#1 0xc018450c in boot (howto=256) at /usr/build/src/sys/kern/kern_shutdown.c:375
#2 0xc0184661 in panic (fmt=Variable "fmt" is not available.
) at /usr/build/src/sys/kern/kern_shutdown.c:800
#3 0xc0182129 in kmalloc (size=78, type=0xc02f5600, flags=2)
at /usr/build/src/sys/kern/kern_slaballoc.c:445
#4 0xc01678b4 in linker_make_file (pathname=0xc641b000 "./nvidia.ko", priv=0xc63ea028,
ops=0xc02f5bc8) at /usr/build/src/sys/kern/kern_linker.c:369
#5 0xc016a773 in link_elf_load_module (filename=0xc641b000 "./nvidia.ko", result=0xc83efc7c)
at /usr/build/src/sys/kern/link_elf.c:604
#6 0xc01684e0 in linker_load_file (filename=0xc641b000 "./nvidia.ko", result=0xc83efca8)
at /usr/build/src/sys/kern/kern_linker.c:272
#7 0xc016871c in sys_kldload (uap=0xc83efcf0) at /usr/build/src/sys/kern/kern_linker.c:724

the problem seems to be that M_LINKER already used 10% of all memory (allegedly). In this case kmalloc() simply panics if passing M_WAITOK without M_NULLOK. This is quite unfortunate. Shouldn't we try to stay alive and print a warning and block, hoping that the problem will resolve itself?

cheers
simon

64MB_X64.jpg (8.83 KB) tuxillo, 02/24/2014 02:25 PM

History

#1 Updated by dillon over 6 years ago

:hey,
:
:I just booted with hw.physmem=64m and got a panic when trying to load a module:
:
:panic: kld: malloc limit exceeded
:
:#3 0xc0182129 in kmalloc (size=78, type=0xc02f5600, flags=2)
: at /usr/build/src/sys/kern/kern_slaballoc.c:445
:#4 0xc01678b4 in linker_make_file (pathname=0xc641b000 "./nvidia.ko", priv=0xc63ea028,
: ops=0xc02f5bc8) at /usr/build/src/sys/kern/kern_linker.c:369
:...
:
:the problem seems to be that M_LINKER already used 10% of all memory (allegedly). In this case kmalloc() simply panics if passing M_WAITOK without M_NULLOK. This is quite unfortunate. Shouldn't we try to stay alive and print a warning and block, hoping that the problem will resolve itself?
:
:cheers
: simon

That's an administrative limit, it sounds like the solution is to simply
assign a reasonable minimum to it.

Well, what's the malloc limit when you have 64MB of ram? From gdb
do 'print M_LINKER'. And... is the amount of memory it thinks it is
using roughly similar to the amount of memory the loaded modules ought
to be taking up?

-Matt

#2 Updated by corecode over 6 years ago

The limit is ~6MB, the linker consumes 8MB. I just realized that we
have in struct malloc_type:

__uint16_t ks_limblocks; /* number of times blocked for hitting
limit */

But we're never using this.

cheers
simon

#3 Updated by dillon over 6 years ago

:The limit is ~6MB, the linker consumes 8MB. I just realized that we
:have in struct malloc_type:
:
: __uint16_t ks_limblocks; /* number of times blocked for hitting
:limit */
:
:But we're never using this.
:
:cheers
: simon

It'l probably never be used. There is no callback facility for
recovering kmalloc'd memory. And, particularly for the linker,
there wouldn't be anything to free anyway so it would just
deadlock.

-Matt
Matthew Dillon
<>

#4 Updated by corecode over 6 years ago

Well, yea, but I'd rather have a blocking kldload than a panic'ed
system. There are probably more such places. Actually I think this
limit business is stupid. Rather than panicing, we could print a
warning and go on. The worst thing could be that we'll panic lateron
because we don't have free memory anymore.

cheers
simon

#5 Updated by dillon over 6 years ago

:Well, yea, but I'd rather have a blocking kldload than a panic'ed
:system. There are probably more such places. Actually I think this
:limit business is stupid. Rather than panicing, we could print a
:warning and go on. The worst thing could be that we'll panic lateron
:because we don't have free memory anymore.
:
:cheers
: simon

Well, there aren't too many places where M_LINKER is used, you could
adjust the code to pass M_NULLOK and deal with NULL returns.

We definitely do not want to block, since it will wind up blocking
forever.

Matthew Dillon
<>

#6 Updated by sepherosa over 6 years ago

On Jan 11, 2008 9:12 AM, Simon 'corecode' Schubert
<> wrote:
> Matthew Dillon wrote:
> > It'l probably never be used. There is no callback facility for
> > recovering kmalloc'd memory. And, particularly for the linker,
> > there wouldn't be anything to free anyway so it would just
> > deadlock.
>
> Well, yea, but I'd rather have a blocking kldload than a panic'ed
> system. There are probably more such places. Actually I think this
> limit business is stupid. Rather than panicing, we could print a
> warning and go on. The worst thing could be that we'll panic lateron
> because we don't have free memory anymore.

I don't think it's the problem of our M_WAITOK implementation. You
could change the caller to use other kmalloc flag so that the system
will not panic; how to properly handle kmalloc failure in this case
may be difficult.

Best Regards,
sephe

#7 Updated by corecode over 6 years ago

I realize this. However, we should avoid panicing if possible. So why
not print a warning ("allocated more than the limit") and continue? If
nothing happens, great! If we run out of memory, we still can panic later.

cheers
simon

#8 Updated by sepherosa over 6 years ago

On Jan 11, 2008 9:34 AM, Simon 'corecode' Schubert
<> wrote:
> Sepherosa Ziehau wrote:
> > I don't think it's the problem of our M_WAITOK implementation. You
> > could change the caller to use other kmalloc flag so that the system
> > will not panic; how to properly handle kmalloc failure in this case
> > may be difficult.
>
> I realize this. However, we should avoid panicing if possible. So why
> not print a warning ("allocated more than the limit") and continue? If

What do you mean by "continue" here?

Best Regards,
sephe

#9 Updated by corecode over 6 years ago

Just allocate some memory, even if you're over the limit. It's not that
we ran out of memory, it's just that we went over the (artificial) limit.

cheers
simon

#10 Updated by dillon over 6 years ago

:Just allocate some memory, even if you're over the limit. It's not that
:we ran out of memory, it's just that we went over the (artificial) limit.
:
:cheers
: simon

That seems reasonable insofar as M_LINKER goes. How about this: Add a
new malloc flag which tells it to ignore the administrative limit.
Instead of panicing malloc() would generate a rate-limited warning to
the console but continue with the allocation.

In most cases hitting the administrative resource limit should be fatal,
but clearly not in the M_LINKER case. A flag would be appropriate.

-Matt
Matthew Dillon
<>

#11 Updated by robgar about 1 month ago

  • Description updated (diff)
  • Status changed from New to Closed

#12 Updated by tuxillo about 1 month ago

  • Category set to Kernel
  • Status changed from Closed to New

#13 Updated by tuxillo about 1 month ago

  • Status changed from New to In Progress
  • Assignee changed from corecode to tuxillo
  • Target version set to 3.8.0

#14 Updated by tuxillo about 1 month ago

  • File 64MB_X64.jpg added
  • Status changed from In Progress to Feedback

Hi,

64MB is very low mem capacity nowadays, at least for the platforms we support.

Real kernels can't even boot with low mem profiles. For example, i386 can't boot on 32MB and x86_64 can't boot with 64MB when both have no swap configured (well x64 could but the login process will be killed upon execution, see attached screenshot).

For me this issue can be closed, but a similar one could be opened if in the future we support additional architectures that run on low mem configs.

Cheers,
Antonio Huete

Also available in: Atom PDF