Bug #1274

hammer_ip_add_bulk reservation failed

Added by mneumann almost 6 years ago. Updated almost 6 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-

Description

Hi,

I got this error yesterday:

hammer_ip_add_bulk reservation failed

The message appeared every 1 second or so and pretty much slowed down
the whole system (it became unusable).

The system is 2.1.1-DEVELOPMENT as of 29th January from the snapshot
build.

Regards,

Michael

History

#1 Updated by dillon almost 6 years ago

:Hi,
:
:I got this error yesterday:
:
: hammer_ip_add_bulk reservation failed
:
:The message appeared every 1 second or so and pretty much slowed down
:the whole system (it became unusable).
:
:The system is 2.1.1-DEVELOPMENT as of 29th January from the snapshot
:build.
:
:Regards,
:
: Michael

If it happens again do a 'sysctl vfs.hammer' and a 'df' and save the
output, and get a kernel core.

How full was the HAMMER partition?

-Matt
Matthew Dillon
<>

#2 Updated by dillon almost 6 years ago

:91% full.
:
:Regards,
:
: Michael

That's pretty full. How large a partition is it?

-Matt
Matthew Dillon
<>

#3 Updated by mneumann almost 6 years ago

Am Mon, 9 Feb 2009 08:00:45 -0800 (PST)
schrieb Matthew Dillon <>:

91% full.

Regards,

Michael

#4 Updated by dillon almost 6 years ago

:Am Mon, 9 Feb 2009 08:08:18 -0800 (PST)
:schrieb Matthew Dillon <>:
:
:> :91% full.
:> :
:
:60 GB.

Ok, it shouldn't have generated that message with so much free
space. The only thing I can think of is that there is a leak
in the hammer_reservation structure. sysctl vfs.hammer.count_reservations
tells you how many structures are active.. it should usually be a low
number. I look over the code.

-Matt
Matthew Dillon
<>

#5 Updated by dillon almost 6 years ago

Whoop, I think I found it. You should be seeing a build-up
up reservations (vfs.hammer.count_reservations). There is a ref
count leak in the hammer_buffer code in the buffer invalidation path.
This is preventing reservations from being disposed of.

I will commit a fix today.

-Matt

#6 Updated by mneumann almost 6 years ago

Am Mon, 9 Feb 2009 08:58:53 -0800 (PST)
schrieb Matthew Dillon <>:

Yep, a "hammer reblock" triggered the error again.

# sysctl vfs.hammer.count_reservations
vfs.hammer.count_reservations: 2386
# df
"64 %"

But as you said you already found the bug, I don't generate a core dump
right now.

Thanks!

Regards,

Michael

#7 Updated by dillon almost 6 years ago

:Yep, a "hammer reblock" triggered the error again.
:
:# sysctl vfs.hammer.count_reservations
: vfs.hammer.count_reservations: 2386
:# df
: "64 %"
:
:But as you said you already found the bug, I don't generate a core dump
:right now.
:
:Thanks!
:
:Regards,
:
: Michael

Ok, I committed a fix, please test.

-Matt
Matthew Dillon
<>

#8 Updated by dillon almost 6 years ago

::# sysctl vfs.hammer.count_reservations
:: vfs.hammer.count_reservations: 2386
::# df
:: "64 %"
:: Michael
:
: Ok, I committed a fix, please test.
:

Oh, note that a large reservation count is only a bug if it nevers
goes back down. e.g. idle the system, sync a few times... if the
reservation count doesn't drop to near zero there is a problem.

-Matt
Matthew Dillon
<>

#9 Updated by corecode almost 6 years ago

committed fix in 4983f1f6fa80b8035dd5fac7d32c4e470fb5766a

Also available in: Atom PDF