DragonFlyBSD

:Some known issues:
:- a warning upon reboot which looks like this:
:
: AcpiOsUnmapMemory: Warning, broken ACPI, bad unmap: 0xc5e1ee00/00000040
:
: IIRC, I've never seen this before updating my patch from 20060707 to
: 20060912, so I may have done something stupid while dealing with the
: changes with regard to Table Manager. I'm investigating this.

I added sanity checks to our AcpiOsMapMemory() and AcpiOsUnmapMemory()
    procedures.  The code tracks all requests and matches unmaps up against
    prior mappings.

:- hand-rolled locking code in AcpiOs{Acquire,Release}Lock() functions:
: ACPI-CA code has been rewritten in 20060623 to make these functions
: to be used as spinlock functions, and called from many places where
: they weren't before. Since our implementation of these locking functions
: used lockmgr lock, which cannot be called from cpu_idle(), this led to
: a panic(mainly when my laptop wakes up from sleep state). After struggling
: with other locking primitives, I ended up with critical section and
: special-cased the idlethread. I believed this shouldn't make the situation
: worse, as ACPI functions called from cpu_idle_hook code in acpi_cpu did
: not using locking before. But I'm open to a better implementation,
: especially how to deal with locking when called from idlethread.
:
:- new ACPI-CA code may be released soon, as 20060912 is more than 2-months
: old now
:
: absolutely :)
:
:Regards.

You could probably use tokens here, but a critical section ought to
    work as well since ACPI functions are only called from cpu #0.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

Hi,
testing it on my Sony Vaio VGN-S4XP hangs after:

npx0: <math processor> on motherboard
npx0: INT 16 interface
Using XMM optimized bcopy/copyin/copyout
acpi0: <Sony> on motherboard

Seems that there is just acpi_toshiba maybe we do also need acpi_sony?

Is this from boot -v? If you set debug.acpi.disabled in the boot loader,
does it still hang? The possible value may be "ec", as acpi_ec subsystem
doesn't probe on my laptop (other possible values are explained in acpi(4)).

If above doesn't work, I'd like you to build acpi driver with debugging
support and activate it to see where it's hung.
$ cd /sys/dev/acpica5
$ export MAKEOBJDIRPREFIX=$HOME/obj ACPI_DEBUG=yes ACPI_DEBUG_LOCKS=yes
$ make cleandir; make cleandir
$ make obj && make depend && make
$ su # make install && reboot
to activate the feature, you need to set two kernel environment variables
debug.acpi.layer="ACPI_ALL_DRIVERS"
debug.acpi.level="ACPI_LV_VERBOSE"
in the boot loader prompt. You may want to adjust the layer and the level
to trim down the number of debugging output (see acpi(4) for detail).

Basically these modules are optional and the acpi driver itself should
work without them.

Thanks for your time.

Ok, I shouldn't unmap the tables returned by AcpiGetTableByIndex, as well
as those by AcpiGetTable. Fixed it as
http://lead.dragonflybsd.org/~y0netan1/acpi-20060912-15.diff

Does that apply to ioctl or sysctl support code, too? Can you shed
some light on me to find how it's guaranteed?

Thanks.

First, why don't you use spinlocks then (note: I didn't read the code)?

To just use a critical section, you'd have to migrate the calling thread to cpu0, or have a acpi thread running on cpu0, and send a synchronous message to this thread, processing your request.

How are the guarantees the ACPI code gives? Will it ever call a blocking function (kmalloc M_WAITOK, etc) while holding the lock? Does it expect the lock to persist? If yes, that's ugly and you'd have to use lockmgr locks and an own thread.

Why are the ACPI functions called from the idle thread, anyways?

cheers
simon

^^{^^}^^^
actually this was 20060608

Because it led to a panic in lwkt_switch() when the laptop woke up
from sleep state:
lwkt_switch: still holding 1 exclusive spinlocks

But that was months ago when I tried to replace lockmgr() calls
in AcpiOs{Acquire,Release}Lock() functions with spinlocks, so I need
to revisit that idea before drawing a conclusion.

I couldn't find an explicit requirement for the locking functions
except for the description in changes.txt(look for "20060608:")
http://developer.intel.com/technology/iapc/acpi/downloads/changes.txt
As far as I read the code, the locks won't persist in most places,
except for the line 1753 in acpi_SetSleepState() (in unpatched code).

acpi_cpu subsystem installs acpi_cpu_idle() as cpu_idle_hook() function.
acpi_cpu_idle() calls some of low-level ACPI functions named AcpiHw*().
These functions are needed to be called to transition to C2 or deeper state.
And either AcpiHwRegister{Read,Write}() or their caller calls
AcpiOsAcquireLock() to lock AcpiGbl_HardwareLock. That was why lockmgr
lock can't be used any longer. I'm not sure if acpi_cpu_idle() can be
moved to its own thread, but maybe worth a try, as it can solve these
nasty things.

Thanks for suggestions.

:Does that apply to ioctl or sysctl support code, too? Can you shed
:some light on me to find how it's guaranteed?
:
:Thanks.

The ACPI idle code is disabled on SMP boxes.   Line 838 of acpi_cpu.c.

Hmm.  I don't see any specific code to force us onto cpu #0 for ioctls
    or sysctls, but those do certainly run under the BGL.  It would be
    fairly easy to force those to run on cpu #0 if it turned out we needed
    to.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

This was not from boot -v.

set debug.acpi.disabled="ec" on loader prompt doesn't help.

This laptop doesn't have serial port and hangs before probing syscons,
so i can't see all the messages it gives.
I copied by hand the ones i can see from boot -v with the options
you suggested:

ACPI: FACS 0x0x3feadfc0/0x0040
ACPI: APIC 0x0x3fe9cefc/0x0068 ( v 1 Sony V0 0x20050523 PTL 0x00000050)
ACPI: BOOT 0x0x3fe9cfd8/0x0028 ( v 1 Sony V0 0x20050523 PTL 0x00000001)
ACPI: MCFG 0x0x3fe9cf9c/0x003c ( v 1 Sony V0 0x20050523 PTL 0x0000005f)
ACPI: SSDT 0x0x3fe983aa/0x064f ( v 1 Sony V0 0x20050523 PTL 0x20030224)
ACPI: SSDT 0x0x3fe97d0e/0x069c ( v 1 Sony V0 0x20050523 PTL 0x20030224)
ACPI: SSDT 0x0x3fe978c9/0x0277 ( v 1 Sony V0 0x20050523 PTL 0x20030224)
ACPI: SSDT 0x0x3fe976aa/0x021f ( v 1 Sony V0 0x20050523 PTL 0x20030224)
ACPI: SSDT @ 0x0x3fe97491/0x0219 ( v 1 Sony V0 0x20050523 PTL 0x20030224)
npx0.nexus0.root0
npx0: <math processor> [tentative] on motherboard
npx0: INT 16 interface
Using XMM optimized bcopy/copyin/copyout
npx0: <math processor> [attached!] on motherboard
acpi0.nexus0.root0
acpi0: <Sony> [tentative] on motherboard

Btw, in case it may help here¹ is the boot -v with old ACPI version

[1]: http://bsdes.net/~victor/dfbsd/dmesg.verbose
--
La prueba más fehaciente de que existe vida inteligente en otros
planetas, es que no han intentado contactar con nosotros.

Oh, I forgot asking wheather ctrl+alt+esc works when it's stuck. If it
works, type in "trace" command, and let me know the list of function names
(you don't need the parameter list yet).

No, the last few to several lines of the console message are probably
enough to determine where it's stuck.

[dmesg output]

I'm afraid ACPI_LV_VERBOSE in debug.acpi.level wasn't enough to give us
extra information. Can you use the following two lines instead of the
ones I wrote in the previous message?
debug.acpi.layer="ACPI_TABLES ACPI_EVENTS ACPI_NAMESPACE"
debug.acpi.level="ACPI_LV_FUNCTIONS ACPI_LV_INFO"

Judging from the dmesg output above and the one from older module, I think
it's in the middle of AcpiInitializeTables(); or one of calls to
AcpiInstallAddressSpaceHandler(). The "acpi_bus_number:" messages are
displayed during the third call to it.

Thanks.

Ok, I saw a similar lock up on a machine I haven't updated since October --
before troubling your hand by writing down many function names, can you
try rebuilding your acpi driver with an environment variable
ACPI_USE_LOCAL_CACHE=yes ?
$ /sys/dev/acpica5
$ export ACPI_USE_LOCAL_CACHE=yes MAKEOBJDIRPREFIX=$HOME/obj
$ make cleandir; make cleandir
$ make obj && make depend && make && su root -c 'make install && reboot'

This helps! now i can go up to kern.seedenable: 0 -> 1, and type
spaces there. If i try to do C-c it hangs again.

This time i can enter at the debugger before pressing C-c, so if you
want that i try something from there just ask. A trace will not help
because you would get just the keyboard interrupt, not the acpi code.

Entering on the debugger and going out of it made the system boot.

Also I noticed this warnings that may be important:

ACPI Warning (evmisc-0603): Cannont release the ACPI Global Lock, it has not been
acquired [20060912]
ACPI Exception (exutils-0425): AE_NOT_ACQUIRED, Could not release ACPI Global Lock [20060912]
ACPI Exception (exutils-0382): AE_TIME, Could not acquire Global Lock [20060912]

You can find the (non-verbose) dmesg with new acpi here¹.

http://bsdes.net/~victor/dfbsd/dmesg.newacpi

If its getting to the actual RC scripts, then try ^T to see what
process is running when it gets stuck.

-Matt

That's just after printing
kern.seedenable: 0 -> 1

load: 0.27 cmd: sysctl 23 [acsem] 0.00u 0.01s 0% 276k
load: 0.08 cmd: dd 20 [piperd] 0.00u 0.05s 0% 164k

Just noticed that after waiting some time i can do C-c without
hanging the system.

You can drop in the boot prompt and type
unset acpi_load
boot
to avoid loading ACPI module(unless acpi_load="YES" is written in
/boot/loader.conf, in which case you have to type `unload' command first).
If it still locks up without loading ACPI, then I'd say it's nothing to do
with ACPI driver.

That's either in /etc/rc.d/random or /etc/rc.d/initrandom.

This laptop doesn't boot without acpi enabled. Hangs after:

ppc0: parallel port not found.

Tried booting not using them and so far doesn't hang.

Might be that it can't gather enough entropy; where does it gather
entropy from? Perhaps from static on the (in your case nonexisting)
parallel port?

Cheers,
--
Thomas E. Spanjaard
tgen@netphreax.net

Don't know where it gathers the entropy by default, but worked before
so i guess that it's a bad interaction with ACPI.

Hi!

I have an HP e-PC (i810, Cel 533) running PREVIEW, and after applying
the patch, the s**tloads of ACPI warnings that appeared before on
booting are now gone. :-)
I only get these:

Using $PIR table, 3 entries at 0xc00f7690
ACPI: RSDP 0x0xfc7e0/0x0014 (v 0 AMI )
ACPI: RSDT 0x0xff60000/0x0028 (v 1 HP-BDD HPBDD-IN 0x00000008 MSFT
0x0100000c)
ACPI: FACP 0x0xff60030/0x0074 (v 1 HP-BDD HPBDD-IN 0x00000008 MSFT
0x0100000c)
ACPI: DSDT 0x0xff600b0/0x20a6 (v 1 HP-BDD HPBDD-IN 0x00000008 MSFT
0x0100000c)
ACPI: FACS @ 0x0xff68000/0x0040
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <HP-BDD HPBDD-IN> on motherboard
acpi0: Power Button (fixed)
Warning: ACPI is disabling APM's device. You can't run both
can't fetch resources for \\_PR_.CPU1 - AE_TYPE
can't fetch resources for \\_SB_.PCI0.SBRG.PSMP - AE_TYPE
can't fetch resources for \\_SB_.PCI0.SBRG.URP1 - AE_TYPE
can't fetch resources for \\_SB_.PCI0.SBRG.LPTP - AE_TYPE
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi

^^{^^}^^

load: 0.08 cmd: dd 20 [piperd] 0.00u 0.05s 0% 164k

That's either in /etc/rc.d/random or /etc/rc.d/initrandom.

Tried booting not using them and so far doesn't hang.

Might be that it can't gather enough entropy; where does it gather
entropy from? Perhaps from static on the (in your case nonexisting)
parallel port?

Don't know where it gathers the entropy by default, but worked before
so i guess that it's a bad interaction with ACPI.

Does it stay unresponsive longer in "[acsem]"? If so, drop in to DDB
and type "trace 23" (where 23 is the number before "[acsem]"), and
let me know a few function names starting with "Acpi*" below
AcpiOsWaitSemaphore().

Thanks.

Probably the addition of the following line in acpi.c did it:
AcpiGbl_EnableInterpreterSlack = TRUE;
The version in the CVS already has this feature, but IIRC the code
for interpreter slack has a bug in that version, which was fixed
in the later version, so I gave up committing this simple fix
(but somehow I can't find the description in changes.txt or mail archives).

I saw this and am trying to find how to fix it.

So, did it make it into the login prompt? Also let me know if you
had to use the driver built with ACPI_USE_LOCAL_CACHE=yes.

Thanks.

It does stay for much longer on dd [piperd]. If i try to trace it
i get a "no kernel stack address" message.

Turned out that an extra crit_exit() in objcache_reclaimlist() appears
to be the cause of this lock up, and the machine now boots up fine
without ACPI_USE_LOCAL_CACHE.
I also updated to use the most recent version of ACPI-CA code.
http://les.ath.cx/DragonFly/acpi-20061109-17.diff

Still TODO items:
- `can't fetch resources for XXXX - AE_TYPE` messages.
- `AE_TIME, Could not acquire Global Lock` messages. (reported by victor@)
- make AcpiOs*Lock() interface callable from interrupt context or
idle thread (if possible).

Cheers.

With latest HEAD, your patch and a SMP kernel I get a hang on acsem on sysctl. I'll look into this tomorrow.

cheers
simon

Was it a complete hang, or stuck for a few minutes? Was that during early
in the boot, or sysctl from the command line (either in single- or multi-
user mode)? Also, if you were able to drop into DDB, please send me the
backtrace, in particular a few function names below (I mean callers of)
AcpiOsWaitSemaphore().

Thanks.

It was stuck on bootup (the random thing: seedenable + dd). I did a ^C (after some time) and this was printed:

ACPI Exception (exutils-0382): AE_TIME, Could not aquire Global Lock [20061109]
acpi_acad0: On Line

there were two processes in acsem: acpi_task and sysctl (why sysctl has to aquire the acpi lock is out of my imagination). both seem to try to aquire the global lock. sysctl can happen to run (sleep) on CPU #0 or CPU #1.

trace of acpi_task (transcribed):
AcpiOsWaitSemaphore(c3bc12f0,1,ffff)
AcpiExSystemWaitSemaphore(c3bc12f0,ffff)
AcpiEvAcquireGlobalLock(ffff)
AcpiExAcquireGlobalLock(10)
AcpiExWriteDataToField
AcpiExStoreObjectToNode
AcpiExStore
AcpiExOpcode_1A_1T_1R
AcpiDsExecEndOp
AcpiPsParseLoop
AcpiPsParseAml
ApciPsExecuteMethod
AcpiNsEvaluate
AcpiEvaluateObject
acpi_GetInteger
acpi_acad_get_status
acpi_acad_init_acline
acpi_task_thread

trace of sysctl:
AcpiOsWaitSemaphore(c3bc12d8,1,ffff)
AcpiExSystemWaitMutex(c3bc12d8,ffff)
AcpiEvAcquireGlobalLock(ffff)
AcpiExAcquireGlobalLock(10)
AcpiExWriteDataToField
AcpiExStoreObjectToNode
AcpiExStore
AcpiExOpcode_1A_1T_1R
AcpiDsExecEndOp
AcpiPsParseLoop
AcpiPsParseAml
AcpiPsExecuteMethod
AcpiNsEvaluate
AcpiEvaluateObject
acpi_GetInteger
acpi_acad_get_acline
acpi_acad_sysctl
sysctl_root
userland_sysctl
sys___sysctl
syscall2

on a (maybe) unrelated matter:

I saw your implementation of AcpiOsAcquireLock is not MP-safe (only uses critical sections) and always allows the idle thread to take the lock (even when it is already held before). Is this intended?

oh well, i see that AS_LOCK(as) == crit_enter(). Obviously, this doesn't work for SMP systems, no?

cheers
simon

[backtrace]

Hmm, this looks very similar to what victor@ has reported before.
I didn't even notice that the global lock has to be rewritten before
looking at your patch you sent me privately.

Originally I thought I had to do it that way because
A. most locking primitives involving switching thread led to a panic
when it's from the idle_thread (so I needed to choose from
critical sections/spinlocks),

B. using just spinlocks ended up in a panic when waking up from S3:
still holding 1 exclusive spinlocks!
(so I thought I can't use spinlock)

C. idle thread won't usually run any ACPI code unless it's on a UP
kernel and acpi_cpu takes over the idle_hook, and the current
implementation uses AcpiHw*() functions with ACPI_MTX_DO_NOT_LOCK
flag.

But after playing with it a bit more, the spinlock still being held
at lwkt_switch() was held by elsewhere and not in any AcpiOs*() functions.
So using spinlocks for AcpiOs{Acquire,Release}Lock() turned out to be
just OK but it also may have revealed that some other place needed a fix.

... Or my Athlon64 X2 box isn't actually an SMP system. Anyway,
I'll try your patch on my system before coming back here.

Cheers.

Heh! I can actually sleep and wakeup now! Of course, there is this panic :)

It is due to holding -1 exclusive spinlocks, though. I wonder how this calculation can happen.

cheers
simon

There's this piece of code in acpi_SetSleepState():

if (state != ACPI_STATE_S1) {
       acpi_sleep_machdep(sc, state);

/* AcpiEnterSleepState() may be incomplete, unlock if locked. */
       AcpiOsReleaseLock(AcpiGbl_HardwareLock, 1);

/* Re-enable ACPI hardware on wakeup from sleep state 4. */
    if (state == ACPI_STATE_S4)

Previously AcpiOsReleaseLock() above was AcpiUtReleaseMutex(), and it's
been there because acpi_sleep_machdep() sometimes leaves hardware mutex
locked and sometimes not. That still applies to the patched code, and
you have to unlock the spinlock only if it's locked here. This doesn't
appear to be the right thing to do, and in fact it's been removed
in revision 1.179 of acpi.c in FreeBSD(actual commit log is in 1.180).

the line 592 of AcpiHwReadRegister(), and the only caller of this function
with ACPI_MTX_LOCK is AcpiRegisterRead() in AcpiEnterSleepState():

/* Wait until we enter sleep state */

do
{
    Status = AcpiGetRegister (ACPI_BITREG_WAKE_STATUS, &InValue);
    if (ACPI_FAILURE (Status))
{
        return_ACPI_STATUS (Status);
    }

/* Spin until we wake */

} while (!InValue);

I need to poke around here after going home.

BTW, merged patch is here:
http://les.ath.cx/DragonFly/acpi-20061109-18.diff

Thanks.

On Wed, Jan 10, 2007 at 01:00:53PM +0900, YONETANI Tomokazu wrote:
[spinlock remains held for no reason]

From the debugging output, I know the spinlock holder is always
the line 592 of AcpiHwReadRegister(), and the only caller of this function
with ACPI_MTX_LOCK is AcpiRegisterRead() in AcpiEnterSleepState():

/* Wait until we enter sleep state */

do {

Status = AcpiGetRegister (ACPI_BITREG_WAKE_STATUS, &InValue);
if (ACPI_FAILURE (Status)) {
return_ACPI_STATUS (Status);
}

/* Spin until we wake */

} while (!InValue);

While looking around, I found that AcpiEnterSleepStateS4Bios() has
similar spin loop at the bottom, but with a call to AcpiOsStall()
(== DELAY) before AcpiGetRegister(). So I tried adding it to
AcpiEnterSleepState() too, and to my amazement, the problem went away.
I tried several ten times of suspending and waking up the laptop,
but I'm no longer able to reproduce it. My question, though, is why
why just a DELAY makes any difference (ISTR making gd_spinlocks_wr
volatile didn't solve the problem).

Anyway, new version is here:
http://les.ath.cx/DragonFly/acpi-20061109-19.diff

Cheers.

:While looking around, I found that AcpiEnterSleepStateS4Bios() has
:similar spin loop at the bottom, but with a call to AcpiOsStall()
:(== DELAY) before AcpiGetRegister(). So I tried adding it to
:AcpiEnterSleepState() too, and to my amazement, the problem went away.
:I tried several ten times of suspending and waking up the laptop,
:but I'm no longer able to reproduce it. My question, though, is why
:why just a DELAY makes any difference (ISTR making gd_spinlocks_wr
:volatile didn't solve the problem).
:
:Anyway, new version is here:
: http://les.ath.cx/DragonFly/acpi-20061109-19.diff
:
:Cheers.

My guess is that it is related to BIOS interactions with system
    management events.  It's the same-old, same-old.  Commercial vendors
    hire too many idiot programmers who make the same sad mistakes over
    and over and over again, like using an unsynchronized clock for the
    ACPI timer that can mess bits up when you read it, or just implement
    24 bits, or depend on specific timing arrangements instead of hard
    handshakes.  It really annoys the heck out of me.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

I still can't resume with this patch. It hangs after acpi_cmbat0: initialized (or so). However, with some modifications before i managed to get it to panic within fork1()/vm_page_insert(), so I guess that's further down the road. Not sure what I changed so that I could reproduce this...

When the system hangs, the keyboard is dead as well, so I can't break to ddb or such.

cheers
simon

Is your system running an SMP kernel? If so, can you try UP kernel and
UP acpi.ko and see if it can resume? That may narrow down the source
of the problem.

Please build your acpi.ko with
ACPI_DEBUG=yes ACPI_DEBUG_LOCKS=yes
on make command's argument, and set these in the boot loader prompt
(or in /boot/loader.conf)
set debug.acpi.layer="ACPI_ALL_DRIVERS"
set debug.acpi.level="ACPI_LV_FUNCTIONS"
so you can find the last ACPI function before it hangs.

Cheers.

With this patch it no longer hangs on kern.seedenable. For now it's
working stable and without issues.
Suspend doesn't work, but never worked for me outside of windows. I
don't have S4BIOS support so maybe it's because of that.

By the way, were you able to suspend with the version currently in CVS?

not sure about suspend, but resume for sure doesn't work. or at least last time i tried (screen sayed black after resume).

cheers
simon