https://bugs.dragonflybsd.org/https://bugs.dragonflybsd.org/favicon.ico?16293952082006-06-08T07:14:20ZDragonFlyBSD bugtrackerDragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=27292006-06-08T07:14:20Zdillon
<ul></ul><p>:Disclaimer: MESSY!<br />:<br />:Took a quick attempt at getting the VKERNEL to shutdown cleanly against<br />:1.8,<br />:and it seems to work just fine as suggested with mailbox signals, etc.<br />:<br />:Attached is my 'cumulative' 1.8 vkernel patch against 1.8_Release<br />:/usr/src/sys/platform/vkernel including the previous multi-vk changes.<br />:...<br />:status:<br />: - OK for simple shutdown case with init already fork()'ed.</p>
<pre><code>Looks good.</code></pre>
<p>: - VK is too fast for me to catch the 'no init yet' shutdown case..</p>
<pre><code>Heh.</code></pre>
<p>: - need to merge against Preview, as things have changed there,<br />: and re-submit. patch here includes previous multivkd work.</p>
<pre><code>Yup, I was just going to say that... multi-vk is already in HEAD.</code></pre>
<p>:changes in:<br />: - platform/vkernel/include/globaldata.h : maibox slot<br />: - platform/vkernel/include/md_var.h : vk_shutdown prototype<br />: - platform/vkernel/platform/kqueue.c : mailbox hook<br />: - platform/vkernel/platform/init.c : vk_shutdown function<br />:<br />:questions/notes:<br />: - I didn't think I really needed a separate sigaction in<br />: kqueue_setup..<br />: - should shutdown mailbox handler dispatch happen higher up<br />: (e.g. in platform/vkernel/i386/trap.c)<br />: - should this actually follow<br />: 'vke_intr(void *xsc, struct intrframe *frame __unused)' conventions?<br />: - we're not really checking for FD related IO kevents() ..<br />: - this implies a more 'full device' kind of thing , with a struct,<br />: etc.</p>
<pre><code>The sigaction initialization for SIGTERM doesn't really belong<br /> in init_kqueue(). Put it in platform/vkernel/i386/exception.c<br /> instead.</code></pre>
<pre><code>Now the question is: should it be a signal mailbox or a real signal?</code></pre>
<pre><code>That's a good question. Well, we have an issue with the small size<br /> of the kernel thread stack, so I think it should be a mailbox.</code></pre>
<pre><code>Now... when to check for and call the function! There are two places<br /> where it needs to be checked:</code></pre>
<pre><code>(1) in go_user().<br /> (2) in cpu_idle().</code></pre>
<pre><code>Check for the mailbox and call a function specific to the shutdown<br /> (which you can also put in platform/vkernel/i386/exception.c). Don't<br /> integrate it into signalmailbox().</code></pre>
<p>: - should the other init(8) signals for e.g. single user, etc. be<br />: implemented? Currently, these are not implement as shutdown is<br />: ~90% of cases, and wanted to verify I'm taking the proper direction.</p>
<pre><code>single-user mode on the master host will send a HUP or a TERM<br /> (I forget which) to all processes, including the vkernel. They<br /> should probably be treated as a shutdown by the vkernel.</code></pre>
<p>: - mailbox signals aren't really documented in the manual at the<br />: moment..</p>
<pre><code>I'll do a quick fix to the sigaction manual page. hmm. I thought<br /> I had documented it.</code></pre>
<p>:Comments / Questions / feedback welcome.<br />:<br />:Thanks!<br />:<br />: - Chris</p> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=27632006-06-08T07:14:20Zc.turner
<ul></ul><p>realise I hadn't responsed .. thanks for feeback. will try to cleanup<br />soonish & resubmit.</p>
<p>how's HEAD stabilizing, or is Preview a better bet for patch testing?</p>
<p>Thanks,</p>
<p>- Chris</p> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=27652006-06-08T07:14:20Zdillon
<ul></ul><p>:realise I hadn't responsed .. thanks for feeback. will try to cleanup<br />:soonish & resubmit.<br />:<br />:how's HEAD stabilizing, or is Preview a better bet for patch testing?<br />:<br />:Thanks,<br />:<br />:- Chris</p>
<pre><code>Preview. HEADs gonna be a moving target for a while.</code></pre>
<pre><code>-Matt<br /> Matthew Dillon <br /> &lt;<a class="email" href="mailto:dillon@backplane.com">dillon@backplane.com</a>&gt;</code></pre> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=28162006-06-08T07:14:20Zluxh
<ul></ul><p>Patch not committed. Feature added through a different approach though, see<br /><a class="external" href="http://leaf.dragonflybsd.org/mailarchive/commits/2007-05/msg00228.html">http://leaf.dragonflybsd.org/mailarchive/commits/2007-05/msg00228.html</a> .</p> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=28362006-06-08T07:14:20Zc.turner
<ul></ul><p>This patch should't be committed - it's messy and not the way Matt<br />suggested I clean it up, but the commit referenced is absolutely not<br />the same thing so this shouldn't be marked resolved - unless people<br />don't want the feature anymore, in which case I'll continue to maintain<br />the patch for as long as I'm interested in it.</p>
<p>The functionality implemented for this issue (<a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: rough-draft VKERNEL host-initiated shutdown patch (Closed)" href="https://bugs.dragonflybsd.org/issues/647">#647</a>), the <strong>host</strong> system<br />is able to signal the <strong>guest</strong> VKERNEL to shut down cleanly (e.g. when<br />the host does a halt and forgets to shut down the guest VK's),</p>
<p>With the committed code above, the VK is able to shut <del>itself</del><br />down cleanly, but the host continues to have no way of doing so -<br />Signals from outside will kill the VK and leave the VKD's in an<br />inconsistant state, possibly resulting in a loss of data within the VK<br />system</p>
<p>I responded to Matt's commit because I had a second patch in mind that<br />did the same thing (vkernel <strong>self</strong>-initiated shutdown/reboot) - was<br />probably going to tackle that later... as I see it, the two<br />compliment each other..</p>
<p>In any case,<br />hoping to clean up / resubmit this weekend, more than likely with a<br />second signal handler to take advantage of Matt's new reboot code.</p>
<p>Thanks,</p>
<p>- Chris</p> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=28372006-06-08T07:14:20Zjustin
<ul></ul><p>On Wed, May 30, 2007 8:47 pm, Chris Turner wrote:</p>
<p>I think it would be useful - it should be possible to start AND stop a<br />vkernel without having to actually be in the virtual environment. I'd<br />probably be more antsy about it if I had a spare 1.9 machine to experiment<br />on right now.</p> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=28532006-06-08T07:14:20Zluxh
<ul></ul><p>I guess I was too eager to close issues on the tracker, sorry.</p>
<pre><code>Please do! Keep up the good work.</code></pre>
<p>Cheers,</p>
<pre><code>-Max</code></pre> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=28552006-06-08T07:14:20Zdillon
<ul></ul><p>:<br />:In any case,<br />:hoping to clean up / resubmit this weekend, more than likely with a<br />:second signal handler to take advantage of Matt's new reboot code.<br />:<br />:Thanks,<br />:<br />:- Chris</p>
<pre><code>Definitely! Having a properly working shutdown signal would be good.<br /> In fact, it might be easiest to just set it up as a mailbox signal and<br /> monitor it in the clock interrupt, then create a kernel thread to initiate<br /> the actual shutdown when the signal is detected. Or something like that.</code></pre>
<pre><code>--</code></pre>
<pre><code>I noticed another thing, which is when you run the vkernel from<br /> a shell script, and you interrupt it with ^\ + ^C (that sequence<br /> will enter ddb, then kill the vkernel), it currently causes the vkernel<br /> to do its shutdown/reboot thing, but also kills the shell script and<br /> disconnects the terminal so you wind up getting a shell prompt with the<br /> vkernel still running in the background. Its very odd and annoying<br /> behavior.</code></pre>
<pre><code>I think there is some sort of termios / tty session magic we have<br /> to do to make the tty interrupt signals only go to the vkernel,<br /> but I'm not sure what it is.</code></pre>
<pre><code>-Matt<br /> Matthew Dillon <br /> &lt;<a class="email" href="mailto:dillon@backplane.com">dillon@backplane.com</a>&gt;</code></pre> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=29492006-06-08T07:14:20Zc.turner
<ul></ul><p>OK.. Updated / cleaned this against -HEAD (as of this morning) and<br />things appear to be working.</p>
<p>(vk still too fast to catch the 'no init yet' case)</p>
<p>As suggested:</p>
<pre><code>- moved shutdown hook and sigaction setup to<br /> sys/platform/vkernel/i386/exception.c.</code></pre>
<pre><code>- changed mailbox check to :</code></pre>
<pre><code>- /sys/platform/vkernel/i386/trap.c,v : go_user()<br /> - /sys/platform/vkernel/i386/cpu_regs.c,v : cpu_idle()</code></pre>
<pre><code>- kept mailbox flag in platform/vkernel/include/md_var.h</code></pre>
<p>But:</p>
<pre><code>- I'm still not 100% on the how the SIGIO mailbox interlock<br /> works, so the mailbox check hooks might be in the wrong /<br /> non-optimal places within go_user() / cpu_idle().<br /> - There isn't really a clean way of getting at the shutdown function<br /> from oustide 'sys/kern/kern_shutdown.c'. This results in compiler<br /> warnings, but the linker sorts things out.<br /> - Updated vkernel manual page also attached with patch - hopefully<br /> initial meaning was kept.<br /> - Didn't implement any other signals yet as am still getting approach<br /> sorted out.</code></pre>
<p>p.s:</p>
<pre><code>- now that there are two mailbox signals, perhaps signalmailbox()<br /> should and the related mdvar should be renamed e.g. iomailbox() ?<br /> didn't do this as not really directly related to the patch..<br /> - .. and mailbox signals still a bit fuzzy in the manual :)</code></pre>
<p>Thanks</p>
<pre><code>- Chris</code></pre> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=29522006-06-08T07:14:20Zdillon
<ul></ul><p>:OK.. Updated / cleaned this against -HEAD (as of this morning) and<br />:things appear to be working.<br />:<br />:(vk still too fast to catch the 'no init yet' case)</p>
<pre><code>This looks quite nice. I just realized though that it isn't safe<br /> to run reboot() (or probably ksignal either) from the idle halt<br /> code, so scratch that part.</code></pre>
<pre><code>We have to find a good place to put this. I think what we may have<br /> to do is create a kernel thread whos only job is to shut the kernel<br /> down, and then check the mailbox from inside signalmailbox() and have<br /> it wake the special kernel thread up.</code></pre>
<pre><code>Would you like to have a go at that or would you like me to do it?</code></pre>
<pre><code>-Matt</code></pre> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=29532006-06-08T07:14:20Zc.turner
<ul></ul><p>thanks. ok.</p>
<p>If the special thread doesn't need to do any other cleanup that I'm<br />likely unaware of, I can give it another shot this weekend.</p>
<p>Not yet knowing the overall code paths unfortunately means I'm<br />kind of coding this blind .. but I suppose the point is to learn as I go.</p>
<p>I assume the reason for the thread is that the way things are currently<br />called, the VK might not be in a consistent internal state when the<br />signal is delivered to init and/or init is rescheduled to run?</p>
<p>to clarify:</p>
<pre><code>- keep signal handler and shutdown function in the exception setup<br /> - create kthread somewhere ( main() ? platform init? )<br /> - remove checks from both idle loop / task switching areas<br /> - add check / wakeup logic to signalmailbox()</code></pre>
<p>Thanks,</p>
<p>- Chris</p> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=29542006-06-08T07:14:20Zdillon
<ul></ul><p>:If the special thread doesn't need to do any other cleanup that I'm<br />:likely unaware of, I can give it another shot this weekend.<br />:<br />:Not yet knowing the overall code paths unfortunately means I'm<br />:kind of coding this blind .. but I suppose the point is to learn as I go.<br />:<br />:I assume the reason for the thread is that the way things are currently<br />:called, the VK might not be in a consistent internal state when the<br />:signal is delivered to init and/or init is rescheduled to run?</p>
<pre><code>Right. The idle loop isn't allowed to block, and the shutdown<br /> sequence involves a lot of blocking. The go-user code is in a<br /> sane state and it would be ok to block there, but we wouldn't<br /> get consistent checks.</code></pre>
<p>:to clarify:<br />:<br />: - keep signal handler and shutdown function in the exception setup<br />: - create kthread somewhere ( main() ? platform init? )<br />: - remove checks from both idle loop / task switching areas<br />: - add check / wakeup logic to signalmailbox()<br />:<br />:Thanks,<br />:<br />:- Chris</p>
<pre><code>I think the platform init code is too early. The memory subsystem<br /> isn't really initialized until it gets into the middle mi_startup().</code></pre>
<pre><code>What I recommend is that you create a SYSINIT in<br /> platform/vkernel/i38/exception.c with a priority of SI_BOOT2_MACHDEP<br /> to create the thread. This looks kinda messy, but it should work.</code></pre>
<pre><code>static void<br /> sigshutdown_daemon(void)
{<br /> while (mdcpu->gd_shutdown == 0)<br /> tsleep(&mdcpu->gd_shutdown, 0, "sswait", 0);<br /> }<br /> mdcpu->gd_shutdown = 0;<br /> kprintf("Caught SIGTERM from host system. Shutting down...\n");<br /> if (initproc != NULL) {<br /> ksignal(initproc, SIGUSR2);<br /> } else {<br /> reboot(RB_POWEROFF);<br /> }<br /> }</code></pre>
<pre><code>static struct thread *sigshutdown_thread;</code></pre>
<pre><code>static struct kproc_desc sigshut_kp = {<br /> "sigshutdown", sigshutdown_daemon, &sigshutdown_thread<br /> };</code></pre>
<pre><code>SYSINIT(sigshutdown, SI_BOOT2_MACHDEP, SI_ORDER_ANY,<br /> kproc_start, &sigshut_kp);</code></pre>
<pre><code>Then put a check in signalmbox which wakes up the thread:</code></pre>
<pre><code>/*
* we only need to wake up our shutdown thread once. Keep it non-zero
* so the shutdown thread can detect it.<br /> */<br /> if (mdcpu->gd_shutdown > 0) {<br /> mdcpu->gd_shutdown = <del>1;<br /> wakeup(&mdcpu</del>>gd_shutdown);<br /> }</code></pre>
<pre><code>-Matt<br /> Matthew Dillon <br /> &lt;<a class="email" href="mailto:dillon@backplane.com">dillon@backplane.com</a>&gt;</code></pre> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=30152006-06-08T07:14:20Zc.turner
<ul></ul><p>Okay. All set (+= roff updates per Sascha).</p>
<p>You'd mostly done all the work.. but now I understand the sysinit<br />things a bit more so that's good for something - and thanks! Would have<br />taken me all day to track things down, although my ctags-foo is improving :)</p>
<p>One thing that I did change (aside from minor <75 column reformatting, etc)<br />was that in the SYSINIT with the SI_BOOT2_MACHDEP, I'd get a panic before<br />init.. (see below)</p>
<p>I started to trace through this this but thought 'why not just try setting<br />this up later on in the sequence' which is why this is now set to<br />SI_BOOT2_PROC0, which is the lowest-numbered item in the sysinit_sub_id<br />enum that worked for me (TM).</p>
<p>Also, I'm wondering why you suggested resetting the mailbox flag to -1<br />when it is already nonzero.. kept this as is, but just for curiosity..</p>
<p>I'll knock out the pidfile thing in a few hours time.</p>
<p>Thanks,</p>
<p>- Chris</p>
<p>below:</p>
<p>real memory = 33554432 (32768K bytes)<br />avail memory = 30138368 (29432K bytes)<br />initclocks<br />panic: zone: invalid zone<br />Trace beginning at frame 0xbfbff858<br />(null)(8211300,828cf40,822e45f,bfbff888,0) at 0x80b5c20<br />(null)() at 0x80b5c20<br />Debugger("panic")<br />Stopped at 0x81f9615: movb $0,0x82bb898<br />db> trace<br />(null)(8211189,828cf40,822e45f,bfbff888,0) at 0x81f9615<br />(null)() at 0x80b5c35<br />db></p> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=30292006-06-08T07:14:20Zc.turner
<ul></ul><p>Chris Turner wrote:</p>
<p>updated -u format, per Sacha. Good eyes! (or bad ones on my part :)</p>
<p>Thanks,</p>
<p>- Chris</p> DragonFlyBSD - Bug #647: rough-draft VKERNEL host-initiated shutdown patchhttps://bugs.dragonflybsd.org/issues/647?journal_id=30342006-06-08T07:14:20Zdillon
<ul></ul><p>:<br />:updated <del>u format, per Sacha. Good eyes! (or bad ones on my part :)<br />:<br />:Thanks,<br />:<br />:</del> Chris</p>
<pre><code>Looks good. I did a quick test and it worked great.</code></pre>
<pre><code>I am committing it now!</code></pre>
<pre><code>-Matt</code></pre>