DragonFlyBSD

:After not being able to wake anyone in IRC, I decide to file this bug report in
:the hopes that the build machine creating the Latest DEVEL can be updated, or I
:can be told how to fix it for myself.

Don't worry about it, I'm going start working on the installer
    for the release tomorrow.  I'll be sure to fix the paths.

-Matt

Quoting Matthew Dillon <dillon@apollo.backplane.com>:

Matt,

Thanks for looking at this. I'm going to have to reinstall when a replacement
drive arrives (one out of five was DOA), so I'll get a chance soon (hopefully
before you release) to ensure it works.

I've also dropped a message on the bsdinstaller's "discussion" list about the
way they filter passwords. Basically, the filter they placed has too many
false postives. I've got a patch here to relax this:

http://www.labthug.com/~adrian/bsdinstaller/passwd.patch

I'm also still interested in how the installer builds, installs, and ties
together with the nrelease process, considering that the only testing I
currently know how to give the above patch is a compile. Any information you,
or someone else, can pass along will be greatly appreciated.

Thanks again,

Adrian

I've got the lua-based installer ported. I need a final pass over it and then it will go into bsdinstaller cvs and I will create a binary package.

cheers
simon

<Snip/>

I don't think a separate binary package is the right long term answer.
Tonight
when I get back from the other things in my life, I will resynch the
dragonflybsd tree and work on merging simon's unoffical patch and the rest of
bsdinstaller's head into something under contrib. The current merge patch is
located here:

http://www.labthug.com/~adrian/bsdinstaller/corecode_merged.diff

Please note that I still don't know how to fix the problem I've
encountered with
passwords greater than nine characters long. The installer says they don't
match when I know full well they do (1234567890 can't be mistyped twice six
times in a row). Don't know if this is something we need to fix before
RELEASE
or not. I can't guarantee how far I will get, but my main goal is to have
something along the lines of `make iso-image` on the current source
tree that I
could then use to install onto this new machine. Then again if all this gets
done before I get back, that would be wonderful too :-D

Thanks,

Adrian

The nrelease directory can be used to generate a release ISO. The reason
the installer is treated as a binary package is so that it's easier to get
it placed as part of that image:

http://www.dragonflybsd.org/cvsweb/src/nrelease/README?rev=1.3

Maybe this is already apparent to you and I'm just misreading things. If
so, my bad.

I don't think this is a good idea. It adds quite some (ugly) bloat, like lua, etc. We really do not need this, at least not now. For the time being adding a binary package is well suited, I'd say.

cheers
simon

:I've got the lua-based installer ported. I need a final pass over it and then it will go into bsdinstaller cvs and I will create a binary package.
:
:cheers
: simon

Simon, its got to be near ready for deployment.  As it stands we aren't
    going to be able to release this weekend, we are going to have to do
    it next weekend.  If we don't have a working installer by wednesday then
    I will have to take the non-LUA version that we have from the last
    release, bring it into our own CVS, and make it suitable for this
    release.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

Quoting Matthew Dillon <dillon@apollo.backplane.com>:
<Snip/>

If we don't have a working installer by wednesday then
I will have to take the non-LUA version that we have from the last
release, bring it into our own CVS, and make it suitable for this
release.

<Snip/>

I put the bsdinstaller under src/contrib, and linked in a `make iso` target to
src/Makefile (it just cds and calls nrelease/ make installer_release -- thanks
Justin). I also tweaked the nrelease Makefile to build contrib/bsdinstaller
instead of just copying the root skeleton files over.

`make iso` is building now, but I'm not far enough along the process to gauge
its worth (currently at gnu/usr.bin/cc34/ and started ~ 5 hours ago).
I'm also
not sure if it will finish by Wednesday. If someone with a quicker build
platform could try the patch below and provide feedback (and hopefully an iso
:-D), I would appreciate.

http://www.labthug.com/~adrian/bsdinstaller/integrated.diff

Good night,

Adrian

:Quoting Matthew Dillon <dillon@apollo.backplane.com>:
:<Snip/>
:> If we don't have a working installer by wednesday then
:> I will have to take the non-LUA version that we have from the last
:> release, bring it into our own CVS, and make it suitable for this
:> release.
:<Snip/>
:
:I put the bsdinstaller under src/contrib, and linked in a `make iso` target to
:src/Makefile (it just cds and calls nrelease/ make installer_release -- thanks
:Justin). I also tweaked the nrelease Makefile to build contrib/bsdinstaller
:instead of just copying the root skeleton files over.
:
:`make iso` is building now, but I'm not far enough along the process to gauge
:its worth (currently at gnu/usr.bin/cc34/ and started ~ 5 hours ago).
:I'm also
:not sure if it will finish by Wednesday. If someone with a quicker build
:platform could try the patch below and provide feedback (and hopefully an iso
::-D), I would appreciate.
:
:http://www.labthug.com/~adrian/bsdinstaller/integrated.diff
:
:Good night,
:
:Adrian

You must have a slow machine, but don't worry.  I think you are on
    the right track though I have my reservations about trying to use a
    lua-integrated version for this release.

I will start playing with your patch just to see how easy or difficult
    it winds up being.  I'm not patricularly concerned about a little
    extra bloat for this release since we have a ton of space on the CD,
    but I definitely don't want to get bogged down in having to have 
    a dozen packages installed on the CD just to support the installer
    (not having tried the patch yet I don't know what the requirements
    are, if any).

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

<Snip/>
: You must have a slow machine, but don't worry.

I do (600MHZ P3 ~512MB RAM), this is one of the reasons I'm putting together
the new one.

: I think you are on the right track though I have my reservations
: about trying to use a lua-integrated version for this release.

Right now, I'm just trying to get it to build and put its data in the right
place. When I get that accomplished, I was going to remove the lua stuff.
I haven't heard a case for keeping it.

: I will start playing with your patch just to see how easy or
: difficult it winds up being.

The current patch that's available for download ends up putting the
installer's libraries in the wrong place on the .iso which makes the
executables unable to run. I'm trying a new iteration now. However,
between the slow build time and the fact that I'm back at work, my debugging
turnaround time is hampered.

That being said, when I get home I'm going to reconfigure my new machine so
I can churn stuff out quicker. Its current RAID device is a new NVIDIA
MediaShield controller that isn't supported by DF. NVIDIA provides a linux
driver for it, so I'll work on porting it at some time in the future. Right
now, I'm going to disable the RAID and install DF onto one of the drives.
Then, with the quicker system, I can hopefully knock the new installer out
pretty quickly.

: I'm not patricularly concerned about
: a little extra bloat for this release since we have a ton of space
: on the CD, but I definitely don't want to get bogged down in having
: to have a dozen packages installed on the CD just to support the
installer
: (not having tried the patch yet I don't know what the requirements
: are, if any).
<Snip/>

My goal is after I get it to build successfully, to streamline the process
to remove the bloat everyone is referring to. I can't yet provide an ETA on
this though.

Thanks,

Adrian

On Mon, 16 Jul 2007 09:43:44 -0700 (PDT)
Matthew Dillon <dillon@apollo.backplane.com> wrote:

It really has about 10 packages. The installer itself calls the Lua
executable which communicates with the installer via the dfui protocol,
for which the support is added to Lua by a C module. There are many
other Lua modules used. All are for Lua 5.0, which is deprecated
upstream. In theory, corecode has upgraded the whole thing to Lua 5.1,
which is basically getting the latest versions of 3rdparty lua modules,
upgrading dfuibe_lua to 5.1 and upgrading the scripts, since 5.0 and
5.1 have different syntax. I have not looked at his results yet (is it
already public?)

It however would not be a problem to distribute Lua & friends, since
they are usually MIT-licensed which is even less restrictive than BSD.

:On Mon, 16 Jul 2007 09:43:44 0700 (PDT)
:Matthew Dillon <dillon@apollo.backplane.com> wrote:
:
:> I definitely don't want to get bogged down in having to
:> have a dozen packages installed on the CD just to support the
:> installer (not having tried the patch yet I don't know what the
:> requirements are, if any).
:
:It really has about 10 packages. The installer itself calls the Lua
:executable which communicates with the installer via the dfui protocol,
:for which the support is added to Lua by a C module. There are many
:other Lua modules used. All are for Lua 5.0, which is deprecated
:upstream. In theory, corecode has upgraded the whole thing to Lua 5.1,
:which is basically getting the latest versions of 3rdparty lua modules,
:upgrading dfuibe_lua to 5.1 and upgrading the scripts, since 5.0 and
:5.1 have different syntax. I have not looked at his results yet (is it
:already public?)
:
:It however would not be a problem to distribute Lua & friends, since
:they are usually MIT-licensed which is even less restrictive than BSD.
:
:-
:Gergo Szakal MD <bastyaelvtars@gmail.com>
:University Of Szeged, HU

Its sounding like I should just pull out the installer we used in
    the last release, fix the pkgsrc path, and use that for this release.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

I'm really close to finishing the installer work. Now it is basically runtime issues, like replacing atacontrol with natacontrol etc. The installer has been ported.

It's just a bit slower than usual because I a) am working now and b) I had those strange read errors.

cheers
simon

:I'm really close to finishing the installer work. Now it is basically runtime issues, like replacing atacontrol with natacontrol etc. The installer has been ported.
:
:It's just a bit slower than usual because I a) am working now and b) I had those strange read errors.
:
:cheers
: simon

I'd like to use it.  The other installer work doesn't look like it has
    progressed far enough (and there seems to be some redundancy).  We are
    up against a time limit, though the installer can wait literally until
    the last day since it is a separate entity.

So your deadline is Saturday (not Wednesday like I said before).

-Matt

Okay. I'll say "fuck this installer":

Fuck this installer.

Okay, it is written in super-duper-fancy lua, BUT:

It is written in lua.

And it is pasta, at least to me.

I've wasted enough of my few spare time on this convoluted piece of code. Somebody else go and wonder why again some function returns nil. Not with me.

In case somebody wants to write a nice installer, I'm all for it. Either you wrap your head around this lua stuff, or you write nice code, I'd say in ruby.

Here is a link to my current status: <http://chlamydia.fs.ei.tum.de/~corecode/unsorted/installer.tar.bz2>
and <http://chlamydia.fs.ei.tum.de/~corecode/unsorted/installer-nrelease.diff>

knock yourselves out.
simon

With everything else we have to do the installer is the last thing I
want people losing cycles over. Should we call it a case of 'installers
gone wild' ?

Just forget the whole mess.  I'll research the CVS and try to find the
    last working non-lua installer release and extract that into our own
    CVS tree.

-Matt

:I've wasted enough of my few spare time on this convoluted piece of code. Somebody else go and wonder why again some function returns nil. Not with me.
:
:In case somebody wants to write a nice installer, I'm all for it. Either you wrap your head around this lua stuff, or you write nice code, I'd say in ruby.
:
:Here is a link to my current status: <http://chlamydia.fs.ei.tum.de/~corecode/unsorted/installer.tar.bz2>
:and <http://chlamydia.fs.ei.tum.de/~corecode/unsorted/installer-nrelease.diff>
:
:knock yourselves out.
: simon

You know what I would like you to do if possible, Simon?  Fix the
    vinum auto-start code for root vinums to take the device to read the
    config from from a kernel environment variable instead of scanning
    available devices.  And then scrap the vinum start junk.  I will
    reenable the write-to-label-area code right now.  I'll chalk
    it up as a win if you can boot from a vinum root with HEAD.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

The old installer is still in pkgsrc. I just have to update the
MASTERSITE entries for the moved location.

Joerg

I don't think vinum ever writes to the label area. You are not supposed to start your partition on sector 0.

what's wrong with vinum right now? I always had to pass all drives or it wouldn't work.

cheers
simon

Oh, please not. Let's just build a binary package for this release and use it, and then hopefully we can get something better. We don't have to re-invent the wheel, just produce a rubber one from the stone wheel prototype we have now.

cheers
simon

:I don't think vinum ever writes to the label area. You are not supposed to start your partition on sector 0.

Unfortunately it does if you happen to start your partition at sector 0.
    It writes to offset 4096 and the label area is the first 8K of the disk.
    Even worse, the auto-start code seems to completely ignore the
    partitioning scheme on the disk.

:what's wrong with vinum right now? I always had to pass all drives or it wouldn't work.
:
:cheers
: simon

There's something going when you 'vinum start' and 'vinum stop'.  Do
    it a few times and the kernel blows up.  Literally.

Maybe it doesn't happen when it does the automatic scan at boot time
    when using a vinum root.  It definitely occurs when I do it manually.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

I mentioned this on IRC but one simple path issue is fixed by

<http://leaf.dragonflybsd.org/mailarchive/submit/2007-01/msg00041.html>

I am quite confused about versions here I assume this patch was
commited against the "new" installer (which has proved hard to port
to Dragonflybsd) rather than the old ("working") installer.

I'm not even sure where the definitive Dragonflybsd installer repo is?

And can't help wondering whether it might be simplier to import the old
installer into the Dragonflybsd CVS tree rather than using pkgsrc or
the new installer repo? And maybe we should just concentrate on
testing the old one (with vmware or whatever) and minor bug fixes/patches?

Steve

::The old installer is still in pkgsrc. I just have to update the
::MASTERSITE entries for the moved location.
::
::Joerg
:...

:Oh, please not. Let's just build a binary package for this release and use it, and then hopefully we can get something better. We don't have to re-invent the wheel, just produce a rubber one from the stone wheel prototype we have now.
:
:cheers
: simon
:...

Ok.  I'll wait for Joerg to fix the MASTESITE and then I'll use that.
    I'll do what I did last time, which is create a local mod to the pkgsrc
    patches for the installer to glue in the changes needed to the source.

Then I'll build a custom package just like I did last time and nrelease
    will pull the binary package just like it does now.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

thanks!

:I mentioned this on IRC but one simple path issue is fixed by
:
:<http://leaf.dragonflybsd.org/mailarchive/submit/2007-01/msg00041.html>
:
:I am quite confused about versions here I assume this patch was
:commited against the "new" installer (which has proved hard to port
:to Dragonflybsd) rather than the old ("working") installer.
:
:I'm not even sure where the definitive Dragonflybsd installer repo is?
:
:And can't help wondering whether it might be simplier to import the old
:installer into the Dragonflybsd CVS tree rather than using pkgsrc or
:the new installer repo? And maybe we should just concentrate on
:testing the old one (with vmware or whatever) and minor bug fixes/patches?
:
:Steve

Yes, that's precisely the correct patch and I'll be adding it to my
    modified patches/ list for the pkgsrc package soon.  Thanks for the
    reference!

The lua issue is totally separate... the problem there is that we do
    not want to have to install a ton of infrastructure JUST to support
    the installer, at least not in the base CD distribution.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

Well, this installer just mocks the whole Lua language. Its Lua part is
a whole mess. I have researched it carefully and found that it works in
the following way:
- The installer starts the Lua commandline whose location is defined in
pfi.conf.
- Lua loads modules written in C and Lua and 'connects' with the
frontend. This is painful, because C modules could have been linked to
Lua statically rather than having them loaded runtime. Moreover,
creating self-running executables from Lua code is possible (srlua
anyone?), or using Lua as a script interpreter (#!/usr/pkg/bin/lua in
the first line) is possible.
- The whole mess uses a goddamn compatibility layer for Lua 5.1. When
we tried to update to the latest 5.0-release, we failed miserably since
incompatibilities were found here and there (repeat after me:
compat-5.1 sucks).

• library
  - The library starts the specific lua scripts that represents menuitems
  (this can even be inherited from the current sources). Or, it should
  load a 'master script' that require()'s and/or dofile()'s the scripts
  necessary.
  This would be good because there would be no conflicts with pkgsrc
  packages later installed, there would be no need to define
  OS-/packagemanager-specific library paths and it would eliminate the
  need for using 25 types of package management frameworks in the CVS
  tree. Licensing is also a non-issue since most Lua stuff is under MIT
  license.

Here is what we tried to do with Adam:
- Transformed the ports stuff into pkgsrc.
- Created pkgsrc packages from the lua-based installer's CVS tree.
- I have created a nice pfi.conf.
- Since distfiles were no longer available, we tried to use the latest
ones. This cause incompatibilities between those s**tloads of
libraries. We thought about forking the whole stuff but we constantly
ran into problems with the C code and none of us knows C.
- Corecode said he would port it into 5.1. As he has no experience with
Lua, his reaction is totally understandable, because even the way the
stuff works is a mess, let alone the installer specific Lua interface is
a Rocky Horror Picture Show as well.

nb1 has it.

Joerg

Ok, I've updated the installer used by nrelease to 1.1.7nb1 from pkgsrc.

This version includes the path fixes Joerg committed to the package tree.
    It looks like they fixed the duplicate aura_free() problem too so I did
    not have to make any local modifications. 

-Matt

dfuibe_installer-1.1.7nb1 has the correct paths.
http://leaf.dragonflybsd.org/mailarchive/commits/2007-07/msg00178.html