I spent a week recently working on the nonfree firmware problem in debian-installer. The problem of course, is that nasty nonfree firmware is needed to use many wireless cards, some other NICs, and some disk controllers, but cannot be included in d-i images.

The fix/workaround landed in unstable today. d-i will detect if a driver needs firmware that is not available. It'll then prompt the user to put the firmware on removable media (such as a USB stick, or floppy, or really anything you can plug into a machine without drawing blood).

The debian-cd team will be building some zip files and tarballs containing all the firmware that debian ships in non-free. (images here) Extract one of those onto a USB stick (vfat formatted), and you're good to go.

The installer will also tell you the names of the firmware files it needs, so if they're some of the ones that Debian cannot even legally distribute (such as the ipw2200 firmware), you can go hunt them down, and dump them onto your USB stick too.

The installer can find firmware files loose on the USB stick, or packaged up in a .deb (or .udeb). The firmware will be used during the installation, and also installed into the finished system. So providing it in .debs is cleanest.

(BTW, if you package a firmware deb for debian, please avoid putting anything but firmware and documentation in it. These firmware debs are also installed into the d-i environment now.)

This could be considered a little clumsy, needing to manually find the bits and put them on a USB stick. I think that people who install Debian frequently will probably end up gravitating toward the "hd-media" installer that boots from USB stick. The firmware can then be put on the same USB stick that contains the installer, and only one thing be needed to install.

Still to do: Use discover's capabilities to scan for firmware that is needed by other devices on the machine, that are not needed to install, but can be used later, and install that firmware too. (This will also support installing non-free kernel modules and other such evil.)

I've now gone fully retro and am running a wiki on gopher. Yes, the protocol from 1991. ;-)

Of course, ikiwiki makes this insanity possible. Full instructions for setting up your own gopher wiki.

So far, the wiki only lives on my laptop, and I suspect that gopher and ipv6 don't get along, so I won't try to give a gopher url to it. :-)

A while ago I wrote down a goal on a whim, without really understanding how important it was:

Completely change everything I'm working on in Debian at least once per 5 years.

This goal has really grown on me. And I've seen a good way to accomplish it. I've decided to stop being involved in packaging third party software for Debian.

There's a bunch of reasons, but the overarching one is that I've been doing that for 13 years, and I don't feel it's a very valuable way for me to spend my time any more. I think I've done a pretty good job, but there is not much room for growth. And it's not as if I spend much time on it. In Steve's survey I estimated I spend about 1/30th of my Debian work time on packaging. I won't gain a lot of time by not doing that work, but I'll lose one mental burden, and slim down my areas of responsibility.

I'll take care of passing packages on to appropriate teams, etc, over the next month. This decision leaves me able to continue to work on teams like d-i and the CD team.

I'll also continue to maintain my own software in Debian. Using a distribution as an incubator for new software, with ready-made community, bug tracking, and of course distribution channels, is something that has been a big benefit to my software, and I think also to Debian, and I plan to continue exploring that space.

I'm excited! This is going to be interesting..

Continuing on with my plans to make ikiwiki more appealing to users without a dedicated server, this evening I've written an ikiwiki plugin that makes it use Amazon S3.

So, it's possible to publish a blog or other static website, built using ikiwiki, without needing your own web server at all. Ikiwiki builds a website and uploads it to Amazon, which then handles the web serving for you.

If you want a traditional wiki that people can edit online, you can still serve the pages out of S3, but you will need to find a "real" web server to host the ikiwiki CGI that handles the page editing. It'll then inject modified files into S3 as necessary.

Amazon EC2 would be the obvious choice for where to run the "real" web server, but probably not the easiest one to set up. In my experiments, I've been running the ikiwiki CGI on nearlyfreespeech, and serving the rest of the wiki out of S3. Since page edits are relatively rare, I estimate this approach will cost a dollar or so a year for the CGI hosting (most of it paying for disk storage). The Amazon S3 hosting of course depends on number of hits and storage size. And presumably it will scale very well, and be very competatively priced, if you believe Amazon's marketing. :-)

---

I'm loving that the design decisions I made about ikiwiki at the very beginning -- that it would use static web pages, and would be backed by a real revision control system, is now letting it be deployed in these interesting ways that I did not begin to envision back then!

Done some interesting stuff in ikiwiki this evening..

Maybe you want to set up a mirror of a wiki. It's easy enough to do with an ikiwiki that's backed by git since you can just clone its repository and set up the mirror. But how to know when there's an update of the origin wiki, to update your mirror? I've added a plugin that allows you to edit a page on the origin wiki, and ask it to ping your wiki. And another plugin that your wiki can use to listen for pings and update itself, pulling down the changes from version control.

Nice thing about this is that any ikiwiki wiki that publishes its revision control, and enables the pinger plugin, can then be mirrored by anyone, with no coordination needed with its admin. Even multilevel mirror networks are possible to set up. (The astute may notice that loops are also possible.. but they will will be broken after 1 cycle.)

But this doesn't only allow mirroring. If you're using distributed version control, it also allows branching of a wiki. Just mirror as usual, but then make changes to the mirror, and don't send them back to the origin. Instant branch, that will be kept up-to-date with changes made to the origin. (Unless there's a conflict, that would need to be manually resolved, obviously.)

Wouldn't it be nice if you could git clone git://wikipedia.org/ or git://wiki.debian.org/ and go off and make it into something you're really happy with? Only thing standing in the way is that neither site uses ikiwiki. For now, you'll have to settle with cloning and branching git://git.ikiwiki.info/ :-)

Technical details here.

Here's a nice toy I added to mpdtoys this evening. vipl allows editing mpd's playlist in a standard text editor. Reorder or delete songs, etc. Enter ">" in front of the song you want to play. Type in partial song, album, or artist names and it will expand them to any matches.

:wq to start the music playing.

I've implemented last night's design, in debhelper 7.0.0.

Currently fully supported are packages that use the autotools, or perl's MakeMaker, or python setup.py, or a simple Makefile. The other classes of packages will be added as bugs are filed about them.

I've uploaded a few of my own packages converted to use debhelper 7 with minimised rules files, so there are a range of examples in incoming:

• mr uses a very simple makefile, and a 3 line debian/rules
• aalib uses automake and autoconf, and is a library package of average complexity. The rules file is minimised where possible using dh, but still does complex things in its build and install targets. It's also a good example of how to set up a -dbg package when using dh, and of how to set up /usr/share/doc symlinks between packages when using dh.
• libaudio-mpd-perl is an example of a perl module package using dh. It shows how to override dh default behavior in two ways: Skipping running dh_auto_test, and passing a changelog filename to dh_installchangelogs.

Early adopters, have at it!

simple case

Imagine a very simple debian/rules, that should work for most packages w/o modification:

#!/usr/bin/make -f
build:
    dh build

clean:
    dh clean

binary-arch:
    dh binary-arch

binary-indep:
    dh binary-indep

binary: build binary-arch binary-indep

absurdly simple case

AKA debian/rules zen guru level.

#!/usr/bin/make -f
%:
    dh $@

the complex cases

Often you'll want to handle configuring by hand. (And maybe, run automake too.)

build:
    dh build --before configure
    #aclocal
    #autoheader
    #automake --add-missing
    #autoconf
    ./configure --kitchen-sink=yes
    dh build --after configure

Here's how to skip two steps in a row (configure and build):

build:
    dh build --before configure
    ./mondoconfig
    make universe-explode-in-delight GROOVINESS=420
    dh build --after build

Another common case is wanting to run some code manually after a particular debhelper command is run.

binary: build
    dh binary --until dh_fixperms
    # dh_fixperms has run, now override it for one program
    chmod 4755 debian/foo/usr/bin/foo
    # and continue
    dh binary

Suppose you hate dh_strip, and just don't want it to run. (This might be a contrived example; if people frequently don't want to run a few commands in a set, it may be better to have -X exclude them.)

binary: build
    dh binary --before dh_strip
    dh binary --after dh_strip

Maybe you want to run a command out of order. This should be no problem:

binary: build
    dh binary --until dh_fixperms
    dh_install foo usr/bin
    dh binary --after dh_fixperms

A separate install target can be added if you like. And maybe you want to run make install by hand in it. (This is tricky because dh binary needs to avoid running dh_auto_install which was skipped over in the install target.)

install: build
    dh install --before install
    $(MAKE) install PREFIX=debian/tmp
    dh install --after install

binary: install
    dh binary

Once you have an install target, you can do this to run some commands manually after the install target runs, and before the remaining steps are run to build a binary.

binary: install
    dh_strip -X foo
    dh_fixperms -X bar
    dh binary --remaining

If the --remaining was left off, dh binary would only run commands in its list after dh_fixperms. With --remaining, it runs commands that haven't run before, avoiding running dh_strip and dh_fixperms again.

A more elaborate way to do the same thing, without needing a separate install target is this.

binary: build
    dh binary --before dh_strip
    dh_strip -X foo
    dh binary --after dh_strip --before dh_fixperms
    dh_fixperms -X bar
    dh binary --after dh_fixperms

the dh_auto_* commands

dh_auto_* are thin wrappers, with enough intelligence to handle 90% of cases (autofoo, perl/python packages, etc) automatically. These will rarely be run by hand, instead if they don't do what is needed, they can just be skipped.

• dh_auto_configure: Figures out how to run autofoo, ./configure, Makefile.PL, or whatever.
• dh_auto_build: runs make
• dh_auto_test: runs make test or the like (may have a config file specifying if test failures should fail the build).
• dh_auto_clean: runs "make clean" etc
• dh_auto_install: runs "make install" or similar

the dh command

dh runs a set of other debhelper commands in order (and prints their names as it does so). The set that it runs is determined by its parameter.

The sets of commands are:

• build: dh_auto_configure, dh_auto_build, dh_auto_test
• clean: dh_testdir, dh_testroot(?), dh_auto_clean, and finally dh_clean.
• install: Everything in the build list, plus dh_installdirs, dh_install, dh_auto_install (and probably dh_install* too)
• binary: Everything in the install list, and all debhelper commands used in binary targets.
• binary-arch: Like binary, but runs commands with -s.
• binary-indep: Like binary, but does not run dhstrip or dhshlibdeps or dh_makeshlibs, and runs commands with -i.

Possibly dh does smart optimisations to avoid running lesser-used commands if they will do nothing. (Or not.)

Each debhelper command will record when it's successfully run in debian/package.log.debhelper. (Which dh_clean deletes.) So dh can tell which commands have already been run, for which packages, and skip those.

Options passed to dh are passed on to each command it runs, which is useful for common options like -v, -X, and the like. For that matter, you can use it for uncommon options as well, since commands will ignore options that don't apply to them.

dh has a few options of its own:

• --until cmd: Run commands until and including cmd, then stop.
• --before cmd: Run commands before cmd, then stop.
• --after cmd: Run commands that are listed after cmd.
• --remaining: Run all listed commands that have yet to be run.

cmd can be a full name of a debhelper command, or a substring. It'll first search for a command exactly matching the name, to avoid any ambiguity. For example, "--before dh_installdeb" does not refer to dh_installdebconf. Otoh, "--before build" is much easier to type, and clearer, than "--before dhautobuild". If there are multiple substring matches, the last in the command list is taken. So "--before install" stops before dh_auto_install, and not before the earlier dh_install.

Each time dh is run, it examines the log, and finds the last logged command that is in its list of commands to run. It then continues with the following command in its list. (Or the one after --after, if that was specified. Or if --remaining was specified, start at the first command in the list that is not in the log.)

So...

If you run debian/rules build twice, and it just has "dh build" in it, the second run will be a no-op. The same as if the build target touched a build-stamp. This is generally considered a feature, so I'll consider it a feature in dh too. (To force a rebuild, run dh_clean to remove the log.)

If you run debian/rules binary, and a build hasn't happened yet, it will run the build steps first. Even if the rules file is a bit lame (or over-golfed), and doesn't have the binary target explicitly depend on the build target. Ditto for the install target.

If a rules file runs a command out of order, it'll need to use --after to get dh back on track.

If you manually run a debhelper command at the command line, running debian/rules binary will skip forward until after that command. Similarly, if debian/rules binary is interrupted (or fails), running it again will resume after the last debhelper command run. This is a bit of a behavior change, and could be considered a bug, but is easy enough to work around by running debian/rules clean binary. (Or by manually deleting the log.)

A year ago I posted an anti-platform with some things I wanted to work on over the next year or three. This is a followup to see what, if anything, has been accomplished.

constantly usable testing

CUT generated some discussion in the project, but I've done nothing toward it myself. The only real progress in this area over the last year has been increasing improvements in the testing security team, and the release team, both of which I'm not currently active in.

deprecating www.debian.org

This continues to happen organically, for example, Debian Weekly News stopped and http://wiki.debian.org/DeveloperNews took its place. Aside from pointing out that this is happening, I haven't done much to push it.

double the archive sync frequency again

Sadly, this has not happened yet.

automated whole system testing

HP donated a test machine, which I set up to run emulated tests of installations of i386, amd64, s390, mips, and mipsel. (Arm and armel should be added soon.) I don't always manage to keep all the tests working at all times by myself, but they have been valuable.

developer (virtual) machines

My test machine spits out disk images for 5 architectures that have debian pre-installed on them, but due to hosting issues I have not been able to share those images with others yet.

find the next nslu2

I was pretty sure this would be OpenMoko's Neo phone, but I have only briefly had access to one, and it's not clear it will be easy enough to get the hardware to make it worthwhile to target it.

Meanwhile, tbm has found a more nslu2-like next nslu2, the QNAP Turbostation NAS device, and there's work underway to fully support it from d-i onwards.

Others have also worked on running Debian on various related AP hardware, including the ASUS WL-500g and Linksys WRTSL54GS, using OpenWRT to boot Debian. There's instructions and downloads on a wiki page. It would be good to get this fully supported without the hacks.

Often if you see a block diagram like this, what comes to mind is a compatability layer in between a program and several operating systems. Generally something that's general-purpose like java, or a web browser, or a widget toolkit.

-------------
|           |
|           |
-------------
|           |
-------------
|  |  |  |  |
-------------

(Generally it's drawn up all purty, but I'm lame.)

But lately I've seen and written a lot of code where the diagram is more complex:

-------------
|           |
|  program  |
|           |
|  |  |  |  |
-------------
| V| C| S|  |
-------------
|    OS     |
-------------

Sometimes the program code is littered with multiple switch statements, as in debcheckout, debcommit, and etckeeper.

case "$vcs" in
git)
svn)
hg)
esac

Sometimes it pushes the VCS-specific code into modules.

use IkiWiki::Rcs::$rcs;
rcs_commit();

But if it does, these modules are specific to that one program. This isn't a general-purpose library. dpkg source v3 doesn't need to use the VCS is the same way as ikiwiki, and even ikiwiki's rcs_commit is very specific to ikiwiki, in its error handling, conflict resolution, locking, etc.

pristine-tar injects and extracts data directly from git, using low-level git plumbing, in a way that probably can't be done at all with other VCSes. But even as I was adding that low-level, very git-specific code into pristine-tar, I found myself writing it with future portability in mind.

if ($vcs eq 'git') {
    # git write-tree, git update-index, and other fun
}
else {
    die "unsupported vcs $vcs";
}

When Avery Pennarun talks about git being the next unix, he's talking about programs like these, that build on top of a VCS.

But if git is the next unix, then so is mercurial, so is darcs, so is bzr, so too even svn (unless it's Windows?). In other words, we're back to the days when every program had to be ported to a bunch of incompatible and not-quite-compatible operating systems. Back to the unix wars.

In Elija's discussion of the "limbo" VCS state he gives several great examples of how multiple VCSes that each seem on the surface to offer similar commands like "$vcs add" and "$vcs commit" can behave very differently.

echo 1 > foo
$vcs add foo
echo 2 > foo
$vcs commit

What was committed, "1" or "2"? Depends on which $vcs you use.

Compare with unix where open(2) always opens a file, perhaps with different options, or different handling of large files, but portably enough that you generally don't need to worry about it. Even if you're porting to Windows, you can probably get away with a compatability layer to call whatever baroque monstrosity Windows uses to open a file, and emulate open(2) close enough to not have to worry about it most of the time.

A thin compatability layer that calls "$vcs add" isn't very useful for a program that builds on top of multiple VCSes. mr is essentially such a thin compatability layer; it manages to be useful only by being an interface for humans, who can deal with different limbo implementations and other quirks.

The VCSes are to some degree converging, but so far it's mostly a surface convergence, with commands that only look the same. Where will things go from here?

• Maybe there will be a standardisation effort like POSIX for VCSes. Though it seems harder; VCSes have a wider interface than just syscalls and filesystems, so there's more scope for incompatibility.
• Or will complicated compatability code like cygwin be developed, to let a program that was written for git use bzr instead, carefully hiding all the differences?
• Another option would be that one VCS wins. So far I'm seeing some consolidation, but little indication that one VCS will emerge as the choice for everyone.
• Maybe the VCSes might begin to support each other's repositories. If a program only supports git, and you use svn, that's fine, if git can transparently access the svn repo.
• Or will we go on for decades spending a lot of time on portability code?