A picture is worth a thousand words:

A slightly longer story follows, resuming from the previous episode, where Gtk seemed to be working.

Making sure Gtk actually works

Since the udebs are quite fat already, I decided to put the contents of the gtk2.0-examples package into a new revision of the new libgtk-x11-udeb package. gtk-demo was running fine (if one forgets that this demo application is really easier to deal with if one uses a window manager), although image handling was broken. Julien and I spent quite some time double-checking that the libpixbufloader-png.so plugin was present; that both previous DirectFB-based udeb and new x11-based udeb were very similar; that configuration was generated properly; we went so far as to include strace in the netboot image, and to check the configuration file was open()’d, but the plugin was never to be even looked for. All this resulting in a Unrecognized file format message every time.

Josselin, our Messiah, enlightened us: although Gtk was previously using some configuration files to determine which pixbufloader plugin to use, it still continues to access those files… but relies on a MIME database in the end. Installing the host’s /usr/share/mime/mime.cache database did the trick (we have a proper fix pending), and all of a sudden, we got image support (which is needed for the logo_installer.png displayed on top of every screen, as well as for some icons during the installation, like the final one).

Tweaking d-i itself

Julien took care of rootskel-gtk. It was mostly about:

• Dropping some DirectFB start-up scripts.
• Dropping a related configuration file.

• Adding a start-up script doing trivial things:

  Xorg -retro -noreset &
  export DISPLAY=:0
  

Another easy step was cdebconf. Mostly: search and destroy on the few bits using DirectFB. An interesting issue we had was that even though a proper cursor theme was installed and made the default, we only had a black cross as X pointer. That’s where we learnt it’s needed to manually set a cursor for the root window, which can be done this way:

GdkCursor *cursor = gdk_cursor_new(GDK_LEFT_PTR);
gdk_window_set_cursor(gdk_get_default_root_window(), cursor);
gdk_cursor_unref(cursor);

Tweaking gtk2-engines was trivial, just a matter of building against the x11 flavour of Gtk instead of the directfb one. Some substitutions in debian/control and debian/rules, and that was done.

Julien also took care of tweaking console-setup so that keyboard-configuration’s and console-setup-udeb’s postinst scripts call setxkbmap when $DISPLAY is set. Meaning we were able to replace kbd-chooser with console-setup-udeb, leading to a correct setting within X right after having selected the layout!

Reverting d-i’s r61645 (which disabled Gtk-based d-i images) was trivial (although while looking back, it was probably unnecessary), and after having messed around with the packages lists (there’s a lot of local udebs at this point), especially to add some fonts, we managed to get an image which seems to work fine (at least in my VirtualBox environment). I tried Chinese, Japanese, Arabic, and French installations, and all of them seem to work fine (installed systems were bootable after an installation in the selected language) while issues were limited to some rendering or translation glitches, which didn’t seem critical at all.

Next steps

Since there were some iterations, and since the “trial-and-error” approach was chosen, the first goal is to generate clean patches against all modified packages, and make sure a single walk through all packages in a clean chroot is sufficient to generate a working d-i image again.

Once that done, which should probably considered (at least, I hope!) quite a good base already, the idea is to look into the various udebs to see whether there are some parts that could be disabled (maybe introducing a trimmed-down flavour especially for the udeb in some non-flavoured-yet libraries), so as to reduce the size of the resulting image.

In the meanwhile, one can fetch the current hand-built netboot image by browsing http://people.debian.org/~kibi/di-x11/. It’s only available for amd64, and weights 23MB. There’s a sha256sum as well as a detached GPG signature in the same directory.

Last night was the third one spent on playing around with X in a d-i environment.

Making sure X actually works

So far, running X was nice already, but is it really possible to run some applications inside? Bad news is that many trivial applications (xeyes, xclock, …) pull more dependencies than previously packaged.

Finally, xev was chosen, and it indeed works fine (provided one makes sure it gets the focus).

X was crashing when the (only|last) client was exiting, and that wasn’t normal. Thankfully the bug was reproducible outside d-i and Julien sent a patch a few minutes later.

In the meanwhile, using -noreset did the trick to keep X running.

Another way to run X applications there, is to point the host environment to the guest environment using DISPLAY=$GUEST:0 xterm. With VirtualBox, I followed those steps:

• Forward the host’s 6000/tcp to the guest’s 6000/tcp.
• Answer the language-related questions in debconf so that network configuration happens, which is needed for the next steps to work (just getting localhost set up is probably sufficient, but let’s be lazy and let the installer take care of it for now).

• Run X in the guest, disabling access control, with:

  X -retro -noreset -ac
  

• Start xterm from the host with:

  DISPLAY=localhost:0 xterm
  

Moving forward: cairo, pango, gtk

The next step was to rework some udebs. Legend:

• Red box: udeb we no longer want to depend on.
• Blue boxes: udebs to be tweaked first.
• Green boxes: udebs to be tweaked afterwards.
• Ellipse: udeb provided by the one pointed to (magenta edge).

Current situation, keeping only the relationships between udebs of interest:

I took care of gtk+2.0 while Julien hacked pango1.0 and cairo. Some more libraries were needed, but being X libraries, tweaking the packaging was as trivial as for the first X libraries, and it only took a couple of minutes: libxcomposite, libxdamage, and libxinerama.

Once those added to localudebs/ and to pkg-lists/local, and once the image rebuilt, I prepared a tiny “Hello world”-style application:

• Grab the source code from the GTK+ 2.0 Tutorial.

• Built it against the x11 flavour:

  gcc -o hello $(pkg-config --cflags gtk+-2.0) hello.c -lgtk-x11-2.0
  

• Grab it from within d-i, and make it executable.

• Start X as previously explained, then:

  DISPLAY=:0 ./hello
  

Pango wasn’t quite happy at first (message wrapped):

(hello:1991): Pango-WARNING **: failed to choose a font, expect
              ugly output. engine-type='PangoRenderFc', script='latin'

Indeed, since no fonts were included, there was a tiny issue. No big deal, adding ttf-dejavu-udeb did the trick. I wanted to play around with a non-Latin language, so I asked for an example (thanks, Theppitak!), and once ttf-thai-tlwg-udeb added as well, one gets:

Of course, there are many more features to test, but that sounds like we’re going in the right direction (and this is not a joke about RTL).

Next step

Have a look at the green boxes, which may require to have a closer look at g-i’s integration within d-i (it might be needed to see how it’s supposed to be started, that might help testing the tweaked packages).

Getting to X

After having prepared a bunch of libraries, the idea was to get X in shape. I’d like to thank Julien Cristau very much for his support all along the way, since I knew nothing about X.

Basically the plan was to get:

• xserver-xorg-core-udeb: X itself.
• xserver-xorg-input-evdev-udeb: for input, as you probably guessed.
• xserver-xorg-video-fbdev-udeb: for output, as you also probably guessed.

Not much to say about the former, it only pulled two additional libraries compared to what was previously udeb-ified: libaudit0 and libselinux1. Since security modules aren’t the primary concerns in this exercise, --enable-xselinux quickly became --disable-xselinux and those dependencies went away. Both drivers were trivial to tweak, nothing really worth noting there.

Once those 3 additional packages added to localudebs/ and to pkg-lists/local, the mklibs tool really wasn’t happy with them during the rebuild of the netboot image, since the drivers are X plugins, meaning they need symbols which aren’t defined in other libraries. Example:

No library provides non-weak EventSwapVector

Since the idea is to get something working rather than getting something perfect at once, d-i’s packages/mklibs/src/mklibs.py (installed as /usr/bin/mklibs) was told to stop raising an exception, and to print a warning message instead. Rebuilding an image was then working again.

It took some time, but finally it got noticed those packages were actually all installed in the initramfs, so there was no need to try and load/install them manually during the installation process, everything was there already, and it was sufficient to switch to VT2 to see the /usr/bin/Xorg binary.

Some packages were still needed: xkb-data, and x11-xkb-utils. The latter also pulled libxkbfile1, but well, an additional library doesn’t sound very scary.

At this point, one has to try and run Xorg. The result was a black screen, and no means of getting back to another terminal, be it inside Qemu or VirtualBox.

Troubleshooting X

First idea: adding the openssh-server-udeb package to the image, and start sshd from there. But then one has to go through the missing config file, tweaking privilege separation, generating a host key, setting a password, etc. And for whatever reasons, it wasn’t possible to access it. That’s where the DebianInstaller/NetworkConsole page on the Debian Wiki came into play: it includes a few steps to get a working sshd within d-i, painlessly.

So, X was actually starting properly, but getting a black screen is now the default. One wants X -retro to get the well-known old behaviour.

From udevadm info --export-db, it looked like input stuff were missing, and indeed, udev-udeb doesn’t include /lib/dev/input_id and /lib/udev/rules.d/60-persistent-input.rules. Since scp seems broken, wget from the host did the trick followed by chmod +x for the input_id file, otherwise that doesn’t work so well.

But still, there were some missing evdev stuff. And indeed evdev wasn’t to be seen in lsmod, nor available to be modprobe’d.

That’s where it became a little dirtier: wget and udpkg on mouse-modules-2.6.30-2-amd64-di, followed by depmod to make sure, and then modprobe was OK. A little udevadm trigger run to ensure udev was fully aware of the new situation, and restarting X… led to a moving cursor there, as well as a working keyboard, since it was possible to switch between terminals!

Prove it!

Alright, I’m proudly providing you with a screenshot:

(Really, I’ve never been so happy to see this fracking black-and-grey-grid screen. I kid you not!)

If we forget about the very few network-console-related questions answered to get an sshd running, please note that the installation process is still just starting, the very first debconf question is still unanswered.

Now, patching udev again so that udev-udeb includes the two mentioned files was almost easy (details aren’t important anyway), while pulling evdev.ko module automatically was trivial, just a matter of adding such a line to a packages list (I picked pkg-lists/netboot/amd64.cfg):

mouse-modules-${kernel:Version}

Which means one can now, out of the box:

• Boot the ISO, select Install.
• Wait for the first question to appear.
• Switch to another terminal, and activate it.
• Run Xorg -retro there.

Next step

Rework the Pango/Cairo/Gtk stack to use X11 instead of DirectFB.

Producing udebs

How to patch

That’s actually quite trivial. Most packages from Josselin’s 1st and 2nd lists are libraries, and generating udebs for libraries is easy. Basically it boils down to declaring a new binary package:

• Duplicate the libfooN entry in debian/control.
• Add the -udeb suffix to the Package line.
• Add a Package-Type: udeb line.
• Make sure Section is debian-installer.
• (Maybe) make sure Priority is only optional.

Now, populating it might depend on the considered source package, but thankfully, pkg-x11 packages are quite sane, using debhelper’s dh_install. So duplicating debian/libfooN.install into debian/libfooN-udeb.install is usually sufficient. It’s a bit trickier when dh_movefiles is used, but calling install and cp -r after the dh_movefiles call should be sufficient.

Next step, tweak shlibs for libfooN. Without that step, a libbarM package built against libfooN would get libfooN in Depends, which is not appropriate. The idea is to get libfooN-udeb instead. Trivial: add --add-udeb=libfooN-udeb to the dh_makeshlibs call.

Possible issues at that step:

• debian/shlibs or debian/libfooN.shlibs is used instead of dh_makeshlibs. Trivial fix: feed its contents to dh_makeshlibs through its -V argument, and delete that file. Caveat: when it comes to applying a patch removing that file, make sure it’s actually removed, and not just empty. I’m going to file a bug about that, the default behaviour with empty files seems a bit buggy.
• Several libraries are shipped: one can’t pass several --add-udeb arguments. Solution: use several dh_makeshlibs -p$pkg -V$depends --add-udeb=$pkg-udeb call. (I thought it was needed for libx11 but I can’t read, actually: libxcb1 ≠ libx11-xcb1.)

Various checks to perform:

• Make sure debian/libfooN/DEBIAN/shlibs is correct: it should have a udeb line. Example:

  libfoo N libfooN (>= some-version)
  udeb: libfoo N libfooN (>= some-version)
  

• Make sure the generated udebs have no non-udeb packages mentioned in Depends. Since I’m still at the proof-of-concept stage, I’ve iterated when I encountered such additional packages, without checking what was actually needed. The idea is to get rid of extra stuff once the final needs figured out. Of course, one has to track down the dependencies to get the packages incrementally built and installed (for the updated shlibs files).

Results

(These files might be moved around during the following steps, but this post shall be kept up-to-date and point to them at all time.)

Legend:

• Red box: package needing a patch.
• Green box: package already udeb-aware.
• Label with [n]: package patched at step N.
• Label with a *: warning when patching, shlibs file to be removed.

Graphs are scaled down to 75%, click for full size.

Step 1:

• Patches: udebs-1/diffs/
• Udebs: udebs-1/udebs/
• Notes: as explained above, libx11-xcb1 is probably not needed.
• Dependency graph:
  

Step 2:

• Patches: udebs-2/diffs/
• Udebs: udebs-2/udebs/
• Dependency graph:
  

Adding udebs to a d-i image

Basic d-i image

General guidelines (see d-i documentation for detailed installation, needed packages, etc.):

svn co svn://svn.debian.org/svn/d-i/trunk/ debian-installer-trunk
cd debian-installer-trunk/installer/build
make build_netboot

If that breaks (e.g. due to missing packages), make sure there’s a similar entry in sources.list.udeb:

deb http://yourmirror/debian squeeze main/debian-installer

If everything goes well, the generated image should be available as dest/netboot/mini.iso, and tmp/netboot/udeb.list should contain the list of embedded udebs.

To give it a try:

qemu-system-x86_64 -cdrom dest/netboot/mini.iso

Actually adding udebs

Drop all additional udebs under localudebs/. Needed Packages and Packages.gz should be generated on the fly (but beware, secure apt is then disabled, even for non-local udebs).

Now, tell d-i we want some more packages. If all of them are wanted, one can do that:

for i in localudebs/*.udeb; do \
  dpkg-deb --showformat='${Package}\n' -W $i; \
done > pkg-lists/local

(Update: Thanks to Guillem for pointing out a better and cleaner way than dpkg --info $i|grep Package:|awk '{print $2}')

Generate a new image (note the rebuild part of the target):

make rebuild_netboot

Make sure they got added by looking in tmp/netboot/udeb.list and profit!

What’s next

Future steps include:

• Making sure those udebs are actually usable.
• Reworking Pango/Cairo/Gtk udebs.
• Taking a look into X itself.

Stay tuned.

(Disclaimer: this post is not about G.I.)

Graphical installer

Josselin posted his thoughts about g-i, and given at least the first parts sound doable, I’m tempted to have a look at dealing with the needed udebs. Never touched anything like this before, but oh well, one learns every day. Time to learn a bit more about X11 as well.

It looks like generating a new mini.iso (using d-i’s trunk) wasn’t that difficult, so hopefully once I’m done with a few udebs, it should be possible to add them to this image and make sure those are usable, at least when loaded manually.

Once that finished, gluing everything together is probably going to be another story, but hopefully that won’t be impossible.

Unrelated stories

• Thanks to DSA (especially weasel), kfreebsd-* now have DSA-admin’d experimental buildds. Aurélien took care of setting them up; they managed to go through all experimental packages, and bugs were filed accordingly. Some were just marked as failed, usually with “Needs porting.”, especially when there were no regressions with respect to what was previously built on those architectures (as a reminder: out-of-date means serious, uncompiled means important).

• On a related note, this buildd.d.o form makes it possible to browse the build logs for various suites. See the dda@ announce for details.

• It looks like I’m going to stop dealing with Java at all: #557230. Thanks to Gabriele Giacone for taking care!

• It also looks like I’m done with Python packages as well.

• A new module-assistant package got uploaded, which should fix issues with newer kernel versions (ya can haz detailz). I’m wondering whether backporting it through backports.org from time to time would be of some help to anyone? It is (or should be) directly installable in a lenny environment, but maybe people would like it to be directly available once the lenny-backports repository enabled? Mail/wishlist bug welcome if you want it.

• If you received some signatures from me lately… no, I wasn’t pretending I met you at LCA or whatever, but that was from Cáceres (Hi Penny! ;)). Key rollover being thought of.

With the current move of X.Org to unstable, a few tips might help xserver-xorg-video-intel’s users.

Current issues involve screen turning to black from time to time, including while typing, or after a few seconds of inactivity, or after a few minutes. Depending on the applied tips, it’s sometimes possible to switch back to a console, trigger a suspend to RAM, and then get back to an un-black screen. (The funny think being that using s2ram may break the Intel wireless driver, which can be fixed by then using s2disk…)

Restarting the X server is usually insufficient. Logs (be it X.Org’s or kernel’s) don’t show anything. Given those issues showed up with the previous X.Org stack too, with recent kernels, it might just be a bug there.

Various stuff that can be tried:

• Disable powersaving by passing i915.powersave=0 on the kernel command-line. grub2 users can add it to GRUB_CMDLINE_LINUX in /etc/default/grub, and run update-grub2. Once the system is started, it can be checked by checking the contents of the /sys/module/i915/parameters/powersave file.

• Disable DPMS in /etc/X11/xorg.conf: Specify Option "DPMS" "False" in the Monitor section. DPMS enabled or so should vanish from /var/log/Xorg.0.log.

With that setup (apparently, the first tip is particularly interesting, the second didn’t seem to help much), turning to black seems to only happen when using MPlayer or VLC, while it was previously happening while e.g. reading a few lines of IRC backlog… Passing various -vo flags to MPlayer, like -vo x11, doesn’t seem to help.

Have to try with some 2.6.33-rcX before reporting this bug, just wanted to let others know about current findings…

After a while playing around with the build system, with python 3.1 (which is required), etc., I finally got around to upload a Blender package for 2.50 alpha 0, just uploaded to experimental (hopefully) given python 3.1 is only available there.

Potential users really want to read the NEWS file.

One can also check the upstream release logs.

For those wondering about the version number, it had to be >= 2.49, and upstream doesn’t seem to be sure about the versioning, since it can be 2.5 on the website, 2.50 as SVN tag, etc. I’m not going to introduce an epoch right now. ;)

It certainly has some rough edges, but testing and reporting are more than welcome.

I really liked doing stuff like Zack’s RCBW in the past, but nowadays I’m busy with some buildds, reporting RC bugs because some maintainers don’t test their uploads, as well as posting patches to fix FTBFS on GNU/kFreeBSD.

But from time to time, it’s also fun to close RC bugs. Here’s a new concept (that I probably won’t reproduce as regularly as Zack and his minions, though): RCBN for Release-Critical Bugs of the Night.

Those are now closed:

• #552862: postman: FTBFS: cgi-lib.c:238: error: conflicting types for 'getline'
• #545621: regina-normal: FTBFS: problem linking with boost
• #540427: jconv: Build-Depends on libc6-dev (>= 2.7)
• #540669: kflickr: FTBFS: '.libs/libkflickrpart.so': No such file or directory
• #554638: jconv_0.8.1-1(ia64/unstable): FTBFS: build-depnds libc6-dev
• #527732: orpie: FTBFS: (.data+0x2c0): undefined reference to 'ml_gsl_vector_float_minmaxindex'
• #520862: libkarma - FTBFS: /usr/bin/csc: Command not found

Many thanks to Peter Green, to Ruben Molina, and to Ilya Barygin for their patches. Special thanks to Richard Darst for orpie’s patches: he even managed to get rid of embedded code copies while addressing the FTBFS. Nice!

Unfortunately, in the same time, I had to open some FTBFS/RC bugs:

• #559766: java3d: FTBFS on non-Linux ports
• #559767: java3d: FTBFS on armel: install: cannot stat 'j3d-core/build/default/…'
• #559768: java3d: FTBFS on hppa: java compilation issues
• #559771: evolution-dev: Might be missing a Depends on libgtkhtml*-dev
• #559781: ktoon: FTBFS with current ffmpeg
• #559794: libfvm-dev: *LOTS* of undefined references

So, what’s next? Get back to work!

Some uploads this week:

• There’s a 2.49.2~dfsg-1 package for Blender sitting in unstable. Despite its version string, it’s actually 2.49b (release logs). Looks like there’s a good reason for ~dfsg vs. +dfsg.

• There’s a pending package for Blender 2.5 alpha 0, but some things still need ironing out. As stated in #558083, it’ll probably go to experimental first, given its dependency on Python 3.1.

• Speaking of which, some dozens of X.Org drivers (xserver-xorg-input-* and xserver-xorg-video-*) were meant to be uploaded to experimental last night, but due to sbuild’s #559659 and lintian’s #542747, they ended up in unstable instead. If you see packages with a Depends: on X.Org stuff from experimental, that’s normal, wait until xorg-server 1.7 moves to unstable, go and fix some bugs instead of reaching for reportbug.

Recipe for a smile:

1. Notice digikam's FTBFS on GNU/kFreeBSD.

2. Work on a patch, send it to the BTS.

3. Receive the ACK from the BTS.

4. Receive the upstream commit notification a dozen of minutes later.

Denightening lights or enlightening nights or nightening delights…

Kudos, Luka.