CRUX for DevTerm A06, dev notes

[1337p337]

As mentioned when I announced the Slackware image for DevTerm R01 ( Slackware image for DevTerm R01 ), I’ve turned my attention to building a CRUX image for the A06. Like then, I have fried another microSD! As this has been a bit more involved, I decided to do a dev log. Since there’s been enough work done to make it actually useful, I thought I’d start before I can’t call it a dev log and I have to call it v1.

The image is not done yet, but there is enough finished that you could roll it into an image. I spent about three weeks getting packages built and have just finished writing a Pkgfile for the DevTerm-specific code, so there is enough to use. (It ran a little hot until I finished that, because I had no fan daemon, and had just done cpufreq-set -g powersave instead of turning off the high-performance cores.)

Why CRUX?

CRUX is a minimal distro (very nice on small machines) with a ports system: https://crux.nu/ . It is like a much simpler version of Gentoo, although it started life as an attempt to produce a Slackware-like distribution, but with a BSD-like ports system. (Slackbuilds didn’t exist yet.) Arch started by using CRUX as a base; although they don’t share any code, the geneology is roughly that Slackware is the parent of CRUX and CRUX is the parent of Arch. I’ve run CRUX on my desktop for a very long time, and it powers most of my laptops and all of my servers (although usually those servers run VMs and the VMs themselves run Slackware or Plan 9), so I am very familiar with it.

Because it’s minimal and flexible, it seems like a great fit for the DevTerm, and because the A06 is really powerful, building a lot of source felt like a great way to flex the A06. I was also hitting the limits of the flexibility I could squeeze out of the default image: after enough fighting with systemd or NetworkManager or things like that, it starts to become less trouble to just do it yourself than to convince the existing system to do things the way you’d like.

Also to tell the DevTerms apart, I had gotten a sticker at a conference, it says “HACKED”. This is the ultimate portable device for hackers and it has a sticker that says “HACKED”, so I have to at least put a hacked-together OS on it, right?

What didn’t work

The first approach, which didn’t work and made me decide to go play with Slackware on the R01, was to try to compile enough of the code to get it started in a chroot, then build enough that the chroot could bootstrap the rest of the code. It was difficult to get the system properly separated from the host system, though: even basic packages ended up linking against system libraries that were present in the official CPi Armbian-based image but that would not be present in the CRUX image. For example, systemd libraries, authentication libraries, things that very basic stuff ends up closely tied to. To solve this, I grabbed a basic aarch64 rootfs from crux-arm.nu, plugged a microSD card into an adapter and put that into the USB port, formatted it, and there I had a chroot.

In an effort to avoid frying a uSD card, I tried a handful of approaches for building packages. Luckily, /etc/pkgmk.conf is just a shell script, so you have some flexibility. Even on my desktop machine, I have some conditionals in there: I run builds in a ramdisk except for builds that are too large to run in RAM. Most of the system is easy to build in RAM, even on the DevTerm: 4GB gives you, by default, a 2GB ramdisk if you mount one on /tmp, and 2GB is much more than enough to compile coreutils or even something like Redis, but the Rust compiler (needed for Firefox but also for things like librsvg) required 12GB of space to build. That would be stretching the free space on a 32GB microSD card!

So, my first approach was to try using sshfs! I have a fairly large server in the rack next to my desk, and it has 80GB RAM, so it wouldn’t be hard to just use sshfs to mount a subdirectory of that server’s /tmp (a ramdisk) locally on the DevTerm, right? Although it’d be slow due to network latency, it would at least work, and since I can just do my work on the R01 DevTerm (running a very comfortable Slackware setup), I could just let it take as long as it wanted to build. This worked for most packages, but it broke whenever a source tarball had extended ACLs, or there were operations that root could do on a file but that wasn’t permitted by a FUSE filesystem; I figured I’d solve these things later.

Finally, the big, extremely annoying issue: some packages (Rust again) required an inordinate amount of RAM to compile. So, although when you have a slow I/O device, you usually want to speed up a compile by just adding more parallel jobs (so that the CPU-heavy parts can be done by some cc processes while others are busy reading or writing disk), in this case my machine kept locking up and sitting there, angry with me, until it finally decided the RAM was exhausted and the build failed. If several instances of gcc (usually, when RAM was exhausted, it was g++) or rustc threads were consuming all my memory, the only option was to turn the parallelism back down. So I tried that, and also added another uSD card to run large builds on, hoping that it’d be faster than sshfs.

…And even then, packages like Rust (again), clang, llvm, and qt5 exhausted the RAM, still. I still had some old 8GB microSD cards and, with the target uSD in one slot and the extra build FS in another, I had one last USB port open on the DevTerm and one uSD to USB adapter left, so I plugged it in and ran mkswap /dev/sdc && swapon /dev/sdc. I figured that was enough (10GB swap, counting the zram swap and the 8GB uSD) so I cranked the parallelism back up and it still ran out of memory.

There was a brief diversion building clang/llvm where I spent an excessive amount of time trying to get their build tool to really just run one job at a time even if it could tell I had six cores; exporting JOBS=1 worked fine.

What worked

After they were fixed, the ports that relied on sshfs still built very slowly! I started November 11, and most of the ports weren’t done compiling until November 25! The worst offenders in terms of build time were gcc, llvm, lld, clang, qemu, qt5, gcc-fortran. (Why gcc-fortran? R still uses some Fortran! So I don’t use Fortran myself, but R is really nice for quick dataviz tasks, and it comes in handy to be able to slice up a bunch of data in awk and then spit out a CSV and have R draw something attractive-looking.)

Eventually, what I did was just carve off a 35GB chunk of RAM on the server inside the sshfs’d slice of /tmp, and then just run mkfs.ext2 on it. No problems with FUSE or non-local filesystems or periodically frying the uSD card I was using as a build directory! Anything root can do to a filesystem, root can do to a filesystem that is mounted over sshfs. So no more issues with being unable to create symlinks, weird timestamps, failure to create device files, things like that.

My /etc/pkgmk.conf eventually evolved things like export V=1 and export NO_COLOR=1 and things like that to debug builds. A lot of ./configure invocations resulted in configure: error: cannot guess build type; you must specify one. This is because autotools apparently tries to acquire more information about the host system than is needed for most codebases: a large number of X11 fonts, for example, and those didn’t require a machine-specific part or even a compile step, but all ./configures call config.guess and config.guess gets upset if it’s never heard of your CPU, something that you will probably see once you start building a lot of code on RISC-V or aarch64 systems. In most cases, you can just copy a newer config.guess in place over the existing one.

The Build

Except for periodically pausing to fix issues listed the above, it went very smoothly. I just dashed off a list of packages that I figured I’d need (compilers and interpreters and ways to talk to larger systems, plus the pretty comfortable environment I have gotten used to being able to run on the DevTerm: ratpoison, conky, urxvt, drawterm, p9p, xpdf, etc.), wrote a little loop to build and install those packages and their dependencies one at a time and log failures to a file in /tmp, turned on logging in screen (so I could get to the scrollback if something failed in the middle of the night and I didn’t notice it), and let it run. Most stuff went off without a hitch: all the little utilities, languages, Lua and Ruby and gcc. It was surprisingly smooth compared to the way this kind of thing played out on ARM years ago.

After I got enough of it built that I figured I’d be happy with the system regardless, I tried rebooting. It came up, it drew Xorg sideways (I hadn’t done the Xorg configs yet), I exited X, and then was idle long enough that the screen blank kicked in…and I hadn’t started up the network. I hit the power switch and hoped for the best, but it didn’t work, so I ended up long-pressing it. I swapped the uSD card in the USB port (which had the official image on it) with the one in the TF slot (which had my CRUX image) and…it booted CRUX again! I looked around, thinking I’d pulled them out and put them back in without swapping, but it turns out I’d blown out the FS and the journal was corrupt. I suppose that’s one way to force me to start using the CRUX image full-time! This was annoying but it was really nice to know that the device can find a rootfs to use if the default one won’t mount. (After a long and punishing fsck, everything was in /lost+found but I managed to recover most of my home directory at least.)

DevTerm-specific Ports

Most of the ports worked as-is. I have to clean up my personal ports repository so that I can make it public. I made one specific to the DevTerm A06, but it ended up with only two ports in it: one for the code in GitHub - clockworkpi/DevTerm: This code repository offers downloads for the latest images of various DevTerm models, as well as kernel patches, keyboard firmware, the source code for screen and printer drivers, hardware schematics, assembly instructions, and essential technical documents. so the printer and gpio pins and fan and whatnot will work. There is one for some of the Xorg config files that are needed, including the joystick-as-mouse one (which is left optional; I use my DevTerm for work and now that the uConsole is here, I haven’t played games on my DevTerm much).

What Remains

This is still a work in progress! You can, if you want, pull down the compiled packages, add enough of them to an empty FS that you can chroot into and install the rest of them (coreutils, pkgtools, and filesystem skeleton) from inside the chroot. I still just copied the ClockworkPi normal kernel.

• Complete set of prebuilt packages, including at least a reasonably full-featured browser; a lot of stuff is in there already (Ruby, LuaJIT, R, mpv, ghostscript, qemu, iftop, nmap, Redis, etc.), but there’s also some stuff missing, most of which is large, Firefox and GIMP and Inkscape and Go.
• Additional configuration.
• A few more scripts and some extra code to fit the DevTerm. For example, persist the time when shutting down and restore it on boot: there’s no RTC so unless you can get to an NTP server when you boot, the clock will read 2013! (If you look at the prebuilt packages, you will possibly notice that the “wpa_supplicant” and “ntp” have timestamps claiming January 2013. After I got those compiled and installed, I used them to fix the clock.)
• A sensible set of defaults for /etc/pkgmk.conf, /etc/rc.conf, some changes to /etc/rc (for example, no more setterm -blank 15).
• A clean build of the system and some test-booting.
• Tweaks for CM4. I’m still planning to do Plan 9 on CM4 DevTerm, but since the uConsole is here, I might change my CM4 uConsole from Raspbian to CRUX.
• I think I will be uploading some tools I have grown around my DevTerm/uConsole. Quality of life stuff, mostly shell scripts and C programs that tweak the backlight or print the battery status. I will add those to the ports repo and have them preinstalled on the image.
• …And several other things that I will not realize until after I have spent a week or two living on this machine.

pix

I’m not running X on it right now, but I haven’t taken enough physical pictures of this wonderful device (all the images in the Slackware thread were just screenshots, not photos), so here is neofetch and top running under screen(1), vertically split. Also visible: my replacement EVQWJN007 trackball, because I have used my DevTerm almost as much as my desktop machine this year, enough to wear out the existing trackball (thanks to The Cheapest Keyboard Hardware Mod for the suggestion), and the dust that has accumulated on the screen while this amazingly beautiful portable machine sat on my desk, chained to the USB-C charger, acting as a build system for itself.

(You can probably guess why the box is called “armitage” if you played the SNES Shadowrun.)

Download

I will edit this section when new pieces are uploaded or updated. (I’ll also try to figure out a better hosting solution for really large files than “use IPFS or just get it from my house via Tor”. I like IPFS much more than using something like mega, though. Maybe I’ll do a torrent, maybe I’ll find enough space on a Frantech box. Even my big secret project starts to choke around 8G, IPFS has trouble with it.)

Ports tree, web: git.debu.gs Git - cpi-ports/summary
Ports tree, git: git://git.debu.gs/cpi-ports
Ports tree, github: GitHub - pete/cpi-ports: CRUX ports tree for DevTerm

Pre-built packages, tor: http://s3ldbb3l5eqd6tjsklzmxy6i47i3fim55fpxmgeaa6rvpcllkt4ci4yd.onion/a06/crux/
Pre-built packages, IPFS: bafybeihl7hbs2zjofmpqd2ttydibxpnx7ai2g5hp4ju5u3fempdffqp7pm . You should be able to do ipfs ls bafybeihl7hbs2zjofmpqd2ttydibxpnx7ai2g5hp4ju5u3fempdffqp7pm or use a mirror (e.g., http://dweb.link/ipfs/bafybeihl7hbs2zjofmpqd2ttydibxpnx7ai2g5hp4ju5u3fempdffqp7pm .)

Image: pending, will upload somewhere as soon as it’s ready; see “What Remains” above. I will probably produce two images: one minimal and one self-contained (i.e., includes full system source and CPi’s documentation in /usr/src and prebuilt packages in /usr/ports, so you can rebuild the entire system with no net connection or reinstall anything without rebuilding it; I hate not being able to find the source for a package because the site went down or I don’t have a net connection). The self-contained one, recommended if you want everything you need, is going to be more than 8GB, but for space-conscious people the minimal one should be only a few gigs.

[1337p337]

Periodically updating the repos and the prebuilt ports directory; trying qutebrowser since it seems like it’ll be easier to build than Firefox, but at this point, everything except the browser is working fine.

The qt6-base package had to be updated because I apparently built it before libxkbcommon. Poor little machine is sitting at 100% CPU for a few days; I had bad luck with distcc lately (it produced binaries that segfaulted) so I’m doing all of the building on-device. So it’s building qt6-webengine now and hopefully that completes successfully.

[1337p337]

I cannot remember what I did to make ninja cap itself at 2 processes instead of auto-detecting 6 cores and deciding to run 6 processes. (ninja is invoked by cmake, according to the docs there is no way to control this through an environment variable for ninja but I think there was a way to get cmake to pass -j2 or something. There are too many build systems. At any rate, since the qt6-webengine build has been running for two days, I don’t want to kill it and start from scratch.)

The issue is that too many parallel invocations of the C++ compiler overfill the RAM, and start bleeding into swap, and once the swap starts filling up, everything starts running at a glacial pace. You can kind of manually do it to a running process, though, by just suspending the ninja process (kill -STOP) and then letting the C++ compilers finish, and resuming the ninja process once RAM use is low enough. Since it’s swapping, I figured it’d be easiest to use renice -n -1 to bump the priority of whatever compiler process was using the most RAM, to get it out of the way faster.

Larry Wall said the three virtues of a great programmer are laziness, impatience, and hubris.

Exercising my laziness, the first virtue, I enlisted awk to just stop ninja when load average got over 4 and bring it back when it went back under 3:

while sleep 1; do uptime; done | mawk -Winteractive '
BEGIN{system("sudo killall -CONT ninja");s=0}
(0+$10) > 4 && !s{system("sudo killall -STOP ninja");s=1}
(0+$10) < 3 && s{system("sudo killall -CONT ninja");s=0}
{print $10, s}'

The second virtue being impatience, maybe there’s a good way to speed the process up. Since the bottleneck for the C++ compiler is RAM and I/O rather than CPU, I think I’ll make another attempt at distcc, but this time, instead of running a cross-compiler, I’ll just copy enough of the rootfs to run a minimal chroot with a compiler under qemu-aarch64 and then run distccd under that. (Shouldn’t have the problem with bad binaries if it’s not just the same version of the compiler, but an emulator running an exact copy of the same compiler, right? …Right?) If that works, it should help with the C++ stuff; I don’t think distcc works for Rust (so it might not help when building Firefox), and gcc doesn’t require nearly as much RAM as g++ (so it’s not needed for regular C compiler invocations).

The third virtue is hubris, and I use this for everything I do.

[1337p337]

I should have probably guessed this, but doing this on an x86-64 host does not do what you might hope it does:

sudo qemu-aarch64 /tmp/a06root/bin/busybox chroot /tmp/a06root

It executes the chroot(2) syscall by emulating a 64-bit ARM CPU, and then tries to execute a shell in that chroot, but cannot do so because it is looking for an x86-64 executable. So I tried the binfmt script that comes with qemu:

$ sudo mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc
$ sudo /usr/local/bin/qemu-binfmt-conf.sh --qemu-path /usr/bin --persistent yes --credential
### [...lotsa output, succeeded...]
$ sudo /tmp/a06root/bin/busybox ls
### Works!
$ sudo /tmp/a06root/bin/busybox chroot /tmp/a06root /bin/busybox
### Fails because it can't find the libraries for qemu!

This would probably work better with qemu statically compiled and it is probably a bad idea to copy the x86-64 libraries into the chroot and then run ldconfig but that is what I did and I have no regrets:

$ sudo cp /lib/libz.so.1 /lib/librt.so.1 /usr/lib/libglib-2.0.so.0 /usr/lib/libgnutls.so.30 /usr/lib/libgmodule-2.0.so.0 /usr/lib/libstdc++.so.6 /lib/libm.so.6 /lib/libpthread.so.0 /usr/lib/libgcc_s.so.1 /lib/libc.so.6 /lib/libpcre.so.1 /usr/lib/../lib/libp11-kit.so.0 /usr/lib/../lib/libtasn1.so.6 /usr/lib/../lib/libnettle.so.8 /usr/lib/../lib/libhogweed.so.6 /usr/lib/../lib/libgmp.so.10 /lib/libdl.so.2 /usr/lib/../lib/libffi.so.7 /tmp/a06root/lib64
$ sudo cp /lib/ld-linux-x86-64.so.2 /tmp/a06root/lib
### I was worried this part wouldn't work:
$ sudo ldconfig -r /tmp/a06root
### ...but it did!  And then, after that, this worked:
$ sudo /tmp/a06root/bin/busybox chroot /tmp/a06root /bin/bash
# uname -m
aarch64

Before attempting that, note that I copied all of them into $chroot/lib64 rather than $chroot/lib. /lib64 is a symlink to /lib, so I just removed the symlink and then added the lbiraries there (the list of which libraries to copy having been generated by doing ldd $(which qemu-aarch64)). /lib/ld-linux is special, that’s the dynamic loader, it’s got to be in /lib.

Anyway, now that that’s working (I wish I had done it sooner; it would have made this build go smoother and likewise with the R01 Slackware build), I can start using that for distcc at least, maybe build more packages in it.

[1337p337]

After two more 3-day builds of qt6-webengine, the last was successful! New packages are present in the .onion link and a couple of small additions to the ports repo.

I was tempted, because distcc was farming all of the compilation to my desktop and I had crammed an 8GB microSD card into the USB port for swap, to leave the JOBS=6 in. It turns out that if the wifi gets congested, distccd will sometimes blink out temporarily and distcc will do the compile locally. This is usually fine, but in this case, there wasn’t enough RAM to do six compiles locally, so it started eating into swap. I wake up and check the screen and the clock is lagging about 30 minutes in the past. After that, a couple of dependent packages needed to be rebuilt, nothing serious, and then qutebrowser was easy.

Nothing special to report for Firefox. Took about a day to compile, distcc handled it, except for the parts where rustc was involved. I think it might have worked out fine without distcc but I didn’t wanna risk it after spending a little more than a week trying to get qt6-webengine to build. I was braced for it to take three days but it was just under one day; I’m actually moderately surprised. (I might have skipped qutebrowser if I’d known Firefox was going to be this easy by comparison, but I had expected the opposite.)

This is, I think, the end of the part where I’m just checking in every few hours to see if it’s done. :‌P

So all that remains is to add ports for a couple of packages (e.g., drawterm is useful, cool-retro-term is fun, maybe try to build retroarch and love2d, though I mostly use my DevTerm for work and the uConsole for games), make some tweaks to the startup scripts, set up some defaults that are friendly for the DevTerm (e.g., minimize writes to disk and generally run lightweight), clean and polish, live with the system a few days to see if I’ve missed anything, and then use these packages to build a fresh image and upload that somewhere a little more accessible than just a Tor site and IPFS. (If you are in a hurry to run it, you can just take the packages and dump them into a rootfs.) I am excessively excited to have my machine available again instead of leaving it plugged in, occupied building packages. Very excited to pass -s 2 to the gear script again!

Except some small changes to the hardware (no rotating the screen, trade the printer for 4G modem, etc.), the same images work on both the DevTerm and the uConsole, so maybe next up I run off a uConsole image.

As a side note, barrier is extremely useful so there’s a package for that: git.debu.gs Git - cpi-ports/blob - barrier/Pkgfile . If you are not familiar, it’s a fork of the last open-source version of synergy, and basically it allows you to just move your mouse off the edge of one screen and onto another, running on another machine, so you can control multiple machines with one keyboard/mouse. (Like VNC but if you can just see the other monitor and thus don’t need the screen mirrored.) So when I’m at my desk, I have my DevTerm between my monitor and keyboard, and I turn on barrier and then I can just move my mouse down. I’ve also used it to control the uConsole from the DevTerm (uConsole is way newer, but you can imagine sitting at a table and using the DevTerm to edit code while using the uConsole to show documentation or play a video), or to control a machine that’s sitting on a desk using the DevTerm that I’m carrying around the room. Highly recommended.

[1337p337]

I still have to produce the full image, but I’m still running the working image I built.

There is no major blocker, just I haven’t taken the time to build it and to figure out a better hosting situation. (Hosting it where my website/email/whatever live is fine but the upstream is bad so I’d have to throttle and my main hosting setup has metered bandwidth, so I will probably end up throwing it onto a Frantech slab.) In the mean time, I don’t know how easily people on this forum can get to a Tor server.

The plan is mostly to take take the kernel/bootloader/partition table from the official image, copy the kernel modules into /lib/modules, add all the packages (prebuilt packages available here, will figure out a method for hosting some bigger files: http://s3ldbb3l5eqd6tjsklzmxy6i47i3fim55fpxmgeaa6rvpcllkt4ci4yd.onion/a06/crux/ ), and do test-boots, debug it, etc., then remove the ssh keys (so new ones are created on startup), set the root password to something simple so that you can log in and create your user account. Basic nice stuff.

I have acquired a TuringPi 2 and some of their RK-1 boards, so I don’t need to use the qemu chroot any more: distccd lets them function as a small build farm, and their CPUs are similar enough (CPi’s A-06 is a RK3399, big.LITTLE 4/2, the TuringPi RK-1 uses an RK3588, which is dynamIQ and 4/4) that I actually used the DevTerm packages to build out the CRUX image for the RK-1. (I’m running it on a couple of headless CM-4s also, so I guess it’s also compatible, drivers aside.) I wish I had gotten one of them sooner (they had the same supply chain issues that Clockwork Pi had): it was painful trying to get the stupid qt6 stuff to build, but if I had a few nodes with excessive amounts of RAM, I wouldn’t have needed to do hacks.

It is just regular Linux, so the screenshots have more to do with what I am doing to the system than they do with what the system does, but I did like CPi’s amber setup so much that I cloned it on my R-01 Slackware image (They look orange in screenshots, but the bluish cast of the LCD makes them amber: Slackware image for DevTerm R01 .), and the view is a little more conventional on the A-06: running Firefox to view AwkiAwki ( GitHub - pete/awki: Awki is a lightweight wiki written in awk. These are my patches, original here: http://awkiawki.bogosoft.com/ ), dvtm in urxvt, cool-retro-term (power-hungry, I don’t use it too often), drawterm to edit code on the Plan 9 system (and in front of it, 9front’s vt(1), showing an ssh interface to a Pleroma instance), and trusty old xpdf showing “Coders at Work”.