As mentioned when I announced the Slackware image for DevTerm R01 ( Slackware image for DevTerm R01 ), I’ve turned my attention to building a CRUX image for the A06. Like then, I have fried another microSD! As this has been a bit more involved, I decided to do a dev log. Since there’s been enough work done to make it actually useful, I thought I’d start before I can’t call it a dev log and I have to call it v1.
The image is not done yet, but there is enough finished that you could roll it into an image. I spent about three weeks getting packages built and have just finished writing a
Pkgfile for the DevTerm-specific code, so there is enough to use. (It ran a little hot until I finished that, because I had no fan daemon, and had just done
cpufreq-set -g powersave instead of turning off the high-performance cores.)
CRUX is a minimal distro (very nice on small machines) with a ports system: https://crux.nu/ . It is like a much simpler version of Gentoo, although it started life as an attempt to produce a Slackware-like distribution, but with a BSD-like ports system. (Slackbuilds didn’t exist yet.) Arch started by using CRUX as a base; although they don’t share any code, the geneology is roughly that Slackware is the parent of CRUX and CRUX is the parent of Arch. I’ve run CRUX on my desktop for a very long time, and it powers most of my laptops and all of my servers (although usually those servers run VMs and the VMs themselves run Slackware or Plan 9), so I am very familiar with it.
Because it’s minimal and flexible, it seems like a great fit for the DevTerm, and because the A06 is really powerful, building a lot of source felt like a great way to flex the A06. I was also hitting the limits of the flexibility I could squeeze out of the default image: after enough fighting with systemd or NetworkManager or things like that, it starts to become less trouble to just do it yourself than to convince the existing system to do things the way you’d like.
Also to tell the DevTerms apart, I had gotten a sticker at a conference, it says “HACKED”. This is the ultimate portable device for hackers and it has a sticker that says “HACKED”, so I have to at least put a hacked-together OS on it, right?
The first approach, which didn’t work and made me decide to go play with Slackware on the R01, was to try to compile enough of the code to get it started in a chroot, then build enough that the chroot could bootstrap the rest of the code. It was difficult to get the system properly separated from the host system, though: even basic packages ended up linking against system libraries that were present in the official CPi Armbian-based image but that would not be present in the CRUX image. For example, systemd libraries, authentication libraries, things that very basic stuff ends up closely tied to. To solve this, I grabbed a basic aarch64 rootfs from crux-arm.nu, plugged a microSD card into an adapter and put that into the USB port, formatted it, and there I had a chroot.
In an effort to avoid frying a uSD card, I tried a handful of approaches for building packages. Luckily,
/etc/pkgmk.conf is just a shell script, so you have some flexibility. Even on my desktop machine, I have some conditionals in there: I run builds in a ramdisk except for builds that are too large to run in RAM. Most of the system is easy to build in RAM, even on the DevTerm: 4GB gives you, by default, a 2GB ramdisk if you mount one on /tmp, and 2GB is much more than enough to compile coreutils or even something like Redis, but the Rust compiler (needed for Firefox but also for things like librsvg) required 12GB of space to build. That would be stretching the free space on a 32GB microSD card!
So, my first approach was to try using sshfs! I have a fairly large server in the rack next to my desk, and it has 80GB RAM, so it wouldn’t be hard to just use sshfs to mount a subdirectory of that server’s /tmp (a ramdisk) locally on the DevTerm, right? Although it’d be slow due to network latency, it would at least work, and since I can just do my work on the R01 DevTerm (running a very comfortable Slackware setup), I could just let it take as long as it wanted to build. This worked for most packages, but it broke whenever a source tarball had extended ACLs, or there were operations that root could do on a file but that wasn’t permitted by a FUSE filesystem; I figured I’d solve these things later.
Finally, the big, extremely annoying issue: some packages (Rust again) required an inordinate amount of RAM to compile. So, although when you have a slow I/O device, you usually want to speed up a compile by just adding more parallel jobs (so that the CPU-heavy parts can be done by some cc processes while others are busy reading or writing disk), in this case my machine kept locking up and sitting there, angry with me, until it finally decided the RAM was exhausted and the build failed. If several instances of gcc (usually, when RAM was exhausted, it was g++) or rustc threads were consuming all my memory, the only option was to turn the parallelism back down. So I tried that, and also added another uSD card to run large builds on, hoping that it’d be faster than sshfs.
…And even then, packages like Rust (again), clang, llvm, and qt5 exhausted the RAM, still. I still had some old 8GB microSD cards and, with the target uSD in one slot and the extra build FS in another, I had one last USB port open on the DevTerm and one uSD to USB adapter left, so I plugged it in and ran
mkswap /dev/sdc && swapon /dev/sdc. I figured that was enough (10GB swap, counting the zram swap and the 8GB uSD) so I cranked the parallelism back up and it still ran out of memory.
There was a brief diversion building clang/llvm where I spent an excessive amount of time trying to get their build tool to really just run one job at a time even if it could tell I had six cores; exporting
JOBS=1 worked fine.
After they were fixed, the ports that relied on sshfs still built very slowly! I started November 11, and most of the ports weren’t done compiling until November 25! The worst offenders in terms of build time were gcc, llvm, lld, clang, qemu, qt5, gcc-fortran. (Why gcc-fortran? R still uses some Fortran! So I don’t use Fortran myself, but R is really nice for quick dataviz tasks, and it comes in handy to be able to slice up a bunch of data in awk and then spit out a CSV and have R draw something attractive-looking.)
Eventually, what I did was just carve off a 35GB chunk of RAM on the server inside the sshfs’d slice of /tmp, and then just run mkfs.ext2 on it. No problems with FUSE or non-local filesystems or periodically frying the uSD card I was using as a build directory! Anything root can do to a filesystem, root can do to a filesystem that is mounted over sshfs. So no more issues with being unable to create symlinks, weird timestamps, failure to create device files, things like that.
/etc/pkgmk.conf eventually evolved things like
export V=1 and
export NO_COLOR=1 and things like that to debug builds. A lot of
./configure invocations resulted in
configure: error: cannot guess build type; you must specify one. This is because autotools apparently tries to acquire more information about the host system than is needed for most codebases: a large number of X11 fonts, for example, and those didn’t require a machine-specific part or even a compile step, but all
config.guess gets upset if it’s never heard of your CPU, something that you will probably see once you start building a lot of code on RISC-V or aarch64 systems. In most cases, you can just copy a newer
config.guess in place over the existing one.
Except for periodically pausing to fix issues listed the above, it went very smoothly. I just dashed off a list of packages that I figured I’d need (compilers and interpreters and ways to talk to larger systems, plus the pretty comfortable environment I have gotten used to being able to run on the DevTerm: ratpoison, conky, urxvt, drawterm, p9p, xpdf, etc.), wrote a little loop to build and install those packages and their dependencies one at a time and log failures to a file in /tmp, turned on logging in screen (so I could get to the scrollback if something failed in the middle of the night and I didn’t notice it), and let it run. Most stuff went off without a hitch: all the little utilities, languages, Lua and Ruby and gcc. It was surprisingly smooth compared to the way this kind of thing played out on ARM years ago.
After I got enough of it built that I figured I’d be happy with the system regardless, I tried rebooting. It came up, it drew Xorg sideways (I hadn’t done the Xorg configs yet), I exited X, and then was idle long enough that the screen blank kicked in…and I hadn’t started up the network. I hit the power switch and hoped for the best, but it didn’t work, so I ended up long-pressing it. I swapped the uSD card in the USB port (which had the official image on it) with the one in the TF slot (which had my CRUX image) and…it booted CRUX again! I looked around, thinking I’d pulled them out and put them back in without swapping, but it turns out I’d blown out the FS and the journal was corrupt. I suppose that’s one way to force me to start using the CRUX image full-time! This was annoying but it was really nice to know that the device can find a rootfs to use if the default one won’t mount. (After a long and punishing
fsck, everything was in
/lost+found but I managed to recover most of my home directory at least.)
Most of the ports worked as-is. I have to clean up my personal ports repository so that I can make it public. I made one specific to the DevTerm A06, but it ended up with only two ports in it: one for the code in GitHub - clockworkpi/DevTerm: This code repository offers downloads for the latest images of various DevTerm models, as well as kernel patches, keyboard firmware, the source code for screen and printer drivers, hardware schematics, assembly instructions, and essential technical documents. so the printer and gpio pins and fan and whatnot will work. There is one for some of the Xorg config files that are needed, including the joystick-as-mouse one (which is left optional; I use my DevTerm for work and now that the uConsole is here, I haven’t played games on my DevTerm much).
This is still a work in progress! You can, if you want, pull down the compiled packages, add enough of them to an empty FS that you can chroot into and install the rest of them (coreutils, pkgtools, and filesystem skeleton) from inside the chroot. I still just copied the ClockworkPi normal kernel.
- Complete set of prebuilt packages, including at least a reasonably full-featured browser; a lot of stuff is in there already (Ruby, LuaJIT, R, mpv, ghostscript, qemu, iftop, nmap, Redis, etc.), but there’s also some stuff missing, most of which is large, Firefox and GIMP and Inkscape and Go.
- Additional configuration.
- A few more scripts and some extra code to fit the DevTerm. For example, persist the time when shutting down and restore it on boot: there’s no RTC so unless you can get to an NTP server when you boot, the clock will read 2013! (If you look at the prebuilt packages, you will possibly notice that the “wpa_supplicant” and “ntp” have timestamps claiming January 2013. After I got those compiled and installed, I used them to fix the clock.)
- A sensible set of defaults for /etc/pkgmk.conf, /etc/rc.conf, some changes to /etc/rc (for example, no more
setterm -blank 15).
- A clean build of the system and some test-booting.
- Tweaks for CM4. I’m still planning to do Plan 9 on CM4 DevTerm, but since the uConsole is here, I might change my CM4 uConsole from Raspbian to CRUX.
- I think I will be uploading some tools I have grown around my DevTerm/uConsole. Quality of life stuff, mostly shell scripts and C programs that tweak the backlight or print the battery status. I will add those to the ports repo and have them preinstalled on the image.
- …And several other things that I will not realize until after I have spent a week or two living on this machine.
I’m not running X on it right now, but I haven’t taken enough physical pictures of this wonderful device (all the images in the Slackware thread were just screenshots, not photos), so here is neofetch and top running under screen(1), vertically split. Also visible: my replacement EVQWJN007 trackball, because I have used my DevTerm almost as much as my desktop machine this year, enough to wear out the existing trackball (thanks to The Cheapest Keyboard Hardware Mod for the suggestion), and the dust that has accumulated on the screen while this amazingly beautiful portable machine sat on my desk, chained to the USB-C charger, acting as a build system for itself.
(You can probably guess why the box is called “armitage” if you played the SNES Shadowrun.)
I will edit this section when new pieces are uploaded or updated. (I’ll also try to figure out a better hosting solution for really large files than “use IPFS or just get it from my house via Tor”. I like IPFS much more than using something like mega, though. Maybe I’ll do a torrent, maybe I’ll find enough space on a Frantech box. Even my big secret project starts to choke around 8G, IPFS has trouble with it.)
Ports tree, web: git.debu.gs Git - cpi-ports/summary
Ports tree, git: git://git.debu.gs/cpi-ports
Ports tree, github: GitHub - pete/cpi-ports: CRUX ports tree for DevTerm
Pre-built packages, tor: http://s3ldbb3l5eqd6tjsklzmxy6i47i3fim55fpxmgeaa6rvpcllkt4ci4yd.onion/a06/crux/
Pre-built packages, IPFS:
bafybeihl7hbs2zjofmpqd2ttydibxpnx7ai2g5hp4ju5u3fempdffqp7pm . You should be able to do
ipfs ls bafybeihl7hbs2zjofmpqd2ttydibxpnx7ai2g5hp4ju5u3fempdffqp7pm or use a mirror (e.g., http://dweb.link/ipfs/bafybeihl7hbs2zjofmpqd2ttydibxpnx7ai2g5hp4ju5u3fempdffqp7pm .)
Image: pending, will upload somewhere as soon as it’s ready; see “What Remains” above. I will probably produce two images: one minimal and one self-contained (i.e., includes full system source and CPi’s documentation in
/usr/src and prebuilt packages in
/usr/ports, so you can rebuild the entire system with no net connection or reinstall anything without rebuilding it; I hate not being able to find the source for a package because the site went down or I don’t have a net connection). The self-contained one, recommended if you want everything you need, is going to be more than 8GB, but for space-conscious people the minimal one should be only a few gigs.