Black Screen on DevtermA04 after sudo apt upgrade? This may be a fix for you

DevTerm
1

On my DevtermA04 i’ve had to redo the sd card several times after a sudo update. It seemed that lightdm did not start properly. It seems that the apt update command is run after boot, this bogs it down and does not allow other processes to do their thing and lightdm tries to restart 5 times then dies.

On stackexchange I saw the suggestion to look at the /etc/init/lightdm.conf where you find settings called respawn and limit.
I changed to the following: respawn limit 200 1500 and from then on it worked 100% everytime after a update/upgrade.
Happy devterming!

2

A more correct way of doing this would be to lower the CPU and IO priority of the apt process, and maybe even delay its start…

3

pkr

I agree 100% but unfortunately my knowledge of GNU/Linux is enthusiast level, not genius level like some of the people here. This will probably date me like a fossil but I remember working and playing around with SysV, now the kids have systemd these days! It also appears that the ubuntu/debian distro way is a lot different than the OpenSuse that I use. Please feel free to educate me in the more elegant way of doing this task, I am happy to learn!

Questions/note to self

  • how is autostart implemented here? is it a file, or more systemd magic?
    -how do i delay startup tasks, in the init file system somewhere? systemd?
    -does the renice command work here as I think it does? needs to be tested.
4

I have no clue how it’s done, but it’s gonna take quite some research. SysV was easy peasy in-your-sleep breezy. Systemd is much more complex, but Debian adds its own voodoo to the mix. renice should work, but ideally it should be nice’d and ionice’d directly at the source/origin.
A good place to start might be systemd-analyze blame and systemd-analyze critical-chain

5

pkr,

thank you for the suggestions, I’ll have a look!

6

I tried the suggested commands:

root@clockworkpi-a04:~# systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @16.539s
└─multi-user.target @16.539s
  └─cups.service @35.320s +109ms           <----------------was in red
    └─cups.socket @35.310s
      └─sysinit.target @6.536s
        └─systemd-update-utmp.service @6.513s +           <----------------was in red
          └─systemd-tmpfiles-setup.service @6.417s +54ms           <----------------was in red
            └─systemd-journal-flush.service @6.390s +20ms           <----------------was in red
              └─systemd-journald.service @6.251s +133ms           <----------------was in red
                └─armbian-ramlog.service @3.590s +2.655s           <----------------was in red
                  └─var-log.mount @3.718s
                    └─dev-zram1.device @3.409s


then ran systemd-analyze blame and got 

root@clockworkpi-a04:~# systemd-analyze blame
15.032s bootsplash-hide-when-booted.service
10.058s logrotate.service
 4.592s accounts-daemon.service
 4.086s devterm-socat.service
 3.136s udisks2.service
 2.655s armbian-ramlog.service
 2.257s systemd-resolved.service
 2.215s NetworkManager.service
 2.047s bluetooth.service
 1.774s systemd-logind.service
 1.771s wpa_supplicant.service
 1.528s dpkg-db-backup.service
 1.426s armbian-zram-config.service
 1.390s dev-mmcblk0p1.device
 1.371s ssh.service
 1.297s lightdm.service
 1.213s chrony.service
 1.213s rsyslog.service
 1.168s smartmontools.service
  987ms networking.service
  942ms blueman-mechanism.service
  893ms e2scrub_reap.service
  812ms user@0.service
  706ms update-notifier-download.service
  644ms loadcpufreq.service
  578ms packagekit.service
  532ms systemd-udevd.service
  517ms colord.service
  492ms systemd-udev-trigger.service
  470ms upower.service
  468ms alsa-restore.service
  447ms user@1000.service
  410ms apport.service
  405ms keyboard-setup.service
  335ms cpufrequtils.service
  331ms dev-zram0.device
  303ms apt-daily-upgrade.service
  236ms sysfsutils.service
  228ms armbian-hardware-optimize.service
  203ms dev-hugepages.mount
  199ms dev-mqueue.mount
  194ms sys-kernel-debug.mount
  188ms fake-hwclock.service
  179ms kmod-static-nodes.service
  170ms modprobe@configfs.service
  170ms armbian-led-state.service
  168ms modprobe@drm.service
  163ms modprobe@fuse.service
  133ms systemd-journald.service
  131ms rc-local.service
  128ms kerneloops.service
  124ms systemd-sysctl.service
  124ms gdm3.service
  121ms systemd-tmpfiles-clean.service
  121ms hostapd.service
  117ms openvpn.service

Looks like time to RTFM!

7

Hmmm… everything in critical chain looks normal. I’ll mark the services that I think might have something to do with apt. You can examine each one with systemctl cat [service]

15.032s bootsplash-hide-when-booted.service
10.058s logrotate.service
 4.592s accounts-daemon.service        <- interesting but probably not it
 4.086s devterm-socat.service
 3.136s udisks2.service
 2.655s armbian-ramlog.service
 2.257s systemd-resolved.service
 2.215s NetworkManager.service
 2.047s bluetooth.service
 1.774s systemd-logind.service
 1.771s wpa_supplicant.service
 1.528s dpkg-db-backup.service        <- much suspicious
 1.426s armbian-zram-config.service
 1.390s dev-mmcblk0p1.device
 1.371s ssh.service
 1.297s lightdm.service
 1.213s chrony.service
 1.213s rsyslog.service
 1.168s smartmontools.service        <- um, is this really needed for a system with no real SSDs/disks?!?
  987ms networking.service
  942ms blueman-mechanism.service
  893ms e2scrub_reap.service
  812ms user@0.service
  706ms update-notifier-download.service      <- sus ඞ
  644ms loadcpufreq.service
  578ms packagekit.service        <- maybe sus
  532ms systemd-udevd.service
  517ms colord.service
  492ms systemd-udev-trigger.service
  470ms upower.service
  468ms alsa-restore.service
  447ms user@1000.service
  410ms apport.service        <- ?!? wut be this?
  405ms keyboard-setup.service
  335ms cpufrequtils.service
  331ms dev-zram0.device
  303ms apt-daily-upgrade.service        <- sus ඞ
  236ms sysfsutils.service
  228ms armbian-hardware-optimize.service
  203ms dev-hugepages.mount
  199ms dev-mqueue.mount
  194ms sys-kernel-debug.mount
  188ms fake-hwclock.service
  179ms kmod-static-nodes.service
  170ms modprobe@configfs.service
  170ms armbian-led-state.service
  168ms modprobe@drm.service
  163ms modprobe@fuse.service
  133ms systemd-journald.service
  131ms rc-local.service
  128ms kerneloops.service
  124ms systemd-sysctl.service
  124ms gdm3.service
  121ms systemd-tmpfiles-clean.service
  121ms hostapd.service
  117ms openvpn.service
8

pkr,

thank you sir, will look at these!

9

apt-daily-upgrade.service looks like it could be your issue. This is an automatic unattended upgrade service that runs after apt-daily.service. apt-daily runs the script /usr/lib/apt/apt.system.daily to download the updates, and apt-daily-upgrade runs the same script to do an unattended install of the updates.

You can see here that the apt-daily-upgrade.timer wont start until after apt-daily.timer has started. So you should be able to easily make these services start after boot by editing the apt-daily.timer service. Here is some more info from the man page about systemd timers if you want to check it out. This will explain my changes below.

In order to make this timer run the service 15 minutes after boot and only once a day (24hrs from last run unless you reboot again) after that, edit the [Timer] section of apt-daily.timer to look like this:

[Timer]
OnBootSec=15min   
OnUnitActiveSec=1d   
AccuracySec=1h

Here is the command to easily edit the timer using the default editor:
# systemctl edit apt-daily.timer

Here is a nice tutorial on how to use systemctl if you arent familiar:

NOTE: If you want to FULLY DISABLE the daily update/upgrade, run the following two commands (this is not recommended):

# systemctl disable apt-daily.timer
# systemctl disable apt-daily-upgrade.timer

Smartmontools is probably left in the image for USB HDD/SSDs and devices with EMMC storage. I would leave the package installed and just disable the service if it isn’t needed. This command will immediately stop smartmontools.service and prevent that service from starting on the next boot:

# systemctl disable smartmontools --now

update-notifier-download.service just tells you when updates are available. Shouldnt be causing any issues.

packagekit.service This is a small daemon that adds some functionality to APT. More info: PackageKit - What is PackageKit?

apport.service <- ?!? wut be this? Apport is an error reporting tool. It collects info about crashes and puts it in a standardized format for sending to bug trackers. You can disable this safely, but it isnt a bad thing to have running.

1 Like
10

wow, thank you for the awesome breakdown! The rabbit hole is so much deeper than I thought!

I think that my original theory may have been incorrect. I postulated that it was apt doing its thing that kept everything back thereby terminating lightdm. I did not save a copy of the logfile with the lightdm log/messages terminating after 5 tries and assumed the above after reading a similar thing on stackexchange. There i found the respawn and limit suggestion.

To test the above, I’ve changed the lightdm.conf file back to the original respawn and limit values and EVERYTHING WORKS. Warm and cold reboots work. auto login and normal login works. Apt updates and installs like a dream, the device is humming along very nicely. Hmmmm now I don’t know what is going on.

I did play around with the systemd-analyzecommand and made a couple of plots if that could be useful to see when what happens.

systemd-analyze plot >bootup.svg shows

bootup
bootup1920×1885 303 KB

systemd-analyze dot --to-pattern='*.target' --from-pattern='*.target'  | dot -Tsvg >targets.svg

shows

targets
targets1920×239 37.5 KB

systemd-analyze dot | dot -Tsvg >systemd2.svg shows all kinds of crazy

systemd2
systemd21920×235 66.1 KB

USFrozen, i will play around with the suggestions that you make! Again thank you for the direction and education!

1 Like
11

No problem! I’m still learning all of this as I go along because I don’t know much about how Linux operates on non PC hardware, so I have been leaning on this community for a lot of information. Answering the questions I can makes it feel like I’m giving back and have a way to contribute, so I’m always happy to help!