Getting suspend to work properly on A06

If I run echo freeze>/sys/power/state or use the xfce suspend it suspends, and resumes when I press the power button - but drains the battery in a few hours (with protected cells).

echo mem>/sys/power/state blanks the screen for a moment then fails with bash: echo: write error: Connection timed out. The dmesg log has errors about xhci-hcd but adding it to SUSPEND_MODULES under /etc/pm/config.d didn’t help.

Has anyone had any more luck with this?

Same issue.

xhci-hcd is compiled into the kernel so it cannot be unloaded…
I’ve checked the source tree, and it seems that the rockchip usb 3.0 driver is not even included.
I’m trying to disable all xhci features in the kernel config. Compiling the kernel now;

1 Like

I removed xhci from the kernel. Suspend now works. Hibernation doesn’t – can go into hibernation but has issues resuming.

2 Likes

Nice! I added kernel argument to avoid having to recompile:

initcall_blacklist=xhci_hcd_init,xhci_pci_init,xhci_plat_init
4 Likes

cool, today I learn something. :slight_smile:

Thanks pkr, I tried adding that to extraargs in /boot/armbianEnv.txt, and I’m not getting any errors in dmesg any more - but suspend still doesn’t work, it blanks the screen for a moment then returns to the desktop. There’s this (not very helpful) message in syslog:

systemd-sleep[3324]: Failed to suspend system. System resumed again: Operation not supported

Yeah, I just noticed mine wasn’t going into deep suspend either. On mine, it tries deep suspend, fails, and silently goes to s2idle (freeze). Debug messages aren’t helpful at all. I even tried disconnecting all USB devices including keyboard and it’s still the same result :eyes:

[   46.736347] PM: suspend entry (deep)
[   48.017450] Filesystems sync: 1.280 seconds
[   48.017482] PM: Preparing system for sleep (deep)
[   48.125025] Freezing user space processes ... (elapsed 0.003 seconds) done.
[   48.128362] OOM killer disabled.
[   48.128369] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[   48.130050] PM: Suspending system (deep)
[   49.743033] PM: suspend of devices complete after 1612.087 msecs
[   49.743068] PM: start suspend of devices complete after 1613.005 msecs
[   49.743079] PM: suspend devices took 1.610 seconds
[   49.744492] PM: late suspend of devices complete after 1.403 msecs
[   49.745819] PM: noirq suspend of devices complete after 1.191 msecs
[   49.746008] Disabling non-boot CPUs ...
[   49.749606] psci: CPU1 killed (polled 0 ms)
[   49.752635] psci: CPU2 killed (polled 0 ms)
[   49.755523] psci: CPU3 killed (polled 0 ms)
[   49.757308] psci: CPU4 killed (polled 0 ms)
[   49.759720] psci: CPU5 killed (polled 0 ms)
[   49.760681] PM: Checking wakeup interrupts
[   49.760694] PM: Calling ledtrig_cpu_syscore_suspend+0x0/0x40
[   49.760717] PM: Calling sched_clock_suspend+0x0/0x40
[   49.760733] PM: Calling timekeeping_suspend+0x0/0x2e0
[   49.760733] PM: Calling irq_gc_suspend+0x0/0x80
[   49.760733] PM: Calling kvm_suspend+0x0/0x40
[   49.760733] PM: Calling fw_suspend+0x0/0x20
[   49.760733] PM: Calling cpu_pm_suspend+0x0/0xe0
[   49.760733] PM: Calling its_save_disable+0x0/0xf0
[   49.760733] PM: Calling its_restore_enable+0x0/0x160
[   49.760733] PM: Calling cpu_pm_resume+0x0/0x70
[   49.760733] PM: Calling kvm_resume+0x0/0x34
[   49.760733] PM: Calling irq_gc_resume+0x0/0x84
[   49.760733] PM: Calling irq_pm_syscore_resume+0x0/0x20
[   49.760733] PM: Calling timekeeping_resume+0x0/0x15c
[   49.760733] PM: Calling sched_clock_resume+0x0/0x60
[   49.760735] PM: Calling ledtrig_cpu_syscore_resume+0x0/0x20
[   49.760774] Enabling non-boot CPUs ...
[   49.761032] Detected VIPT I-cache on CPU1
[   49.761071] GICv3: CPU1: found redistributor 1 region 0:0x00000000fef20000
[   49.761142] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[   49.761847] CPU1 is up
[   49.762036] Detected VIPT I-cache on CPU2
[   49.762067] GICv3: CPU2: found redistributor 2 region 0:0x00000000fef40000
[   49.762122] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[   49.762930] CPU2 is up
[   49.763107] Detected VIPT I-cache on CPU3
[   49.763136] GICv3: CPU3: found redistributor 3 region 0:0x00000000fef60000
[   49.763188] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[   49.764141] CPU3 is up
[   49.764341] Detected PIPT I-cache on CPU4
[   49.764370] GICv3: CPU4: found redistributor 100 region 0:0x00000000fef80000
[   49.764422] CPU4: Booted secondary processor 0x0000000100 [0x410fd082]
[   49.765625] CPU4 is up
[   49.765844] Detected PIPT I-cache on CPU5
[   49.765867] GICv3: CPU5: found redistributor 101 region 0:0x00000000fefa0000
[   49.765906] CPU5: Booted secondary processor 0x0000000101 [0x410fd082]
[   49.767154] CPU5 is up
[   49.768849] PM: noirq resume of devices complete after 1.679 msecs
[   49.769831] PM: early resume of devices complete after 0.827 msecs
[   50.838253] PM: resume of devices complete after 1068.403 msecs
[   50.840841] PM: resume devices took 1.070 seconds
[   50.840869] PM: Finishing wakeup.
[   50.840878] OOM killer enabled.
[   50.840883] Restarting tasks ... done.
[   50.851450] PM: suspend exit
[   50.853645] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
[   50.919339] mmc_host mmc1: Bus speed (slot 0) = 50000000Hz (slot req 50000000Hz, actual 50000000HZ div = 0)

Looks like it’s something between its_save_disable and its_restore_enable

Oh well… I tried to suspend last night with DT under my pillow…
A few hours later I touched something hot 0.0

disabled all my usb ports (lsusb displays nothing) and still no deep suspend. Probably not related to usb :expressionless:

But on the bright side, with all my poking and twiddling things, my screen ended up black, usb is dead, and it’s now drawing 470mA from USB-C while still connected to wifi! now that’s what i call reasonable power consumption!

Could you try lowering it to Gear 1 (maybe even lower, 1 LITTLE core @ 400MHz) and measure power consumption?

I believe it will be almost exactly the same, or at least too small a difference for my meter to measure (<10mA). One little thing I noticed is the big cores are stuck at 816MHz minimum instead of going down to 408MHz, so manually setting the frequency or even disabling the big cores should affect power consumption on light load.
In Manjaro you can use cpu-temp-speed to check current clock speeds.

I see. Indeed the lower clock states are marked as “inefficient”.
We may have some luck if we enable ftrace for the kernel and poke around…

Edit: note to self:

loc comment
kernel/power/suspend.c:L391 suspend_enter
L425 call to pm_sleep_disable_secondary_cpus, then cpuidle_pause. We observe a log sequence to shutdown non-booting cores.
L434 call to syscore_suspend
drivers/base/syscore.c:L47 syscore_suspend – goes through registered syscore_ops. cpu_pm is one of them (maybe irrelevant).
arch/arm64/kernel/sleep.S, arm64/kernel/suspend.c arm64 sleep. It’s hooked up to cpuidle.

In between, there’re two paths that may cause immediate resume:

  • pm_sleep_disable_secondary_cpus fails, or suspend_test fails (update: that’s for pm_test).
  • syscore_suspend() fails.

Update:

After a few rounds of printk hacking…

	error = syscore_suspend();
  pr_err("after syscore_suspend, error=%d", error);
	if (!error) {
		*wakeup = pm_wakeup_pending();
    pr_err("after pm_wakeup_pending");
		if (!(suspend_test(TEST_CORE) || *wakeup)) {
			trace_suspend_resume(TPS("machine_suspend"),
				state, true);
      pr_err("before suspend_ops->enter");
			error = suspend_ops->enter(state);
      pr_err("after suspend_ops->enter");
			trace_suspend_resume(TPS("machine_suspend"),
				state, false);
		} else if (*wakeup) {
			error = -EBUSY;
      pr_err("wakeup BUSY");
		}
		syscore_resume();
	} else {
    pr_err("syscore_suspend fails: %d", error);
  }
[  965.639523] PM: suspend entry (deep)
[  965.930618] Filesystems sync: 0.291 seconds
[  965.974514] Freezing user space processes ... (elapsed 0.003 seconds) done.
[  965.978488] OOM killer disabled.
[  965.978494] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[  965.980077] printk: Suspending console(s) (use no_console_suspend to debug)
[  966.526141] PM: suspend devices took 0.550 seconds
[  966.529183] PM: after platform_suspend_prepare_noirq
[  966.529248] Disabling non-boot CPUs ...
[  966.532391] psci: CPU1 killed (polled 0 ms)
[  966.536375] psci: CPU2 killed (polled 0 ms)
[  966.539808] psci: CPU3 killed (polled 0 ms)
[  966.542967] psci: CPU4 killed (polled 0 ms)
[  966.545572] psci: CPU5 killed (polled 0 ms)
[  966.546524] PM: after syscore_suspend, error=0
[  966.546524] PM: after pm_wakeup_pending
[  966.546524] PM: before suspend_ops->enter
[  966.546524] PM: after suspend_ops->enter
[  966.546543] Enabling non-boot CPUs ...
[  966.547097] Detected VIPT I-cache on CPU1

So it’s suspend_ops->enter not doing its work now;

For arm64 suspend should be handled by PSCI. drivers/firmware/psci/psci.c
Down there (PSCI firmware), it’s really out of my reach. But I’ll finish the tracing anyway.

[    0.000000] psci: psci_init_smccc: feature = 0
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] psci: psci_init_cpu_suspend: feature = 0
[    0.000000] psci: psci_init_system_suspend: ret = 0
[    0.000000] psci: psci_system_reset2: ret = -1
[   80.919482] Disabling non-boot CPUs ...
[   80.923640] psci: CPU1 killed (polled 0 ms)
[   80.927784] psci: CPU2 killed (polled 0 ms)
[   80.931379] psci: CPU3 killed (polled 0 ms)
[   80.935126] psci: CPU4 killed (polled 10 ms)
[   80.938110] psci: CPU5 killed (polled 0 ms)
[   80.939568] PM: after syscore_suspend, error=0
[   80.939568] PM: after pm_wakeup_pending
[   80.939568] PM: before suspend_ops->enter
[   80.939568] psci: enter psci_system_suspend_enter
[   80.939568] psci: enter psci_system_suspend, pa_cpu_resume=312F490
[   80.939568] PM: after suspend_ops->enter
[   80.939599] Enabling non-boot CPUs ...

So… PSCI is correctly initialized. But invoke_psci_fn(PSCI_FN_NATIVE(1_0, SYSTEM_SUSPEND), pa_cpu_resume, 0, 0); bounces back immediately indicating an error.

Suspecting psci is not properly configured in device tree.

This is the original patch: [PATCH 0/2] PSCI: system suspend support


psci_system_suspend_enter -> cpu_suspend -> psci_system_suspend

arm64 suspend.c, L119

	if (__cpu_suspend_enter(&state)) { // state is 0 for system suspend
		/* Call the suspend finisher */
		ret = fn(arg); // but PSCI doesn't like it...

		/*
		 * Never gets here, unless the suspend finisher fails.
		 * Successful cpu_suspend() should return from cpu_resume(),
		 * returning through this code path is considered an error
		 * If the return value is set to 0 force ret = -EOPNOTSUPP
		 * to make sure a proper error condition is propagated
		 */
		if (!ret)
			ret = -EOPNOTSUPP;
	} else {
		RCU_NONIDLE(__cpu_suspend_exit());
	}

Possibly related: 0012-add-suspend-to-rk3399-PBP.patch · master · manjaro-arm / packages / core / linux-pinebookpro · GitLab

The patch looks promising. It uses a rockchip-specific way of talking to the trusted firmware (ATF).
It cannot be directly applied to DevTerm, because it uses ACPI pm_poweroff_prepare. (It will be removed soon, anyway: [v7,20/20] reboot: Remove pm_power_off_prepare() - Patchwork)

If we want to do it quick and dirty, we can just plug it into psci.c without even touching device tree ((
If clockworkpi cares enough they should be able to pick it up and polish…

1 Like

Heh, I was just about to suggest that you check the work on Pinebook Pro, but looks like you already started :slight_smile:. They still haven’t gotten suspend to work for everyone, but for certain configurations it works perfectly.

Unfortunately I’m a bit computer illiterate, so I can’t help much here :sweat_smile:

Unfortunately I’m a bit computer illiterate, so I can’t help much here :sweat_smile:

Says the one who taught me about initcall_blacklist :stuck_out_tongue:

4AM now. I need to suspend.

Research led me to initcall_blacklist (and initcall_debug). I figured a little bit of research and hackery was much faster than trying to compile a kernel.

Update.

I managed to patch the kernel with the rockchip pm code. Kernel is compiling, let’s see what will happen.

Update.

Fixed some errors in DT, power-key confirmed to be: <&gpio0 RK_PA5 GPIO_ACTIVE_HIGH>; (see A06 schematics, 2 of 15, RK3399_C

Kernel module loads fine. And… nothing happened. As expected. Maybe we have to use “virtual poweroff”?

Update.

As the rockchip pm config code loads after PSCI, we can reliably hijack the syscore suspend descriptor – after that, we intercept suspend requests that were meant to go to PSCI.

Turns out the rockchip patch is really half-baked. That “virtual poweroff” command will try to parse a device tree node for… ir keys, which seems to be talking about a remote controller that comes with a development board. You point the ir led to the dev board and shoot a resume command. Lazy devs :smiley: :smiley: (edit: wait, an rk3399 android TV box?)

The core of that function is very similar to what we have in PSCI driver.

PSCI ver:

static int psci_system_suspend(unsigned long unused)
{
	phys_addr_t pa_cpu_resume = __pa_symbol(function_nocfi(cpu_resume));

	return invoke_psci_fn(PSCI_FN_NATIVE(1_0, SYSTEM_SUSPEND),
			      pa_cpu_resume, 0, 0);
}

rockchip ver:

int sip_smc_virtual_poweroff(void)
{
	struct arm_smccc_res res;

	res = __invoke_sip_fn_smc(PSCI_FN_NATIVE(1_0, SYSTEM_SUSPEND), 0, 0, 0);
	return res.a0;
}

… which is only worse, right.

Result? Doesn’t even go into s2idle, because it thinks the suspend is successful :man_facepalming:

[   48.432366] rockchip_pm_virt_pwroff: begin
[   48.432366] rockchip_pm_virt_pwroff: end. ret = 0

So the real deal may be hiding inside ATF.
I really hope clockworkpi can chime in here to assist. They might have a BSP uboot that bundles proper suspend ATF.

3 Likes

Update. Looking into upstream ATF, many ROCKCHIP_SIP smc functions are not handled at all.
Rockchip does not provide source code for their ATF.

We can try their binary version.

Update: Tried rkbin bl31.elf + mainline uboot, no banana

At this point we’re out of (obvious) options. I’ll back off for a while to recharge myself. If anyone is interested, the next target will be to examine the upstream ATF code for rockchip platform. TLDR: It’s not compatible with the pm code in the PBP patch, and it’s known that cpu suspend works, but system suspend, which asks to suspend the last active cpu, fails.

One possible reason would be that power domains and the regulators are not properly shut down. image

Update:

One possible reason would be that power domains and the regulators are not properly shut down.

That was close!

I think I’ve got s2ram working by disabling the rk808 irq: Is A06 rk808 PMIC_INT_L actually GPIO1_C5? - #10 by yatli.
USB charger showing 0.00A-0.07A – previously never got it under 0.2A!
The other sign of success is that I cannot wake it up anymore :smiley: :smiley:
No need for the PBP patches. The upstream ATF + stock linux code is just fine!

Caveats:

  1. Make sure the fan is turned off before going to sleep. That thing draws some 60mA.
  2. Find a way to wake it up
  3. If power is cut during sleep it enters a weird state, the power LED dims, and the core is shut down.
  4. The usb keyboard is not powered off during the whole time. I’m pretty sure about this because my custom keyboard FW remembers states.
  5. The hibernation resume is not interrupted any more, but it does not resume successfully either.
    a) This is because a hibernation resume requires going into suspend and back after the memory image is loaded. So it depends on s2ram working first.

Update:

The blocking issue is that, the configuration of wakeup sources, GPIO states during suspend, suspend method etc., are not standardized by PSCI. Instead, these are platform-specific (rk3399-specific, in our case). Currently these configurations are missing from upstream ATF, and there isn’t Linux code to work with that, either.

Rockchip released a Linux patch to work with their closed-source ATF, which is what the PBP patch is about. It’s old, It’s bad.

So the plan is like this.

  • For quick-and-dirty experiments, we can first apply the PBP patches, and use Rockchip closed-source ATF. That may work, but I don’t want it that way.
  • I have started to look at the closed-source ATF and try to port it to upstream ATF. This is actually doable. Just need to figure out how to configure the registers.
  • This will have to work with Linux module. We can port the Rockchip one and tailor it to our needs.

Update:

The upstream ATF actually has all the registers mapped out and set, but they are not exposed through platform SIP so not configurable from Linux dt.
For anyone interested, rkbin, rk3399 bl31.elf, address 0x00047a88 is what resembles the upstream ATF plat/rockchip/rk3399/drivers/pmu/pmu.c – see suspend_gpio, sys_slp_config etc.

For example, upstream ATF:

	slp_mode_cfg = BIT(PMU_PWR_MODE_EN) |
		       BIT(PMU_WKUP_RST_EN) |
		       BIT(PMU_INPUT_CLAMP_EN) |
		       BIT(PMU_POWER_OFF_REQ_CFG) |
		       BIT(PMU_CPU0_PD_EN) |
		       BIT(PMU_L2_FLUSH_EN) |
		       BIT(PMU_L2_IDLE_EN) |
		       BIT(PMU_SCU_PD_EN) |
		       BIT(PMU_CCI_PD_EN) |
		       BIT(PMU_CLK_CORE_SRC_GATE_EN) |
		       BIT(PMU_ALIVE_USE_LF) |
		       BIT(PMU_SREF0_ENTER_EN) |
		       BIT(PMU_SREF1_ENTER_EN) |
		       BIT(PMU_DDRC0_GATING_EN) |
		       BIT(PMU_DDRC1_GATING_EN) |
		       BIT(PMU_DDRIO0_RET_EN) |
		       BIT(PMU_DDRIO0_RET_DE_REQ) |
		       BIT(PMU_DDRIO1_RET_EN) |
		       BIT(PMU_DDRIO1_RET_DE_REQ) |
		       BIT(PMU_CENTER_PD_EN) |
		       BIT(PMU_PERILP_PD_EN) |
		       BIT(PMU_CLK_PERILP_SRC_GATE_EN) |
		       BIT(PMU_PLL_PD_EN) |
		       BIT(PMU_CLK_CENTER_SRC_GATE_EN) |
		       BIT(PMU_OSC_DIS) |
		       BIT(PMU_PMU_USE_LF);

PBP patch DT:

+		rockchip,sleep-mode-config = <
+			(0
+			| RKPM_SLP_ARMPD
+			| RKPM_SLP_PERILPPD
+			| RKPM_SLP_DDR_RET
+			| RKPM_SLP_PLLPD
+			| RKPM_SLP_OSC_DIS
+			| RKPM_SLP_CENTER_PD
+			| RKPM_SLP_AP_PWROFF
+			)
+		>;

Upstream ATF:

	mmio_setbits_32(PMU_BASE + PMU_WKUP_CFG4, BIT(PMU_GPIO_WKUP_EN));

PBP patch:

+		rockchip,wakeup-config = <
+			(0
+			| RKPM_GPIO_WKUP_EN
+			)
+		>;

… which allows GPIO interrupt wakeup.

However, this is missing:

+		rockchip,wakeup-config = <
+			(0
+			| RKPM_GPIO_WKUP_EN
+			)
+		>;

The upstream ATF does not configure GPIO pins in sleep mode according to Linux configuration:

static void suspend_gpio(void)
{
	struct bl_aux_gpio_info *suspend_gpio;
	uint32_t count;
	int i;

	suspend_gpio = plat_get_rockchip_suspend_gpio(&count);

	for (i = 0; i < count; i++) {
		gpio_set_value(suspend_gpio[i].index, suspend_gpio[i].polarity);
		gpio_set_direction(suspend_gpio[i].index, GPIO_DIR_OUT);
		udelay(1);
	}
}

plat_get_rockchip_suspend_gpio is in plat/rockchip/common/params_setup.c, pointing to a block of gpio pin descriptors, which is populated through bl31_setup -> bl31_early_platform_setup2 -> params_early_setup -> bl_aux_params_parse -> rk_aux_param_handler

Which is… expecting configuration from uboot? I have no ideas.
nvm, the problem is that, GPIO0_A5 should stay HIGH or PULLUP in suspend, otherwise RK808-PWRON and axp228-PWRON will stay low.

A little diagram to save some alt-tab and ctrl-f:

Forget about rk_aux_param_handler for a while:

static void suspend_gpio(void)
{
	struct bl_aux_gpio_info *suspend_gpio;
	uint32_t count;
	int i;

	suspend_gpio = plat_get_rockchip_suspend_gpio(&count);

	for (i = 0; i < count; i++) {
		gpio_set_value(suspend_gpio[i].index, suspend_gpio[i].polarity);
		gpio_set_direction(suspend_gpio[i].index, GPIO_DIR_OUT);
		udelay(1);
	}

  // hack: set GPIO0_PA5 high
  // port = pin / 32
  // num = pin % 32
  // bank = num / 8
  // id = num % 8
  // GPIO0_PA5: port = 0, bank = 0(A), id = 5
  // num = bank * 8 + id = 5
  // pin = port * 32 + num = 5
  gpio_set_value(5, 1);
  gpio_set_direction(5, GPIO_DIR_OUT);
}

Trying that one now;

Update:

I got the custom ATF running. The only thing that works as a debug indicator, is the FAN. “Spin for x seconds, stop for y seconds” is my new printf:

  // GPIO3_A0 is the FAN, port = 3, bank = 0(A), id = 0
  // num = 0*8+0 = 0
  // pin = 3 * 32 + 0 = 96
  // works!
  // GPIO4_C4 is UART TX (with led), port = 4, bank = 2(C), id = 4
  // num = 2 * 8 + 4 = 20
  // pin = 4 * 32 + 20 = 148
  // doesn't work

  int dbg_pin = 96;
  gpio_set_value(dbg_pin, 0);
  gpio_set_direction(dbg_pin, GPIO_DIR_OUT);
  for (int n = 0; n < 5; ++n) {
    udelay(3000000);
    gpio_set_value(dbg_pin, 1);
    udelay(3000000);
    gpio_set_value(dbg_pin, 0);
  }

Here’s the RM for rk3399:

From page 496 on there’s a listing about suspend registers.

I have tried the following, without success yet:

  • GPIO0_A5, PWR_KEY_L, input, pullup
  • GPIO1_C1, CPU_B_SLEEP, output, high
  • GPIO1_B6, GPU_SLEEP, output, high
  • GPIO1_A5, PMIC_SLEEP_H, output, high

Update:

  while (true) {
    int v = gpio_get_value(5);
    gpio_set_value(96, v);
    udelay(100000);
  }

Nope, that doesn’t change FAN status when I press/unpress the pwr key. That’s why there’s no wakeup source.

Here’s my FAN-based printf:

static void cpi_fan_bit(int bit) {
  // start
  // protocol: HIGH 3 sec
  gpio_set_value(96, 1);
  udelay(3000000);
  // protocol: LOW 3 sec
  gpio_set_value(96, 0);
  udelay(3000000);

  // each bit is transmitted in 1sec.
  // bit set   = 800ms high 200ms low
  // bit clear = 200ms high 800ms low
  int dhigh = bit ? 800000 : 200000;
  int dlow = bit ? 200000: 800000;
  for (int i = 0; i < 4; ++i) {
    gpio_set_value(96, 1);
    udelay(dhigh);
    gpio_set_value(96, 0);
    udelay(dlow);
  }
} 
  1. Start: 3sec high, 3sec low
  2. Then repeat 4 times: If bit set, 800ms high 200ms low If bit clear, 200ms high, 800ms low

With this I’m able to dump PMUGRF_GPIO0A_IOMUX:

RUN #01

raw:
0100 0000 0010 0000 0000 0000 0000 0000

reversed:
high16:
0000 0000 0000 0000
low16:
00   [gpio0a7 gpio]
00   [gpio0a6 gpio]
01   [gpio0a5 emmc_pwren]
00   [gpio0a4 gpio]
00   [gpio0a3 gpio]
00   [gpio0a2 gpio]
00   [gpio0a1 gpio]
10   [gpio0a0 test_clkout0]

And we need to get GPIO0_A5 back to GPIO.

  // see: https://github.com/ARM-software/arm-trusted-firmware/blob/5e529e32ee6cf6ac9203ada4fade49a47893fa51/plat/rockchip/rk3399/drivers/pwm/pwm.c#L61
  int iomux = mmio_read_32(PMUGRF_BASE + PMUGRF_GPIO0A_IOMUX);
  if (((iomux >> PMUGRF_GPIO0A5_IOMUX_SHIFT) & GRF_IOMUX_2BIT_MASK) != GRF_IOMUX_GPIO) {
    iomux = BITS_WITH_WMASK(GRF_IOMUX_GPIO, GRF_IOMUX_2BIT_MASK, PMUGRF_GPIO0A5_IOMUX_SHIFT);
    mmio_write_32(PMUGRF_BASE + PMUGRF_GPIO0A_IOMUX, iomux);
  }
  udelay(1);
  iomux = mmio_read_32(PMUGRF_BASE + PMUGRF_GPIO0A_IOMUX);
  udelay(1);
  for (int i = 0; i < 32; ++i) {
    cpi_fan_bit(iomux & 1);
    iomux >>= 1;
  }

It comes back:

RUN #05, fix bugs... write enable bits should go together with write bits.

raw:
0100 0000 0000 0000 0000 0000 0000 0000

Bits 11:10 cleared. Still no wakeup.

Try the read-evaluate-fan loop again:

  gpio_set_direction(5, GPIO_DIR_IN);
  gpio_set_pull(5, GPIO_PULL_UP);
  udelay(1);

  while (true) {
    int v = gpio_get_value(5);
    gpio_set_value(96, v);
    udelay(100000);
  }

The read-evaluate-fan loop now outputs a constant 1 instead of 0.
It means PWR_KEY_L never send anything other than HIGH??

According to the schematics, PWR_KEY_L → Diode → PWRON → R114 → Power SW → GND
I have measured R114 to be 1.8V/0V for power key.
rk3399 GPIO should be 3.3V, the barrier diode should have 0.2V voltage drop.
PWRON measures 1.8v, I don’t know what does that mean…

Update: R124 on my board is NC. What, the, frap?

After connecting R124, the fan loop is good. Now try suspend instead of the loop…
After suspend, push power button, PMU goes into a much higher pitch. It definitely did something. Not resuming, though.

~Update: Connecting R124 may have unwanted side effects.~

I measure some 100mA extra current draw, PWRON measures 2.5v instead of 1.8v, and the machine is hot. So I have disconnected it for now.
Maybe just my illusions? After disconnection it’s still that hot -___-"
So I connected it back :smiley:

Update: Got BL31 serial console working!

Now we can do some thorough examination without the fan loop…

  • Nothing has been passed to BL31 in bl_aux_params_parse, phew!!
  • BL31 debug console is lost after Linux kernel boots.
  • Tried to retain PWM2 during sleep. No good.

Extracted the PMU OS REG values:

NOTICE:  osreg0 = 5242c300
NOTICE:  osreg1 = 0
NOTICE:  osreg2 = 3aa1faa1
NOTICE:  osreg3 = 20000005

The closed-source ATF will examine OSREG2 for DDR-related configuration.

                    /* DDR_RET */
    if ((rk_sleepmode_config >> 3 & 1) != 0) {
      if ((_PMUGRF_OS_REG2 >> 0xd & 7) == 7) {
        uVar6 = 0x20660000;
      }
      else {
        uVar6 = 0x20770000;
      }
                    /* uVar7 = 0x34663E71 or 0x34773E71
                        */
      uVar7 = uVar7 | uVar6;
    }

This is critical for sys_slp_cfg power mode configuration:

  // rkbin default:  
  // 0x1477bf39
  // 0b1010001110111 1011111100111001
  //      xx (reserved)
  // "manually" executed with PBP example:
  // 0x1477bf79 or 0x1466bf79
  // 0b1010001110111 1011111101111001
  // 0b1010001100110 1011111101111001
  //      xx   ^   ^
		       
  // upstream  rkbin 77  66
	// 1         1     1   1   PMU_PWR_MODE_EN
	// 1         0     0   0   PMU_WKUP_RST_EN,
	// 1         0     0   0   PMU_INPUT_CLAMP_EN,
	// 1         1     1   1   PMU_OSC_DIS,

	// 1         1     1   1   PMU_ALIVE_USE_LF,
	// 1         1     1   1   PMU_PMU_USE_LF,
	// 1         0     1   1   PMU_POWER_OFF_REQ_CFG,
	// 0         0     0   0   PMU_CHIP_PD_EN,

	// 1         1     1   1   PMU_PLL_PD_EN,
	// 1         1     1   1   PMU_CPU0_PD_EN,
	// 1         1     1   1   PMU_L2_FLUSH_EN,
	// 1         1     1   1   PMU_L2_IDLE_EN,

	// 1         1     1   1   PMU_SCU_PD_EN,
	// 1         1     1   1   PMU_CCI_PD_EN,
	// 1         0     0   0   PMU_PERILP_PD_EN,
	// 1         1     1   1   PMU_CENTER_PD_EN,

	// 1         1     1   0   PMU_SREF0_ENTER_EN,  <----
	// 1         1     1   1   PMU_DDRC0_GATING_EN,
	// 1         1     1   1   PMU_DDRIO0_RET_EN,
	// 1         0     0   0   PMU_DDRIO0_RET_DE_REQ,

	// 1         1     1   0   PMU_SREF1_ENTER_EN,  <----
	// 1         1     1   1   PMU_DDRC1_GATING_EN,
	// 1         1     1   1   PMU_DDRIO1_RET_EN,
	// 1         0     0   0   PMU_DDRIO1_RET_DE_REQ,

  // x
  // x
	// 1         1     1   1   PMU_CLK_CENTER_SRC_GATE_EN = 26,
	// 1         0     0   0   PMU_CLK_PERILP_SRC_GATE_EN,

	// 1         1     1   1   PMU_CLK_CORE_SRC_GATE_EN,
	// 0         0     0   0   PMU_DDRIO_RET_HW_DE_REQ,
	// 0         0     0   0   PMU_SLP_OUTPUT_CFG,
	// 0         0     0   0   PMU_MAIN_CLUSTER,

May also need to work this out:

rkbin psci_power_down_wfi

void rockchip_soc_sys_pd_pwr_dn_wfi(void)

{
  undefined *puVar1;
  
  psci_power_down_wfi();
  puVar1 = (undefined *)FUN_000414ec();
  if (puVar1 == (undefined *)0x0) {
    rk_printf("WARNING: Do nothing when system off\n");
  }
  else {
    if (*(int *)(puVar1 + 4) == 0x26) {
      _PMUGRF_GPIO1A_IOMUX = 0x30000000;
    }
    FUN_00040d60(*(undefined4 *)(puVar1 + 4),0);
    FUN_00040d80(*(undefined4 *)(puVar1 + 4),*puVar1);
  }
  FUN_00040c6c(2000);
  do {
                    /* WARNING: Do nothing block with infinite loop */
  } while( true );
}

void psci_power_down_wfi(void)

{
  DataSynchronizationBarrier(3,3);
  if ((_PMUGRF_OS_REG2 & 0xe000) == 0xe000) {
    do {
    } while ((_DAT_ffa8032c & 0x10 & _DAT_ffa8832c & 0x10) == 0);
  }
  WaitForInterrupt();
  do {
                    /* WARNING: Do nothing block with infinite loop */
  } while( true );
}

upstream version

void __dead2 rockchip_soc_sys_pd_pwr_dn_wfi(void)
{
	psci_power_down_wfi(); // huh??
}

func psci_power_down_wfi
	dsb	sy		// ensure write buffer empty
	wfi
	no_ret	plat_panic_handler
endfunc psci_power_down_wfi

Something is definitely missing.

FFA8032C is:
#define DDRC0_BASE (MMIO_BASE + 0x07A80000)

Update: The reason that the console is unavailable after kernel boots is that the dbg pins are shared with sdmmc.

Tried to spam a function to forcibly enable the console, end up not having a rootfs. Oh well.

Update: big difference upstream vs. rkbin: rkbin does not have the pmusram section at all

There are two pieces of sram in rk3399:

  • sram 192KB@0xff8c0000: stores m0 binary, sram text, data, stack, DATA LOST DURING SUSPEND
  • pmusram 8KB@0xff3b0000: stores m0 pmu binary, pmusram text, data, DATA KEPT DURING SUSPEND

The sram 192KB section does not persist across suspension, so it must be saved to DDR before suspend and resumed afterwards.

Upon suspend, BOTH versions configure (SGRF_SOC_CON(1), ff33c004) the chip to boot from pmusram entry point (ff3b0000), but rkbin does not have that section??.

OSREG2 = 0x3aa1faa1

	sdram_config.dramtype = SYS_REG_DEC_DDRTYPE(os_reg2_val);
	sdram_config.num_channels = SYS_REG_DEC_NUM_CH(os_reg2_val);
	sdram_config.stride = (mmio_read_32(SGRF_BASE + SGRF_SOC_CON3_7(4)) >>
				10) & 0x1f;

So:

  • dramtype = 0x7 (LPDDR4)
  • num_channels = 2
  • ch1 rank = 2
  • ch1 col = 0xc
  • ch1 bk=1
2 Likes

Yay, progress!

The farthest I ever got was:

Update: curiosity aside, I want working suspend for everyone first…

Trying to get the closed-source ATF to properly suspend.
After fitting everything in… it seems even worse. Won’t go into deep sleep. Oh well.

Got latest kernel (5.18-9) + latest uboot + Rockchip ATF running.
Can confirm it is the Rockchip ATF handling suspend, because it prints debug info to serial console that contains strings only from rkbin version (verified with ghidra).

Here’s the log for a suspend run:

yatli@devterm ~ % sudo systemctl suspend
yatli@devterm ~ % [   68.911090] ieee80211 phy0: brcmf_netdev_set_mac_address: Setting cur_etheraddr failed, -52
[   69.619210] PM: suspend entry (deep)
[   70.165212] Filesystems sync: 0.541 seconds
[   70.545696] Freezing user space processes ... (elapsed 0.002 seconds) done.
[   70.556145] OOM killer disabled.
[   70.559762] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[   70.924834] vcc3v0_sd: Entering suspend 3, disabling forcibly, was on
[   70.951337] PM: suspend devices took 0.380 seconds
[   70.962615] ------------[ cut here ]------------
[   70.967802] clk_spi4 already disabled
[   70.972004] WARNING: CPU: 1 PID: 826 at drivers/clk/clk.c:971 clk_core_disable+0xb0/0xc0
[   70.981088] Modules linked in: nft_nat nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun zstd zstd_compress bnep 8021q garp mrp stp llc snd_soc_hdmi_codec btsdio cdc_acm rockchip_vdec(C) hci_uart hantro_vpu(C) brcmfmac btqca btrtl brcmutil btbcm btintel v4l2_vp9 snd_usb_audio snd_usbmidi_lib v4l2_h264 snd_hwdep cfg80211 snd_rawmidi bluetooth snd_seq_device panfrost videobuf2_dma_contig v4l2_mem2mem dw_hdmi_i2s_audio snd_soc_es8328_spi videobuf2_memops videobuf2_v4l2 gpu_sched ecdh_generic dw_hdmi_cec videobuf2_common snd_soc_es8328_i2c ecc snd_soc_simple_card drm_shmem_helper snd_soc_es8328 rfkill snd_soc_rockchip_i2s snd_soc_simple_amplifier snd_soc_simple_card_utils gpio_keys axp20x_adc axp20x_battery axp20x_pek dw_wdt rockchip_thermal spidev dm_multipath crypto_user ocp8178_bl panel_cwd686 rockchipdrm drm_cma_helper analogix_dp drm_dp_helper dw_hdmi cec rc_core dw_mipi_dsi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
[   70.981744]  rtc_rk808
[   71.081810] CPU: 1 PID: 826 Comm: systemd-sleep Tainted: G        WC        5.18.9-1-ARCH-ARM #1
[   71.091649] Hardware name: Clockworkpi A06 (DT)
[   71.096718] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   71.104517] pc : clk_core_disable+0xb0/0xc0
[   71.109208] lr : clk_core_disable+0xb0/0xc0
[   71.113896] sp : ffff80000b1d39e0
[   71.117604] x29: ffff80000b1d39e0 x28: ffff0000007de0f8 x27: 0000000000000002
[   71.125611] x26: ffff80000886ebe0 x25: ffff8000098021e4 x24: ffff800009941510
[   71.133617] x23: 0000000000000000 x22: ffff8000099e5000 x21: ffff000004b2db00
[   71.141621] x20: ffff000000590500 x19: ffff000000590500 x18: ffffffffffffffff
[   71.149627] x17: 000000040044ffff x16: 00000032b5503510 x15: 0000000000000006
[   71.157631] x14: 0000000000000000 x13: ffff8000098028c8 x12: 0000000000000783
[   71.165635] x11: 0000000000000281 x10: ffff8000098b28c8 x9 : ffff8000098028c8
[   71.173631] x8 : 00000000ffffdfff x7 : ffff8000098b28c8 x6 : 80000000ffffe000
[   71.181635] x5 : ffff0000f775cb90 x4 : 0000000000000000 x3 : 0000000000000027
[   71.189638] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00000725bd00
[   71.197643] Call trace:
[   71.200369]  clk_core_disable+0xb0/0xc0
[   71.204673]  clk_disable+0x34/0x50
[   71.208490]  rockchip_spi_suspend+0x34/0x70
[   71.213185]  dpm_run_callback+0x50/0xf0
[   71.217492]  __device_suspend_noirq+0x6c/0x1f0
[   71.222478]  dpm_noirq_suspend_devices+0x13c/0x23c
[   71.227843]  dpm_suspend_noirq+0x24/0x9c
[   71.232231]  suspend_enter+0xac/0x410
[   71.236342]  suspend_devices_and_enter+0x158/0x1f0
[   71.241717]  enter_state+0x144/0x1f0
[   71.245727]  pm_suspend+0x58/0xd0
[   71.249448]  state_store+0x80/0xec
[   71.253266]  kobj_attr_store+0x1c/0x30
[   71.257475]  sysfs_kf_write+0x48/0x60
[   71.261586]  kernfs_fop_write_iter+0x120/0x1f0
[   71.266569]  new_sync_write+0xd0/0x160
[   71.270777]  vfs_write+0x240/0x2a0
[   71.274595]  ksys_write+0x6c/0x100
[   71.278415]  __arm64_sys_write+0x20/0x30
[   71.282808]  invoke_syscall+0x48/0x114
[   71.287016]  el0_svc_common.constprop.0+0x44/0xec
[   71.292292]  do_el0_svc+0x24/0x84
[   71.296011]  el0_svc+0x2c/0x84
[   71.299440]  el0t_64_sync_handler+0x10c/0x140
[   71.304328]  el0t_64_sync+0x18c/0x190
[   71.308434] ---[ end trace 0000000000000000 ]---
[   71.313764] ------------[ cut here ]------------
[   71.319130] clk_spi4 already unprepared
[   71.323449] WARNING: CPU: 4 PID: 826 at drivers/clk/clk.c:829 clk_core_unprepare+0xf0/0x110
[   71.332798] Modules linked in: nft_nat nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun zstd zstd_compress bnep 8021q garp mrp stp llc snd_soc_hdmi_codec btsdio cdc_acm rockchip_vdec(C) hci_uart hantro_vpu(C) brcmfmac btqca btrtl brcmutil btbcm btintel v4l2_vp9 snd_usb_audio snd_usbmidi_lib v4l2_h264 snd_hwdep cfg80211 snd_rawmidi bluetooth snd_seq_device panfrost videobuf2_dma_contig v4l2_mem2mem dw_hdmi_i2s_audio snd_soc_es8328_spi videobuf2_memops videobuf2_v4l2 gpu_sched ecdh_generic dw_hdmi_cec videobuf2_common snd_soc_es8328_i2c ecc snd_soc_simple_card drm_shmem_helper snd_soc_es8328 rfkill snd_soc_rockchip_i2s snd_soc_simple_amplifier snd_soc_simple_card_utils gpio_keys axp20x_adc axp20x_battery axp20x_pek dw_wdt rockchip_thermal spidev dm_multipath crypto_user ocp8178_bl panel_cwd686 rockchipdrm drm_cma_helper analogix_dp drm_dp_helper dw_hdmi cec rc_core dw_mipi_dsi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
[   71.333021]  rtc_rk808
[   71.433058] CPU: 4 PID: 826 Comm: systemd-sleep Tainted: G        WC        5.18.9-1-ARCH-ARM #1
[   71.442871] Hardware name: Clockworkpi A06 (DT)
[   71.447929] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   71.455710] pc : clk_core_unprepare+0xf0/0x110
[   71.460676] lr : clk_core_unprepare+0xf0/0x110
[   71.465640] sp : ffff80000b1d39e0
[   71.469338] x29: ffff80000b1d39e0 x28: ffff0000007de0f8 x27: 0000000000000002
[   71.477318] x26: ffff80000886ebe0 x25: ffff8000098021e4 x24: ffff800009941510
[   71.485296] x23: 0000000000000000 x22: ffff8000099e5000 x21: ffff000004b2db00
[   71.493274] x20: ffff000001152600 x19: ffff000000590500 x18: ffffffffffffffff
[   71.501252] x17: 000000040044ffff x16: 00000034b5503510 x15: 0000000000000006
[   71.509231] x14: 0000000000000000 x13: ffff8000098028c8 x12: 0000000000000813
[   71.517209] x11: 00000000000002b1 x10: ffff8000098b28c8 x9 : ffff8000098028c8
[   71.525187] x8 : 00000000ffffdfff x7 : ffff8000098b28c8 x6 : 80000000ffffe000
[   71.533165] x5 : ffff0000f7795b90 x4 : 0000000000000000 x3 : 0000000000000027
[   71.541142] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00000725bd00
[   71.549120] Call trace:
[   71.551837]  clk_core_unprepare+0xf0/0x110
[   71.556414]  clk_unprepare+0x30/0x50
[   71.560408]  rockchip_spi_suspend+0x3c/0x70
[   71.565082]  dpm_run_callback+0x50/0xf0
[   71.569370]  __device_suspend_noirq+0x6c/0x1f0
[   71.574337]  dpm_noirq_suspend_devices+0x13c/0x23c
[   71.579691]  dpm_suspend_noirq+0x24/0x9c
[   71.584071]  suspend_enter+0xac/0x410
[   71.588165]  suspend_devices_and_enter+0x158/0x1f0
[   71.593518]  enter_state+0x144/0x1f0
[   71.597511]  pm_suspend+0x58/0xd0
[   71.601212]  state_store+0x80/0xec
[   71.605010]  kobj_attr_store+0x1c/0x30
[   71.609199]  sysfs_kf_write+0x48/0x60
[   71.613289]  kernfs_fop_write_iter+0x120/0x1f0
[   71.618254]  new_sync_write+0xd0/0x160
[   71.622443]  vfs_write+0x240/0x2a0
[   71.626243]  ksys_write+0x6c/0x100
[   71.630043]  __arm64_sys_write+0x20/0x30
[   71.634427]  invoke_syscall+0x48/0x114
[   71.638615]  el0_svc_common.constprop.0+0x44/0xec
[   71.643872]  do_el0_svc+0x24/0x84
[   71.647573]  el0_svc+0x2c/0x84
[   71.650985]  el0t_64_sync_handler+0x10c/0x140
[   71.655853]  el0t_64_sync+0x18c/0x190
[   71.659941] ---[ end trace 0000000000000000 ]---
[   71.665191] ------------[ cut here ]------------
[   71.670346] pclk_spi4 already disabled
[   71.674541] WARNING: CPU: 4 PID: 826 at drivers/clk/clk.c:971 clk_core_disable+0xb0/0xc0
[   71.683589] Modules linked in: nft_nat nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun zstd zstd_compress bnep 8021q garp mrp stp llc snd_soc_hdmi_codec btsdio cdc_acm rockchip_vdec(C) hci_uart hantro_vpu(C) brcmfmac btqca btrtl brcmutil btbcm btintel v4l2_vp9 snd_usb_audio snd_usbmidi_lib v4l2_h264 snd_hwdep cfg80211 snd_rawmidi bluetooth snd_seq_device panfrost videobuf2_dma_contig v4l2_mem2mem dw_hdmi_i2s_audio snd_soc_es8328_spi videobuf2_memops videobuf2_v4l2 gpu_sched ecdh_generic dw_hdmi_cec videobuf2_common snd_soc_es8328_i2c ecc snd_soc_simple_card drm_shmem_helper snd_soc_es8328 rfkill snd_soc_rockchip_i2s snd_soc_simple_amplifier snd_soc_simple_card_utils gpio_keys axp20x_adc axp20x_battery axp20x_pek dw_wdt rockchip_thermal spidev dm_multipath crypto_user ocp8178_bl panel_cwd686 rockchipdrm drm_cma_helper analogix_dp drm_dp_helper dw_hdmi cec rc_core dw_mipi_dsi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
[   71.683795]  rtc_rk808
[   71.783839] CPU: 4 PID: 826 Comm: systemd-sleep Tainted: G        WC        5.18.9-1-ARCH-ARM #1
[   71.793660] Hardware name: Clockworkpi A06 (DT)
[   71.798718] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   71.806498] pc : clk_core_disable+0xb0/0xc0
[   71.811169] lr : clk_core_disable+0xb0/0xc0
[   71.815840] sp : ffff80000b1d39e0
[   71.819538] x29: ffff80000b1d39e0 x28: ffff0000007de0f8 x27: 0000000000000002
[   71.827516] x26: ffff80000886ebe0 x25: ffff8000098021e4 x24: ffff800009941510
[   71.835494] x23: 0000000000000000 x22: ffff8000099e5000 x21: ffff000004b2db00
[   71.843472] x20: ffff00000057ab00 x19: ffff00000057ab00 x18: ffffffffffffffff
[   71.851450] x17: 000000040044ffff x16: 00000034b5503510 x15: 0000000000000006
[   71.859428] x14: 0000000000000000 x13: ffff8000098028c8 x12: 00000000000008a3
[   71.867405] x11: 00000000000002e1 x10: ffff8000098b28c8 x9 : ffff8000098028c8
[   71.875383] x8 : 00000000ffffdfff x7 : ffff8000098b28c8 x6 : 80000000ffffe000
[   71.883361] x5 : ffff0000f7795b90 x4 : 0000000000000000 x3 : 0000000000000027
[   71.891338] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00000725bd00
[   71.899316] Call trace:
[   71.902034]  clk_core_disable+0xb0/0xc0
[   71.906317]  clk_disable+0x34/0x50
[   71.910115]  rockchip_spi_suspend+0x48/0x70
[   71.914790]  dpm_run_callback+0x50/0xf0
[   71.919076]  __device_suspend_noirq+0x6c/0x1f0
[   71.924041]  dpm_noirq_suspend_devices+0x13c/0x23c
[   71.929394]  dpm_suspend_noirq+0x24/0x9c
[   71.933775]  suspend_enter+0xac/0x410
[   71.937867]  suspend_devices_and_enter+0x158/0x1f0
[   71.943219]  enter_state+0x144/0x1f0
[   71.947212]  pm_suspend+0x58/0xd0
[   71.950914]  state_store+0x80/0xec
[   71.954711]  kobj_attr_store+0x1c/0x30
[   71.958900]  sysfs_kf_write+0x48/0x60
[   71.962990]  kernfs_fop_write_iter+0x120/0x1f0
[   71.967953]  new_sync_write+0xd0/0x160
[   71.972140]  vfs_write+0x240/0x2a0
[   71.975939]  ksys_write+0x6c/0x100
[   71.979738]  __arm64_sys_write+0x20/0x30
[   71.984120]  invoke_syscall+0x48/0x114
[   71.988309]  el0_svc_common.constprop.0+0x44/0xec
[   71.993564]  do_el0_svc+0x24/0x84
[   71.997267]  el0_svc+0x2c/0x84
[   72.000678]  el0t_64_sync_handler+0x10c/0x140
[   72.005545]  el0t_64_sync+0x18c/0x190
[   72.009634] ---[ end trace 0000000000000000 ]---
[   72.014836] ------------[ cut here ]------------
[   72.019995] pclk_spi4 already unprepared
[   72.024408] WARNING: CPU: 4 PID: 826 at drivers/clk/clk.c:829 clk_core_unprepare+0xf0/0x110
[   72.033749] Modules linked in: nft_nat nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun zstd zstd_compress bnep 8021q garp mrp stp llc snd_soc_hdmi_codec btsdio cdc_acm rockchip_vdec(C) hci_uart hantro_vpu(C) brcmfmac btqca btrtl brcmutil btbcm btintel v4l2_vp9 snd_usb_audio snd_usbmidi_lib v4l2_h264 snd_hwdep cfg80211 snd_rawmidi bluetooth snd_seq_device panfrost videobuf2_dma_contig v4l2_mem2mem dw_hdmi_i2s_audio snd_soc_es8328_spi videobuf2_memops videobuf2_v4l2 gpu_sched ecdh_generic dw_hdmi_cec videobuf2_common snd_soc_es8328_i2c ecc snd_soc_simple_card drm_shmem_helper snd_soc_es8328 rfkill snd_soc_rockchip_i2s snd_soc_simple_amplifier snd_soc_simple_card_utils gpio_keys axp20x_adc axp20x_battery axp20x_pek dw_wdt rockchip_thermal spidev dm_multipath crypto_user ocp8178_bl panel_cwd686 rockchipdrm drm_cma_helper analogix_dp drm_dp_helper dw_hdmi cec rc_core dw_mipi_dsi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
[   72.033946]  rtc_rk808
[   72.133987] CPU: 4 PID: 826 Comm: systemd-sleep Tainted: G        WC        5.18.9-1-ARCH-ARM #1
[   72.143806] Hardware name: Clockworkpi A06 (DT)
[   72.148863] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   72.156643] pc : clk_core_unprepare+0xf0/0x110
[   72.161606] lr : clk_core_unprepare+0xf0/0x110
[   72.166569] sp : ffff80000b1d39e0
[   72.170267] x29: ffff80000b1d39e0 x28: ffff0000007de0f8 x27: 0000000000000002
[   72.178237] x26: ffff80000886ebe0 x25: ffff8000098021e4 x24: ffff800009941510
[   72.186215] x23: 0000000000000000 x22: ffff8000099e5000 x21: ffff000004b2db00
[   72.194192] x20: ffff000004b2d280 x19: ffff00000057ab00 x18: ffffffffffffffff
[   72.202170] x17: 000000040044ffff x16: 00000034b5503510 x15: 0000000000000006
[   72.210148] x14: 0000000000000000 x13: ffff8000098028c8 x12: 0000000000000933
[   72.218126] x11: 0000000000000311 x10: ffff8000098b28c8 x9 : ffff8000098028c8
[   72.226103] x8 : 00000000ffffdfff x7 : ffff8000098b28c8 x6 : 80000000ffffe000
[   72.234081] x5 : 0000000000000000 x4 : ffff0000f7795b90 x3 : ffff0000f7798b38
[   72.242059] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00000725bd00
[   72.250036] Call trace:
[   72.252754]  clk_core_unprepare+0xf0/0x110
[   72.257329]  clk_unprepare+0x30/0x50
[   72.261321]  rockchip_spi_suspend+0x50/0x70
[   72.265995]  dpm_run_callback+0x50/0xf0
[   72.270280]  __device_suspend_noirq+0x6c/0x1f0
[   72.275245]  dpm_noirq_suspend_devices+0x13c/0x23c
[   72.280598]  dpm_suspend_noirq+0x24/0x9c
[   72.284977]  suspend_enter+0xac/0x410
[   72.289068]  suspend_devices_and_enter+0x158/0x1f0
[   72.294421]  enter_state+0x144/0x1f0
[   72.298413]  pm_suspend+0x58/0xd0
[   72.302114]  state_store+0x80/0xec
[   72.305912]  kobj_attr_store+0x1c/0x30
[   72.310101]  sysfs_kf_write+0x48/0x60
[   72.314190]  kernfs_fop_write_iter+0x120/0x1f0
[   72.319153]  new_sync_write+0xd0/0x160
[   72.323342]  vfs_write+0x240/0x2a0
[   72.327143]  ksys_write+0x6c/0x100
[   72.330941]  __arm64_sys_write+0x20/0x30
[   72.335323]  invoke_syscall+0x48/0x114
[   72.339512]  el0_svc_common.constprop.0+0x44/0xec
[   72.344767]  do_el0_svc+0x24/0x84
[   72.348469]  el0_svc+0x2c/0x84
[   72.351879]  el0t_64_sync_handler+0x10c/0x140
[   72.356746]  el0t_64_sync+0x18c/0x190
[   72.360834] ---[ end trace 0000000000000000 ]---
[   72.366327] Disabling non-boot CPUs ...
[   72.375590] psci: CPU1 killed (polled 0 ms)
[   72.384281] psci: CPU2 killed (polled 10 ms)
[   72.393286] psci: CPU3 killed (polled 0 ms)
[   72.399586] psci: CPU4 killed (polled 0 ms)
[   72.406698] psci: CPU5 killed (polled 0 ms)
INFO:    sleep mode config[0xde]:
INFO:           AP_PWROFF
INFO:           SLP_ARMPD
INFO:           SLP_PLLPD
INFO:           DDR_RET
INFO:           SLP_CENTER_PD
INFO:    wakeup source config[0x4]:
INFO:           GPIO interrupt can wakeup system
INFO:    PWM CONFIG[0x4]:
INFO:           PWM: PWM2D_REGULATOR_EN
INFO:    APIOS info[0x0]:
INFO:           not config
INFO:    GPIO POWER INFO:
INFO:           GPIO1_C1
INFO:           GPIO1_B6
INFO:    PMU_MODE_CONG: 0x1466bf71

Power saving isn’t as good as upstream ATF because not all power flags are enabled.
Edit: if rockchip,sleep-debug-en is disabled then power saving is really good.
Upon wakeup, current jumps back to ~0.5A but there’s no response from serial console yet.

This patch reportedly solved suspend problem for PinePhonePro:

https://patchwork.ozlabs.org/project/uboot/patch/20220222013131.3114990-3-pgwipeout@gmail.com/

Source: March Update: Introducing the QuartzPro64 | PINE64

Tried that, didn’t work. :confused: