I finally decided to try another way to wake it up, by plugging in an external mouse and keyboard, and I discovered the system immediately woke up and was responsive.
Check powertop. One of the screens in it has a list of USB devices and whether their power management status puts them to sleep or not. If the internal keyboard/trackball go to sleep (or maybe are put to sleep by the OS/DE combination you are using, so if you are using a DE, check if it does any power management in its settings), but an external keyboard doesn’t, then that might be that the reason your device has trouble waking up.