You can compile Retroarch and the cores from source and it’ll run but I find that performance kinda sucks on the CM4 if you’re running it from inside a desktop environment (especially if you use shaders).
I know this isn’t the answer you’re looking for but you’re better off using the Retropie image with a spare SD card. Performance will be better running directly on the framebuffer than if you try to run Retroarch in Wayland or X11, and everything’s already set up for you.