Hello, @neocska92!
@QuantumKraken in “Fixing uConsole power delivery limitations with CM5” replaced a capacitor on the mainboard and looks like that fixed the issue of keyboard disconnecting when plugging a device to the USB-A port.
Would it be possible to include a similar capacitor upgrade in your design? It looks fairly simple and seems to solve the power-delivery limit.