People have tried BananaPis without success; I don’t have a BPi or a Milky-V (though I have a DevTerm with the R01 installed and it’s delightful) to test, but I imagine you’d need, at minimum, the DTB overlays and whatever kernel patches the Milky-V wants.
There’s also some variation in models, like the embedded eMMC, wifi, things like that. So it depends on how compatible with the CM4 the Milky-V Mars CM is. The eMMC models apparently require some fiddling to boot because they have difficulty finding the boot device although it appears that the A06 will boot from a USB drive (which I found out by accident when I rebooted after my microSD card failed; I was building a CRUX system on the uSD card plugged in by a USB adapter and apparently had built out enough of the system to get a login prompt).