I wonder if you could get a any benefit by introducing two more MCUs.

Dedicate one MCU to simply reading and refreshing the RAM as fast as possible which can act as an abstraction layer for another MCU that it can talk to over I2C or SPI.

Then use a second MCU to act as the MMU and talk to the RAM MCU.

Finally, run Linux on the third MCU which talks to the MMU MCU.