If you read the article, it is indeed full Linux because the 4004 is running a MIPS emulator that provides the necessary memory management features. Pretty much all of the “run Linux on some old chip incapable of running Linux” projects achieve it via emulating a more featured architecture that Linux supports, not by somehow compiling Linux to natively run on a 4 bit, MMU-less architecture.
Leave it on, but turn off the monitor. I have it set up as a GitLab runner for some projects and also want to be able to SSH/SFTP in to access files, run updates, etc.