If you want a clean install, go for a carefully curated set of packages, rather than trying to mix and match to create your own selection - that’s bound to result in a Frankenstein installation.

I’m partial to Fedora Silverblue, which is essentially just a single package containing the things you need to have a usable desktop. You can install what you need separately on top of that, but on updates, the whole base gets replaced wholesale - including, which is most relevant to your concerns, removing stuff that is no longer used/needed, rather than having that clog up over time.