How to perform a "Clean Install" these days? Long ago, back in the OS 7 and earlier days, it was a snap to perform what was formally called a "Clean Install" of the OS.  It would utterly remove the OS and place a brand-spanking new one in its place.  Applications and data (other than system preferences and such) would be untouched, and you rarely even had to reinstall a single application.  Adjust a few system prefs and you were good as new in no time at all.

I believe this feature went away around OS 8, and since then I've never known of a good way to perform a clean install.  Migration Assistant is useless and ranting about it is unhelpful here.

I have a mid-2011 iMac, running "low" Sierra from an external SSD (Thunderbolt).  I've got vestiges of Mavericks and all kinds of applications I no longer use floating around, and I feel that it's time to clean up.  But every time I've done this in every flavor of OS X, it's ended up being an utter nightmare of lost application files (prefs, extensions, etc.), lost system settings, lost chat logs (Messages, Skype, etc.) and so much more.  It's as bad to clean my Mac these days as it always had been on Windows!

My approach to date has been to install a fresh OS on a newly created partition of my drive, and go through countless reboots back &amp; forth between the two systems while I copy applications and their preferences and settings over.  Still it's inevitable that something will get lost or not be able to transfer over like mail rules, chat histories, or all manner of other stuff.  Eventually I can delete the original partition once I'm confident in the migration (though there are often problems doing that, too!)

So is there a "best practice" or reliable way to completely replace my current macOS Sierra installation with a fresh copy, without losing any application data including preferences, mail &amp; chat histories, app settings, etc.?  I'm OK with having to reset basic system settings as I expect that, but losing anything related to my user-installed applications, is really quite unacceptable IMO.
The functionality you're looking for that is closest to the Clean Installation option in the old Mac OS 'System' days is the Reinstall macOS option available from Recovery mode.

See my A bit of history info at end on what the old Clean Installation option was and how it differs slightly from the Reinstall macOS option today.

Reinstalling macOS without deleting applications, data, etc

Prior to following this process, you need to ensure:


you have a known good backup of all your data (be it via Time Machine or another process) just in case something goes wrong
your iMac is able to connect to the internet during the installation process


Assuming you have the above sorted, follow these steps to reinstall macOS without deleting your apps, data, etc.


Shutdown your iMac
Power up your iMac and immediately press and hold the commandR keys
Let go of the keys when it's clear you're booting into Recovery mode
Soon the macOS Utilities window will appear
Make sure your iMac is connected to the internet
Now choose the Reinstall macOS option 
Click the Continue button
Follow the prompts (to select your disk, etc)
Click the Install button


Once completed, your iMac will restart as normal. This process does not erase your drive and therefore all your apps and data should still be in place. In other words, this process does nothing but reinstall macOS over the previous version on your drive.



A bit of history 

In the old Mac OS System days users were presented with three options when installing system software:


normal (“easy”) installation - this automatically installed all the Mac OS system software appropriate for that particular model of Mac
“clean” installation - (see more details below)
custom installation - this allowed users to select a specific combination of system software files or to install/update specific files


The old clean installation option was actually designed as a troubleshooting option. This allowed users to discover which item(s) in their original System Folder was causing problems. A clean installation created a brand new System Folder and saved everything in the original System Folder to a different location. It also left user data (and user installed apps) alone. Users could then reinstall system extensions, control panels, etc one at a time from the old System Folder to the new System Folder. This procedure allowed users to determine which item in the old System Folder was the source of a problem. 

In Apple's own words: 


  Perform a clean installation if you can’t determine what is damaged in
  your System Folder (especially if you think any special software, such
  as control panels, system extensions, or custom utilities, may be
  causing the problems you’re experiencing). You should also perform a
  clean installation if you’re still having problems with your computer
  after you’ve reinstalled system software by performing a normal
  installation.


Source: Mac OS 7.1 Installation manual 

Today a clean installation has a totally different meaning (even to Apple) to what it did back then. Today a clean installation involves erasing the Mac's startup drive and starting afresh with a totally new installation of macOS, meaning users have to rely on a backup and/or installers to transfer any data or software over. Today's Reinstall macOS option accessible via Recovery Mode is the closest option to replicating the old clean installation option.

Where this differs is that the old option actually kept everything in the original System Folder by copying it to another location so users could move system items (e.g. control panels, system extensions, etc) back one by one to try and determine what was causing problems. This was a troubleshooting process of elimination option.

Today's Reinstall macOS option is not designed so much for troubleshooting as it is simply to reinstall macOS over an existing installation without having to install/migrate apps, data etc. 

However, if you're installing a new drive or migrating to a new Mac, then users will need to use Time Machine / Migration Assistant instead (or another backup option) in order to migrate apps and data.