"Clean up by name" isn't sorting my files in alphabetical order I have a folder containing six subfolders. Right-clicking on the Finder background and clicking "Clean up by name" sorts them into the order "1.0", "1.5", "1.6", "1.25", "1.56", and "2.0". This is quite annoying, because that's neither in any lexicographic order (as I'd expected) nor in increasing numerical order, so it always takes a moment to find the right subfolder I want. What order does "Clean up by name" sort files into?

OS X Yosemite v10.10.5
Selecting Clean Up By Name sorts things by alphanumeric order1. 

Using your example, you're actually mistaken in saying that the items aren't listed in some sort of increasing numerical order. The order you provided is:

 1.0, 1.5, 1.6, 1.25, 1.56, and 2.0.


However, the 4th and 5th values above can be interpreted as one point twenty-five and one point fifty-six. So, 25 is larger than 6, and 56 is larger than 25, and so on.

By way of comparison, MS Windows does the same thing.

So, why does this happen? 

Well, just like databases and spreadsheets, most operating systems use strings to store data, in this case filenames.  Normally this works pretty well, but in examples like yours it is counterintuitive and that's why many people end up using naming conventions that add zeros to a filename (for example, instead of 1.5 it'd be 1.50 instead).

Recently I had to set up a workflow for an organisation that was going to produce an animation every Wednesday for the next three years. They wanted their files to be called NewsImage1, NewsImage2, etc. I had to explain to them it needed to be NewsImage001, NewsImage002 and so on so that by the time they were finished the animations would still be listed in order from NewsImage001 to NewsImage156. (If it were up to me the filenames would have used a dating scheme such as NewsImage201701 and so on, but alas it wasn't my call).

This sorting is also known to cause problems in certain industries, but as far as I know the only workaround in macOS is to add the zeroes to filenames as I've described above. In Windows you used to be able to manually edit a registry key, but I'm not sure that's still possible.

For what it's worth, I've always thought it should be an option in System Preferences somewhere, but I suspect it's not provided for compatibility purposes.

1 By Alphanumeric I mean it's a combination of alphabetic (Latin) characters and numeric (Arabic) digits that also takes into account symbols (e.g. punctuation marks, mathematical signs, etc). So, ordering works on the basis of symbols/special characters first, followed by numerals in order, followed by letters in order. I do not recall the order of sorting for the special characters, but for numerals anything after a decimal point is a whole number as opposed to a fraction. By this I mean that 1.1 is lower than 1.2, however 1.15 is higher than 1.2 because the '15' in 1.15 is seen as fifteen and the '2' in 1.2 is seen as two (not as twenty).