How do I figure out what crankset replacement will fit my derailleur? I have a bike with a Shimano FC-M560 crankset in the front which I want to replace. I naively bought a Shimano FC-TY301 crankset, thinking that there's norms and that 3-speed cranksets are more or less interchangeable.

Fast forward; the distance between the new crankset's chainrings is larger than the old ones. Since the Shimano Deore LX derailleur doesn't go as close to the seat tube, the chain will rub against the cage when on the smallest chainring.

My question is: How do I select a part that will fit my particular derailleur without relying on the exact series? (The exact components are expensive to get at this point since they're not made anymore).

The service manual here seems to refer to a quantity named stroke, is that the relevant number here?
According to the Bikepro.com bottom bracket compatibility page, the M560 crank requires a 113mm x 68mm bottom bracket (113 mm is the spindle length), which is in agreement with your service manual (113 mm for a 47.5 chainline, or 118 for a 50 chainline).

The Shimano FC-TY301 crankset requires a BB-UN100 / BB-UN26 bottom bracket. The BB-UN100 only comes with a spindle length of 123 mm. 

What this means is that you're using a bottom bracket with a spindle length of 113 mm with a crankset that was designed to be used with a bottom bracket that has a spindle of 123 mm. Since the entire crankset it basically shifted a whole centimeter towards the frame, this could be causing the problems that you are describing.

Usually you can get away with using a different bottom bracket with a slightly shorter or longer spindle, but it appears that in your case you require the correct bottom bracket to make everything work properly.