What are the factors that limit how long an aircraft can stay aloft? The most obvious one is fuel. Many modern military aircraft, however, are capable of refueling in mid-air. Crew fatigue, another obvious one, can also be addressed by rotating a crew out, and is a non-issue in unmanned aircraft. 

The question therefore is, what factors would prevent an aircraft from staying in mid-air indefinitely? Are there any such factors that cannot be addressed, and would make it impossible to design an aircraft that could stay up indefinitely, or at least for an exceptionally long time?
Maintenance is probably the biggest issue. What types of maintenance is needed and the failures that not performing those steps will lead to depends on the design of the aircraft.

For example, on a piston engine used in a small aircraft:

The engine will need oil replaced at some point, and unless a system was created to cycle in new oil in-flight this is probably the first thing that will break down after crew and fuel (as you mentioned). Oil breaking down will lead to early failure of the engine.

Fuel and air filters will get clogged, leading to the engine shutting down.

Which leads to engine wear. Eventually piston rings, ignition systems, etc. will wear out.