What is the theoretical maximum RPM of a typical aircraft piston engine? Given a standard 6 cylinder horizontally opposed engine like the Continental IO-550 or Lycoming IO-540, the maximum RPM for those engines is 2700 RPM. That's due to keeping the propeller tip speed under the supersonic range.

But what is preventing these engines from turning at a higher RPM for another, non-aircraft use?

I'm thinking if they would work up to around 5000 RPM or so. Maybe the fuel mapping must be modified? Will the valvetrain start to float at that RPM? The application would be automotive, so the load would be far, far less from a duty cycle standpoint then an aircraft application: basically, 5000 RPM would only be seen for a second or two at a time under full (or nearly full) load.

Is there anything else I'm missing? I realize this is a weird question, just doing some thought experimenting.
The size and weight of the pistons is going to be a major hurdle. With a 5.25 in bore and a 4.25 inch stroke, the pistons are massive and don't change directions very easily—there's only so many Gs you can put a shape this large through at the top and bottom of each stroke before the metal starts deforming. Bounce an open umbrella up and down really fast, and you'll start to get the picture.

Cooling would also be a major factor for non-aviation use. Aircraft engines (more or less rightly) assume that there's going to be a large fan out front and/or 100+ mph of airflow. Liquid cooling would allow the heads to maintain a more constant temperature, but would also add weight and complexity.

If you want a faster-revving flat six that will put out more than 100hp and is suitable for automotive use, I might suggest pulling an engine out of a Honda Goldwing or Valkyrie. Given the edited description, though, I'm guessing you're thinking of a drag racer.