Question: Has anyone had good experiences of talking direct to RPG programs running on a V5R4 iSeries machine from Java? If so, what are the recommendations of the community, and what pitfalls should I try to avoid?

From the various pieces of literature and spike solutions I have attempted it looks as though we can use ProgramCallBeans (either through PCML or xPCML), talking to the DataQueues (for asynchronous comms), or even JNI.

I'm looking for something that's robust, performant, quick to develop, easy to maintain, and easy to test (aren't we all!?!).

Answer: 
I've had some success with PCML documents. I decided to use PCML since encoding the commandcall into a string when passing parameters to an RPG program can get really ugly.

PCML allows you to somewhat transparently pass java data types to an rpg program as a parameter. The drawback is that the xml in the PCML doc becomes a static interface and must be updated if the program is ever updated. With the right build tools, it might be pretty straightforward to automate the update of the pcml xml, but right now I'm doing this manually.

I've used this approach when the rpg program needs to be called from java, and the logic flow is controlled by the java program.

In a case where the logic flow was controlled by an rpg program, I've used dataqueues for both synchronous and asynchronous calls to java. This required writing a significant amount of code to standardize on how to read and write to dataqueues in a coordinated manner from different programming languages
Hmm, I'm new here and would vote KC Baltz answer up, but cannot yet. Stored procedures are the way to go. I've used JT open to call programs natively and found issues with the number of parms that could be passed, issues with data types, etc. Once you have an SQL procedure wrapper around your program you'll find the Java support for SQL to be far superior than the java support for native 400 calls.