title: How to design a JMS message containing large amounts of data
body: <p>I am working on designing a system that uses an ETL tool to retrieve batches of data, i.e., insert/update/deletes for one or more tables, and puts them on a JMS topic to be processed later by multiple clients. Right now, each message on the topic represents a single record I/U/D and we have a special message to delimit the end of the batch. It's important to process the batches in a single transaction, so having a bunch of messages delimited by a special one is not ideal: both sessions publishing and receiving messages must be designed for multiple messages; the batch delimiter message is a messy solution (each time we receive a message we need to check if it's the last) and very error prone; the system is difficult to debug and maintain; the number of messages on the topic becomes quickly huge (up to millions).</p>
  
  <p>Now, I think that the next natural step to improve the architecture is to pack all the records in a single JMS message so that when a message is received, it encompasses a single transaction, it's easy to detect failures, there are no "orphan" records on the topic, etc. I only see advantages in doing so! Now here are my questions:</p>
  
  <ul>
  <li>What's the best way to create such a packed message? I think my choices are <code>StreamMessage</code>, <code>ByteMessage</code> or <code>ObjectMessage</code>. I excluded text and map messages because the first will require text parsing, which will kill performance, and I assume the second one doesn't really seem to fit the scenario. I'm kinda leaning towards <code>StreamMessage</code> because it seems quite compact although it will require a lot of work writing custom serialization code (even worse for ByteMessage). Not sure about ObjectMessage, how does it perform? Is there an out of the box solution for this?</li>
  <li>What's the maximum size allowed per message? Could it be in the order of hundreds of KB or even few MB?</li>
  </ul>
  
  <p>Thanks for the thoughts!</p>
  
  <p>Giovanni</p>