title: Multiple different consumers of same Kinesis stream
body: <p>I have a Kinesis producer which writes a single type of message to a stream. I want to process this stream in multiple, completely different consumer applications. So, a pub/sub with a single publisher for a given topic/stream. I also want to make use of checkpointing to ensure that each consumer processes every message written to the stream.</p>
  
  <p>Initially, I was using the same App Name for all consumers and producers. However, I started getting the following error once I started more than one consumer:</p>
  
  <blockquote>
    <p>com.amazonaws.services.kinesis.model.InvalidArgumentException: StartingSequenceNumber 49564236296344566565977952725717230439257668853369405442 used in GetShardIterator on shard shardId-000000000000 in stream PackageCreated under account ************ is invalid because it did not come from this stream. (Service: AmazonKinesis; Status Code: 400; Error Code: InvalidArgumentException; Request ID: ..)</p>
  </blockquote>
  
  <p>This seems to be because consumers are clashing with their checkpointing as they are using the same App Name.</p>
  
  <p>From reading the documentation, it seems the only way to do pub/sub with checkpointing is by having a stream per consumer application, which requires each producer to know about all possible consumers. This is more tightly coupled than I want; it's really just a queue.</p>
  
  <p>It seems like Kafka supports what I want: arbitrary consumption of a given topic/partition, since consumers are completely in control of their own checkpointing. Is my only option to move to Kafka, or some other alternative, if I want pub/sub with checkpointing?</p>
  
  <p>My RecordProcessor code, which is identical in each consumer:</p>
  
  <pre><code>override def processRecords(processRecordsInput: ProcessRecordsInput): Unit = {
    log.trace("Received record(s) from kinesis")
    for {
      record &lt;- processRecordsInput.getRecords
      json   &lt;- jawn.parseByteBuffer(record.getData).toOption
      msg    &lt;- decode[T](json.toString).toOption
    } yield subscriber ! msg
    processRecordsInput.getCheckpointer.checkpoint()
  }
  </code></pre>
  
  <p>The code parses the message and sends it off to the subscriber. For now, I'm simply marking all messages as successfully received. I can see messages being sent on the AWS Kinesis dashboard, but no reads happen, presumably because each application has its own AppName and doesn't see any other messages.</p>