title: Android Camera2 pipeline: How do I encode h.264 units using MediaCodec from an input Surface?
body: <p>I have an Android application using the Camera2 API. The ultimate goal is to get h264 units to write to a stream. So far I have</p>
  
  <ol>
  <li>Successfully created a capture session and can write to preview, local recording and streaming surfaces via:</li>
  </ol>
  
  <pre><code>    session.device.createCaptureRequest(CameraDevice.TEMPLATE_RECORD).run {
                      addTarget(previewSurface)
                      addTarget(recorder.surface)
                      addTarget(streamer.surface)
                      set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(args.fps, args.fps))
                      build()
                  }
  </code></pre>
  
  <ol start="2">
  <li>Set up a MediaCodec to encode data from the <code>streamer.suface</code> param above where the surface is derived from a call to <code>mediaCodec.createInputSurface</code> from the MediaCodec created as follows</li>
  </ol>
  
  <pre><code>    internal fun streamingCodec(args: CameraFragmentArgs): MediaCodec {
          val mediaFormat = MediaFormat.createVideoFormat("video/avc", args.width, args.height).apply {
              setInteger(MediaFormat.KEY_BIT_RATE, 2000 * 1024)
              setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 2)
              setInteger(MediaFormat.KEY_FRAME_RATE, args.fps)
              setInteger(
                  MediaFormat.KEY_COLOR_FORMAT,
                  MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface
              )
          }
  
          val encoderName = MediaCodecList(MediaCodecList.REGULAR_CODECS).findEncoderForFormat(mediaFormat)
  
          return MediaCodec.createByCodecName(encoderName).apply {
              configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
          }
      }
  </code></pre>
  
  <ol start="3">
  <li>Used the asynchronous callback for when the <code>surface</code> above has information in it's <code>ByteBuffer</code></li>
  </ol>
  
  <pre><code>private class StreamingCallBack) : MediaCodec.Callback() {
  
      override fun onInputBufferAvailable(codec: MediaCodec, index: Int) = Unit
  
      override fun onOutputBufferAvailable(codec: MediaCodec, index: Int, info: BufferInfo) {
  
         val byteBuffer = codec.getOutputBuffer(index)
         // Is the data in the buffer properly encoded as h.264 here? Did I need to use MediaExtractor?
      }
  
      override fun onOutputFormatChanged(codec: MediaCodec, format: MediaFormat) = Unit
  
      override fun onError(codec: MediaCodec, e: MediaCodec.CodecException) {
          Log.i("TEST", "onError in codec")
      }
  
  }
  </code></pre>
  
  <p>My confusion is, at this point, are the values in the  <code>ByteBuffer</code> encoded properly? Do I need to use <code>MediaExtractor</code> to clean up the data coming in from the input <code>Surface</code> before I pass it the <code>MediaCodec</code> to encode? The pipeline is clean enough, but I'm not quite sure what's necessary or not. This <a href="https://bigflake.com/mediacodec/" rel="nofollow noreferrer">document</a> has been the biggest guide, and it mentions the <code>MediaCodec</code> operates on raw data, which makes me think I need the <code>MediaExtractor</code>, but that doesn't take a <code>Surface</code> as an input making the right order of items in the pipeline all the more confusing.</p>