title: Spring-Boot logging to Kafka: how to eliminate warning; best practices
body: <p>I was asked to how to capture logging output from a Spring Boot application that runs as a Java action from Oozie.</p>
  
  <p>My initial thought was that it would be possible edit some log4j properties to capture the application logs inside YARN or Oozie. Then it occurred to me that that Kafka would be a much easier way to capture and aggregate log messages, for a specific application, running on various cluster nodes. It's much easier to monitor a distributed system by subscribing to a topic than fishing through log files.</p>
  
  <p>I notice that Kafka has a log4j appender and so I tried to create a minimal reproducible example (posted on github: <a href="https://github.com/alexwoolford/spring-boot-log-to-kafka-example" rel="noreferrer">https://github.com/alexwoolford/spring-boot-log-to-kafka-example</a>). Here's a snippet from <code>pom.xml</code>:</p>
  
  <pre><code>&lt;parent&gt;
      &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
      &lt;artifactId&gt;spring-boot-starter-parent&lt;/artifactId&gt;
      &lt;version&gt;1.4.4.RELEASE&lt;/version&gt;
  &lt;/parent&gt;
  
  &lt;dependencies&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
          &lt;artifactId&gt;spring-boot-starter&lt;/artifactId&gt;
          &lt;exclusions&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
                  &lt;artifactId&gt;spring-boot-starter-logging&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
                  &lt;artifactId&gt;logback-classic&lt;/artifactId&gt;
              &lt;/exclusion&gt;
          &lt;/exclusions&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
          &lt;artifactId&gt;spring-boot-starter-log4j&lt;/artifactId&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
          &lt;artifactId&gt;kafka-log4j-appender&lt;/artifactId&gt;
          &lt;version&gt;0.10.0.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;net.logstash.log4j&lt;/groupId&gt;
          &lt;artifactId&gt;jsonevent-layout&lt;/artifactId&gt;
          &lt;version&gt;1.7&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;commons-logging&lt;/groupId&gt;
          &lt;artifactId&gt;commons-logging&lt;/artifactId&gt;
          &lt;version&gt;1.2&lt;/version&gt;
      &lt;/dependency&gt;
  &lt;/dependencies&gt;
  </code></pre>
  
  <p>My log4j.properties file looks like this:</p>
  
  <pre><code>log4j.rootLogger=INFO
  log4j.appender.KAFKA=org.apache.kafka.log4jappender.KafkaLog4jAppender
  log4j.appender.KAFKA.layout=net.logstash.log4j.JSONEventLayoutV1
  log4j.appender.KAFKA.topic=logs
  log4j.appender.KAFKA.brokerList=hdp-single-node:6667
  log4j.appender.KAFKA.syncSend=true
  log4j.appender.KAFKA.producer.type=async
  log4j.logger.io.woolford=INFO, KAFKA
  </code></pre>
  
  <p>This works, except that it generates a warning:</p>
  
  <pre><code>log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.producer.ProducerConfig).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  </code></pre>
  
  <p>Even though this application runs and does what I need it to, the warnings suggest that I've misconfigured something. Can you see what needs to be changed?</p>
  
  <p>Also, I notice that Spring Boot, by default, uses <a href="https://logback.qos.ch/" rel="noreferrer" title="Logback">Logback</a> and I notice that there's an open source project, <a href="https://github.com/danielwegener/logback-kafka-appender" rel="noreferrer" title="logback-kafka-appender">logback-kafka-appender</a>, that allows Logback to append to Kafka. Is the Kafka log4j appender the best way for Spring Boot to log to Kafka? </p>