events.action: opened
events.author: seglo
events.masked_author: username_0
events.text: Hi @username_1.  I've run into a problem with the feature introduced in PR #61.  It appears that the Java prometheus client library does not let you register metrics more than once with different sets of labels.  I created this PR which recreates the issue, by modifying one of your tests to add a second set of custom labels for another cluster definition.  It results in the following exception (see the inner exception).
  
  ```
  Could not create Prometheus Endpoint
  java.lang.Exception: Could not create Prometheus Endpoint
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSink$.$anonfun$apply$2(PrometheusEndpointSink.scala:23)
  	at scala.util.Failure.fold(Try.scala:240)
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSink$.apply(PrometheusEndpointSink.scala:23)
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSinkTest.$anonfun$new$5(PrometheusEndpointSinkTest.scala:60)
  	at org.scalatest.fixture.TestSuite$TestFunAndConfigMap.$anonfun$apply$1(TestSuite.scala:132)
  	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  	at org.scalatest.fixture.TestSuite$TestFunAndConfigMap.apply(TestSuite.scala:132)
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSinkTest.withFixture(PrometheusEndpointSinkTest.scala:31)
  	at org.scalatest.fixture.FreeSpecLike.invokeWithFixture$1(FreeSpecLike.scala:540)
  	at org.scalatest.fixture.FreeSpecLike.$anonfun$runTest$1(FreeSpecLike.scala:551)
  	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
  	at org.scalatest.fixture.FreeSpecLike.runTest(FreeSpecLike.scala:551)
  	at org.scalatest.fixture.FreeSpecLike.runTest$(FreeSpecLike.scala:532)
  	at org.scalatest.fixture.FreeSpec.runTest(FreeSpec.scala:232)
  	at org.scalatest.fixture.FreeSpecLike.$anonfun$runTests$1(FreeSpecLike.scala:594)
  	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
  	at scala.collection.immutable.List.foreach(List.scala:392)
  	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
  	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:373)
  	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:410)
  	at scala.collection.immutable.List.foreach(List.scala:392)
  	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
  	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
  	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
  	at org.scalatest.fixture.FreeSpecLike.runTests(FreeSpecLike.scala:594)
  	at org.scalatest.fixture.FreeSpecLike.runTests$(FreeSpecLike.scala:593)
  	at org.scalatest.fixture.FreeSpec.runTests(FreeSpec.scala:232)
  	at org.scalatest.Suite.run(Suite.scala:1147)
  	at org.scalatest.Suite.run$(Suite.scala:1129)
  	at org.scalatest.fixture.FreeSpec.org$scalatest$fixture$FreeSpecLike$$super$run(FreeSpec.scala:232)
  	at org.scalatest.fixture.FreeSpecLike.$anonfun$run$1(FreeSpecLike.scala:615)
  	at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
  	at org.scalatest.fixture.FreeSpecLike.run(FreeSpecLike.scala:615)
  	at org.scalatest.fixture.FreeSpecLike.run$(FreeSpecLike.scala:614)
  	at org.scalatest.fixture.FreeSpec.run(FreeSpec.scala:232)
  	at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
  	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1346)
  	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1340)
  	at scala.collection.immutable.List.foreach(List.scala:392)
  	at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1340)
  	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1031)
  	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1010)
  	at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1506)
  	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)
  	at org.scalatest.tools.Runner$.run(Runner.scala:850)
  	at org.scalatest.tools.Runner.run(Runner.scala)
  	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131)
  	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
  Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: kafka_partition_latest_offset
  	at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)
  	at io.prometheus.client.SimpleCollector$Builder.register(SimpleCollector.java:246)
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSink.$anonfun$metrics$4(PrometheusEndpointSink.scala:38)
  	at scala.collection.immutable.List.map(List.scala:286)
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSink.$anonfun$metrics$1(PrometheusEndpointSink.scala:33)
  	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
  	at scala.collection.immutable.Map$Map2.foreach(Map.scala:159)
  	at scala.collection.TraversableLike.map(TraversableLike.scala:237)
  	at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
  	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSink.<init>(PrometheusEndpointSink.scala:31)
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSink$.$anonfun$apply$1(PrometheusEndpointSink.scala:22)
  	at scala.util.Try$.apply(Try.scala:213)
  	at com.lightbend.kafkalagexporter.PrometheusEndpointSink$.apply(PrometheusEndpointSink.scala:22)
  	... 47 more
  ```
  
  It seems that adding dynamic labels in this manner is an anti-pattern in prometheus that should be handled with relabeling rules in Prometheus server config instead.  Whether or not we agree with this assessment I'll have to rollback this feature until another solution is found, and cut a new release.  Based on the info in the following ticket, it's possible that you could create a custom metrics collector that works around the problem.
  
  https://github.com/prometheus/client_java/issues/121
events.title: Metrics with different sets of labels can't be registered
events.type: issue
events.action: created
events.author: seglo
events.masked_author: username_0
events.text: Another idea is to let users configure arbitrary labels for all clusters and then configure custom label values for each cluster if they wish (otherwise they'll just be blank).
events.type: comment
events.action: created
events.author: anbarasantr
events.masked_author: username_1
events.text: @username_0 Thanks for the explanation. It was an oversight from my part. I did not test with multiple cluster setup. 
  I still think this would be benefit the user and I also like the idea that you suggested. I will first try to see if there are any alternative approaches around.
events.type: comment
events.action: created
events.author: seglo
events.masked_author: username_0
events.text: Resolved with #82
events.type: comment