<p dir="auto">After upgrading to Druid 0.16.0-incubating, I am receiving a MySQLIntegrityConstraintViolationException complaining about:</p>
<p dir="auto">"Duplicate entry XXX for key 'PRIMARY' [statement:"INSERT INTO druid_pendingSegments (id, dataSource.."</p>
<p dir="auto">This results in the Kafka indexing tasks not being able to complete and the eventual failure of the coordinator/overlord nodes. This scenario only seems to happen after I drop some segments from Druid and then push in new data for the time period which was dropped. The only way I have found to fix this has been to force stop all of my Kafka indexing supervisors &amp; tasks and manually delete all of the entires in the druid_pendingSegments table. After I do that, I no longer receive the sql exception and corresponding duplicate entry error message. Any thoughts on this would be greatly appreciated!</p>
<p dir="auto"><strong>How to Reproduce:</strong></p>
<ol dir="auto">
<li>Suspend a Kafka indexing supervisor for a given data source and wait for the indexing task(s) to complete.</li>
<li>Drop segments for a certain time period from the given data source and wait for the segments to be unloaded from the historical nodes.</li>
<li>Resume the Kafka indexing supervisor for a given data source.</li>
<li>Push new data through Kafka for the same time period which was previously dropped on the given data source.</li>
<li>Check the indexing logs for the Kafka indexing tasks to see them complaining about duplicate primary key errors.</li>
</ol>
<p dir="auto"><strong>Other Notables:</strong></p>
<ol dir="auto">
<li>The druid_pendingSegments table doesn't seem to get cleaned up once a Kafka indexing supervisor is suspended. Entries are still left in this table for the given data source despite all the segments having been published to deepstorage / historical nodes. I do have <code class="notranslate">druid.coordinator.kill.pendingSegments.on=true</code> enabled. Maybe this is normal?</li>
</ol>
<p dir="auto">There have been multiple discussions about Druid source having too many sub-projects which really doesn't provide much value and often leads to complications e.g. <code class="notranslate">server/src/main/java/io/druid/client/indexing/ClientXXXQuery.java</code> classes are equivalent to <code class="notranslate">XXXTask.java</code> classes in <code class="notranslate">indexing-service/</code> and needs to stay in sync .. more such classes are created in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="275054823" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/5102" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/5102/hovercard" href="https://github.com/apache/druid/pull/5102">#5102</a> .<br>
similar desire has been mentioned in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="283950444" data-permission-text="Title is private" data-url="https://github.com/metamx/java-util/issues/74" data-hovercard-type="pull_request" data-hovercard-url="/metamx/java-util/pull/74/hovercard?comment_id=353473234&amp;comment_type=issue_comment" href="https://github.com/metamx/java-util/pull/74#issuecomment-353473234">metamx/java-util#74 (comment)</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="275054823" data-permission-text="Title is private" data-url="https://github.com/apache/druid/issues/5102" data-hovercard-type="pull_request" data-hovercard-url="/apache/druid/pull/5102/hovercard?comment_id=158430657&amp;comment_type=review_comment" href="https://github.com/apache/druid/pull/5102#discussion_r158430657">#5102 (comment)</a> .</p>
<p dir="auto">So, here is my initial idea....</p>
<ol dir="auto">
<li>merge <code class="notranslate">druid-api, druid-commons, java-utils, druid-processing</code> into <code class="notranslate">druid-core</code></li>
<li>merge <code class="notranslate">druid-server, druid-indexing-service</code> into <code class="notranslate">druid-server</code></li>
</ol>
<p dir="auto">we can merge things further if necessary.</p>