<p dir="auto">Hi,</p>
<p dir="auto">We are suffering a big performance hit with druid due to a dimension in our data source which has a very high cardinality. We need that dimension but only if there are significant rows with that dimension, i.e. we should ignore/discard cases where this dimension with a unique value only appears once.</p>
<p dir="auto">We thought about doing a two-step process: as we are inserting in realtime from Storm, when we receive the data we don't know if it's first appear or not, so we are inserting all, and once segment is closed after an hour, we execute a second query in a HIVE view to group manually the ones that only appear one time under something like "Unknown", and insert them again. The problem with this is that during the current hour there is still a degradation performance issue, but at least it works fine for data outside the realtime segment.</p>
<p dir="auto">This is an in-house provisional solution, but we would like to know if someone can help with a more elegant one.</p>
<p dir="auto">We are using version 0.9.2.</p>
<p dir="auto">Thanks for your time.</p>
<p dir="auto">I see a problem with the current way of managing metadata about publishing segments in AppenderatorDriver. Lets say a task calls AppenderatorDriver to publish segments for a sequence, then the driver will remove the sequence information from activeSegments and publishPendingSegments map. Now, if the task is restarted at a point after in memory data is persisted and metadata is committed but before any mergeAndPush or publish happens, then on restart if the task again tells driver to publish the same sequence, there is no way for the driver to know what segments to publish.</p>
<p dir="auto">If you look at the code for publish or publishAll method of AppenderatorDriver the first thing that is done is to remove the sequence information from activeSegments and publishPendingSegments map. After that push is called on Appenderator with wrapped committer which will contain driver metadata (with the sequence information removed). In the push implementation in memory data is persisted using persist method which also commits the metadata to disk. So, what I was saying is that if task is restarted at this point, the restored metadata might be incomplete as this sequence information will not be restored. Any further calls to publish this sequence wouldn't do anything.</p>
<p dir="auto">One way to resolve this would be to maintain an additional in-memory structure that contains sequences which are being published. On call to publish, sequence information is removed from activeSegments map but not from publishPendingSegments and added to the new in-memory structure. On successful publish sequence information is removed from the publishPendingSegments and in-memory structure. Any call to publish should try to publish sequence if it is not in the in-memory structure (it may not be in the activeSegment map when task is restarted at the point of time above described).</p>