title: Hive Window Function ROW_NUMBER without Partition BY Clause on a large (50 GB) dataset is very slow. Is there a better way to optimize?
body: <p>I have a HDFS file with 50 Million records and raw file size is 50 GB. </p>
  
  <p>I am trying to load this in a hive table and create unique id for all rows using the below, while loading. I am using Hive 1.1.0-cdh5.16.1.</p>
  
  <p>row_number() over(order by  event_id, user_id, timestamp) as id</p>
  
  <p>While executing I see that in the reduce step, 40 reducers are assigned. Average time for 39 Reducers is about 2 mins whereas the last reducer takes about 25 mins which clearly makes me believe that most of the data is processed in one reducer.</p>
  
  <p>I suspected Order By clause to be the reason for this behavior and tried the below,</p>
  
  <p>row_number() over() as id</p>
  
  <p>Yet, I see the same behavior.</p>
  
  <p>Thinking about the Map Reduce Paradigm, it makes me feel that if we do not specify a Partition BY Clause, the data has to be processed in one reducer (un-distributed) in order to see all rows and attach the correct row number. This could be true for any Window function with no partition By clause or partition By on skewed column.</p>
  
  <p>Now, my question is, how do we circumvent this problem and optimize window functions when we have to avoid Partition BY clause?</p>