Answers: <p>You're definitely not alone in this problem, it's well known. So much so that there's <a href="https://blog.eduonix.com/bigdata-and-hadoop/improve-dataops-with-dynamic-indexing-in-data-lake/" rel="nofollow noreferrer">numerous articles around it</a>. I personally have had issues with partitioning, queries failing completely and overall poor performance. The aforementioned article mentions Varada which I think could be a good solution. How it works:</p>
  <blockquote>
  <p>What they do is to break down a large dataset into what they call a nano block of 64k rows. Their technology looks at each nano block (thus original dataset) and automatically chooses an index for each nano block.</p>
  </blockquote>
  
Answers: CC BY-SA 4.0
Answers: 2022-03-27T00:29:52.443Z
Answers: 2022-03-28T06:23:01.713Z
Answers: 2022-03-28T06:23:01.713Z
Answers: Turnip
Body: <p>I am running into problems with AWS Athena data lakes. They were initially very quick, but as we've scaled, they're performing poorly and are becoming expensive and inefficient. What can we do to solve this without reducing data size?</p>
  
Tags: <amazon-web-services><amazon-s3><databases>
Title: How can I efficiently scale a data lake?
UserDisplayName: Joseph
ThreadText: # How can I efficiently scale a data lake?
  
  (Asked by: username_0 on 2022-03-26)
  
  I am running into problems with AWS Athena data lakes. They were initially very quick, but as we've scaled, they're performing poorly and are becoming expensive and inefficient. What can we do to solve this without reducing data size?
  
  **Tags:** amazon-web-services, amazon-s3, databases
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-27. Score: 1 | Accepted answer
  
  You're definitely not alone in this problem, it's well known. So much so that there's [numerous articles around it](https://blog.eduonix.com/bigdata-and-hadoop/improve-dataops-with-dynamic-indexing-in-data-lake/). I personally have had issues with partitioning, queries failing completely and overall poor performance. The aforementioned article mentions Varada which I think could be a good solution. How it works:
  
  > What they do is to break down a large dataset into what they call a nano block of 64k rows. Their technology looks at each nano block (thus original dataset) and automatically chooses an index for each nano block.
  
  
  ---
  
  