Answers: <p>The only options I think you have are time consuming <code>find</code> and <code>du</code> commands.</p>
  <p>You can use the <code>du</code> command to look at all files, sort the output and give you the top 5.  To do this for individual directories you can update the path.</p>
  <pre><code>du -sk /path/to/efs/* |sort -nr |head -n 5
  </code></pre>
  <p>With <code>find</code> you can search for files greater than a certain size and dump the list to a file.  You can probably parse it down to the top 5 with a little bit of piping.</p>
  <pre><code>find /path/to/efs -type f -size +200M &gt;&gt; /file_to_store_output.txt &amp;
  </code></pre>
  <p>The size units for your usage will probably be <code>M</code>egabytes or <code>G</code>igabytes</p>
  <p>I'd recommend running these as background processes and writing the output to a file.</p>
  <p><strong>References</strong><br />
  <a href="https://man7.org/linux/man-pages/man1/find.1.html" rel="nofollow noreferrer">find man page</a><br />
  <a href="https://linuxcommand.org/lc3_man_pages/du1.html" rel="nofollow noreferrer">du man page</a></p>
  
Answers: CC BY-SA 4.0
Answers: 2022-03-08T20:41:57.933Z
Answers: Thanks - generally good advice.  But the listings are taking so long that we're trying to find an alternate method that does not rely on commands that enumerate entire directories at once.  E.g. pretty sure find or du both would take a long time on a million-file-directory.  I'm trying to figure out if there is a way to just list a partial range of those files (e.g. top 10 without the enumeration of all being required).  du would list all, then head limits to 5, but it still took forever to list them all in the first command in that case, for example :(.  Upvoting for the good commands tough!
Answers: John Humphreys
Answers: CC BY-SA 4.0
Answers: 2022-03-08T15:40:09.313Z
Answers: 2022-03-08T15:40:09.313Z
Answers: kenlukas
Body: <p>I have an EFS with a crazily high number of files, many being small.  The directory tree is large as well.</p>
  <p>Listing directories can take an immensely long time.</p>
  <p>The EFS clearly has some very large files that need to be pruned based on the size (I can say this from app architecture/file size estimations for its main use case), but I can't locate the big files given the directory listing time.</p>
  <p>I could enable intelligent tiering to save money, but I'd like to identify and fix the root issue.  Are there any options to speed up a search?</p>
  <p>It would be nice if I could go to every directory and just get the size of the first 5 files to see if any specific directories have large files in general, but any listing seems to be exhaustive and time consuming.</p>
  
Tags: <amazon-web-services>
Title: Find biggest files in AWS EFS?
UserDisplayName: John Humphreys
ThreadText: # Find biggest files in AWS EFS?
  
  (Asked by: username_0 on 2022-03-07)
  
  I have an EFS with a crazily high number of files, many being small. The directory tree is large as well.
  
  Listing directories can take an immensely long time.
  
  The EFS clearly has some very large files that need to be pruned based on the size (I can say this from app architecture/file size estimations for its main use case), but I can't locate the big files given the directory listing time.
  
  I could enable intelligent tiering to save money, but I'd like to identify and fix the root issue. Are there any options to speed up a search?
  
  It would be nice if I could go to every directory and just get the size of the first 5 files to see if any specific directories have large files in general, but any listing seems to be exhaustive and time consuming.
  
  **Tags:** amazon-web-services
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-08. Score: 1
  
  The only options I think you have are time consuming `find` and `du` commands.
  
  You can use the `du` command to look at all files, sort the output and give you the top 5. To do this for individual directories you can update the path.
  
  ```
  du -sk /path/to/efs/* |sort -nr |head -n 5
  
  ```
  
  With `find` you can search for files greater than a certain size and dump the list to a file. You can probably parse it down to the top 5 with a little bit of piping.
  
  ```
  find /path/to/efs -type f -size +200M >> /file_to_store_output.txt &
  
  ```
  
  The size units for your usage will probably be `M`egabytes or `G`igabytes
  
  I'd recommend running these as background processes and writing the output to a file.
  
  **References**  
  [find man page](https://man7.org/linux/man-pages/man1/find.1.html)  
  [du man page](https://linuxcommand.org/lc3_man_pages/du1.html)
  
  > **Comments:**
  > **username_0**: Thanks - generally good advice.  But the listings are taking so long that we're trying to find an alternate method that does not rely on commands that enumerate entire directories at once.  E.g. pretty sure find or du both would take a long time on a million-file-directory.  I'm trying to figure out if there is a way to just list a partial range of those files (e.g. top 10 without the enumeration of all being required).  du would list all, then head limits to 5, but it still took forever to list them all in the first command in that case, for example :(.  Upvoting for the good commands tough!  
  
  ---
  
  