Answers.Body: <ol>
  <li><p>Tim has answered this (as of Apr 2023) here:</p>
  <p><a href="https://lab.civicrm.org/dev/core/-/issues/4251#note_90037" rel="nofollow noreferrer">https://lab.civicrm.org/dev/core/-/issues/4251#note_90037</a></p>
  <p>Short version is: yes, a more consistent api for file serving is under consideration, it's currently a bit of mess, but with an entertainingly written back story.</p>
  </li>
  <li><p>For the narrower question of &quot;how would I do something like this now&quot;, the answer would be to look at this file:</p>
  <p><a href="https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/BAO/File.php" rel="nofollow noreferrer">https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/BAO/File.php</a></p>
  <p>which contains the methods for managing (uploading, moving, etc.) &quot;entity files&quot;, i.e. those that are attached to an entity, which probably is the main scalability concern.</p>
  </li>
  <li><p>@bgm has answered a different question of &quot;how can I use S3 to host my CiviCRM uploaded files&quot; with a straightforward solution of using the <a href="https://linux.die.net/man/1/s3fs" rel="nofollow noreferrer">S3 Fuse filesystem</a>. That might be the best way to solve my specific issue of scalability, by mounting a bucket to customFileUploadDir. The potential downside is that it relies on translating low level filesystem requests, which means some types of operations might be quite inefficient (e.g. big directory listings).</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-22T19:48:22.600Z
Answers.LastActivityDate: 2023-08-22T20:25:34.710Z
Answers.LastEditDate: 2023-08-22T20:25:34.710Z
Answers.UserDisplayName: Alan Dixon
Body: <p>Drupal has a project called &quot;<a href="https://www.drupal.org/project/s3fs/" rel="noreferrer">S3 filesystem</a>&quot; which allows a site to put some of it's &quot;files&quot; on a remote S3 bucket (or other object-based filesystem). It works because Drupal accesses files through a pluggable api.</p>
  <p>It appears that most civicrm access of its &quot;<a href="https://docs.civicrm.org/dev/en/latest/framework/filesystem/#local-data-files" rel="noreferrer">local data files</a>&quot; is now programmatic, i.e. those files are rarely accessed based on an assumption of where they are relative to the document root, so it seems like it might not be too far off that all of civicrm core's file access could be pluggable and a civicrm extension similar to the s3 filesystem possible.</p>
  <p>Any thoughts about this? I'm particularly interested in the files in customFileUploadDir.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-22T04:01:41.630Z
Comments.Text: Some related discussion: https://lab.civicrm.org/dev/core/-/issues/4251#note_90037 and https://gist.github.com/totten/e1c7be1ee949fe372413c25f8bd11bd8
Comments.UserDisplayName: Tim Otten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-22T14:27:10.293Z
Comments.Text: Thanks Tim, but that's more than related discussion, and your answer in that issue is exactly the answer I was looking for - feel free to convert your comment above to an answer! The history at the top of issue also counts as A+ entertainment (thanks Coleman!).
Comments.UserDisplayName: Alan Dixon
Tags: <filesystem>
Title: What parts of the civicrm filesystem's "local data files" can be converted to e.g. S3?
ThreadText: # What parts of the civicrm filesystem's "local data files" can be converted to e.g. S3?
  
  (Asked by: username_0 on 2023-08-21)
  
  Drupal has a project called "[S3 filesystem](https://www.drupal.org/project/s3fs/)" which allows a site to put some of it's "files" on a remote S3 bucket (or other object-based filesystem). It works because Drupal accesses files through a pluggable api.
  
  It appears that most civicrm access of its "[local data files](https://docs.civicrm.org/dev/en/latest/framework/filesystem/#local-data-files)" is now programmatic, i.e. those files are rarely accessed based on an assumption of where they are relative to the document root, so it seems like it might not be too far off that all of civicrm core's file access could be pluggable and a civicrm extension similar to the s3 filesystem possible.
  
  Any thoughts about this? I'm particularly interested in the files in customFileUploadDir.
  
  **Tags:** filesystem
  
  > **Comments:**
  > **username_1**: Some related discussion: https://lab.civicrm.org/dev/core/-/issues/4251#note_90037 and https://gist.github.com/totten/e1c7be1ee949fe372413c25f8bd11bd8  
  > **username_0**: Thanks Tim, but that's more than related discussion, and your answer in that issue is exactly the answer I was looking for - feel free to convert your comment above to an answer! The history at the top of issue also counts as A+ entertainment (thanks Coleman!).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-08-22. Score: 2 | Accepted answer
  
  1. Tim has answered this (as of Apr 2023) here:
  
     <https://lab.civicrm.org/dev/core/-/issues/4251#note_90037>
  
     Short version is: yes, a more consistent api for file serving is under consideration, it's currently a bit of mess, but with an entertainingly written back story.
  2. For the narrower question of "how would I do something like this now", the answer would be to look at this file:
  
     <https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/BAO/File.php>
  
     which contains the methods for managing (uploading, moving, etc.) "entity files", i.e. those that are attached to an entity, which probably is the main scalability concern.
  3. @bgm has answered a different question of "how can I use S3 to host my CiviCRM uploaded files" with a straightforward solution of using the [S3 Fuse filesystem](https://linux.die.net/man/1/s3fs). That might be the best way to solve my specific issue of scalability, by mounting a bucket to customFileUploadDir. The potential downside is that it relies on translating low level filesystem requests, which means some types of operations might be quite inefficient (e.g. big directory listings).
  
  
  ---
  
  