Answers: <blockquote>
    <p>we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images).</p>
  </blockquote>
  
  <p>This is indeed good properties to have. You are on the right track.</p>
  
  <blockquote>
    <p>The problem, however, comes with our read-only data assets (similar to a game's asset files).</p>
  </blockquote>
  
  <p>This can be solved with multiple solutions, it's hard to tell what is the best for you. Here are some alternatives.</p>
  
  <p><strong>Persistent Volume - ReadOnlyMany</strong></p>
  
  <blockquote>
    <p>I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image.</p>
  </blockquote>
  
  <p>Yes, you could create a <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/" rel="nofollow noreferrer">Persistent Volume</a> with <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes" rel="nofollow noreferrer">access modes</a>:</p>
  
  <pre><code>accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  </code></pre>
  
  <p>then (by an automatic process)</p>
  
  <ol>
  <li>Load your data onto the volume with a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/" rel="nofollow noreferrer">Job</a> using access mode <code>ReadWriteOnce</code></li>
  <li>Load the volumes to your pods/services using access mode <code>ReadOnlyMany</code></li>
  </ol>
  
  <p>and keep the volume immutable, you create a new when you need to change content. You can also use multiple volumes this way.</p>
  
  <p><strong>CDN</strong></p>
  
  <blockquote>
    <p>our read-only data assets (similar to a game's asset files).</p>
  </blockquote>
  
  <p>This inherently also sounds what typically is served by <a href="https://en.wikipedia.org/wiki/Content_delivery_network" rel="nofollow noreferrer">Content Deliver Networks</a>. But it depends on your use case if this is a good solution or not.</p>
  
  <p><strong>Buckets</strong></p>
  
  <p>Data for e.g. CDN is usually stored by "buckets" at cloud providers, that may also be an option, then your applications load data using HTTP typically. <a href="https://min.io/" rel="nofollow noreferrer">Minio</a> may be an alternative if you want to do this in your cluster.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-05-04T16:29:29.287Z
Answers: But does this provide a volume "file" that I can then take and plug into another cluster, or do I have to repeat the "create volume, load volume" steps every time I want to deploy the data?
Answers: stix
Answers: CC BY-SA 4.0
Answers: 2020-05-04T16:39:21.347Z
Answers: PVC is only within cluster. CDN is "global". And buckets is also widely accessible.
Answers: Jonas
Answers: CC BY-SA 4.0
Answers: 2020-05-01T19:35:19.233Z
Answers: 2020-05-01T19:35:19.233Z
Answers: Jonas
Answers: <p>This may not have been available when the question was asked, but a relatively new option is to use &quot;Image Volumes&quot;:</p>
  <p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/</a></p>
  <p>Basically, package your assets into an image that can be distributed using the same channels as your application images, and mount that image into your application containers at runtime.</p>
  <p>I'm looking at using this technique to inject client-specific branding assets (logos, some CSS tweaks) into a standard image which would be shared by multiple clients.</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-12-18T21:30:00.370Z
Answers: 2024-12-18T21:30:00.370Z
Answers: Simon Geard
Body: <p>I'm migrating an HPC app to Kubernetes, and am trying to determine the best way to provide read only data assets as a configuration managed snapshot. </p>
  
  <p>Previously, my team had delivered our application as a set of RPMs, but as we move to Kubernetes, we're delivering Docker images. This works fine for our application binary, as instead of a pile of RPMs that all have to agree, we can just deliver a known working image. </p>
  
  <p>The problem, however, comes with our read-only data assets (similar to a game's asset files). Several different Docker images might rely on a single set of data assets, and so we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images). </p>
  
  <p>We're unsure as to the best approach for this. The first idea would be to create a "data container" that simply runs NFS and serves out the data. This successfully isolates the data from the application and allows us to collapse a set of data RPMs into a single tagged docker image, but to me it seems like it might be overkill. </p>
  
  <p>I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image. </p>
  
  <p><strong>Is there a better way to provide this read-only data as a version controlled snapshot?</strong></p>
  
Comments: CC BY-SA 4.0
Comments: 2020-05-01T17:30:55.053Z
Comments: Could you indicate what you mean with "data"? E.g. text files, blobs that are larger than 1 GB?
Comments: 030
Comments: CC BY-SA 4.0
Comments: 2020-05-01T17:36:30.883Z
Comments: A directory tree with files. Basically I'm looking for something like an ISO snapshot that we can version control and deliver as a mount point that the application pods pick up. 
  
  Our biggest dataset is on the order of 15 GB, but it's broken into multiple files.
Comments: stix
Comments: CC BY-SA 4.0
Comments: 2020-05-01T19:01:50.377Z
Comments: What cloud provider do you use? E.g. AWS, GCP, Azure, bare-metal?
Comments: 030
Comments: CC BY-SA 4.0
Comments: 2020-05-02T23:47:00.237Z
Comments: @030 Bare metal. These are our own clusters.
Comments: stix
Tags: <kubernetes><data>
Title: How can I best deliver read only data assets to a Kubernetes application?
UserDisplayName: stix
ThreadText: # How can I best deliver read only data assets to a Kubernetes application?
  
  (Asked by: username_0 on 2020-05-01)
  
  I'm migrating an HPC app to Kubernetes, and am trying to determine the best way to provide read only data assets as a configuration managed snapshot.
  
  Previously, my team had delivered our application as a set of RPMs, but as we move to Kubernetes, we're delivering Docker images. This works fine for our application binary, as instead of a pile of RPMs that all have to agree, we can just deliver a known working image.
  
  The problem, however, comes with our read-only data assets (similar to a game's asset files). Several different Docker images might rely on a single set of data assets, and so we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images).
  
  We're unsure as to the best approach for this. The first idea would be to create a "data container" that simply runs NFS and serves out the data. This successfully isolates the data from the application and allows us to collapse a set of data RPMs into a single tagged docker image, but to me it seems like it might be overkill.
  
  I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image.
  
  **Is there a better way to provide this read-only data as a version controlled snapshot?**
  
  **Tags:** kubernetes, data
  
  > **Comments:**
  > **username_1**: Could you indicate what you mean with "data"? E.g. text files, blobs that are larger than 1 GB?  
  > **username_0**: A directory tree with files. Basically I'm looking for something like an ISO snapshot that we can version control and deliver as a mount point that the application pods pick up.   Our biggest dataset is on the order of 15 GB, but it's broken into multiple files.  
  > **username_1**: What cloud provider do you use? E.g. AWS, GCP, Azure, bare-metal?  
  > **username_0**: @username_1 Bare metal. These are our own clusters.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-05-01. Score: 1
  
  > we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images).
  
  This is indeed good properties to have. You are on the right track.
  
  > The problem, however, comes with our read-only data assets (similar to a game's asset files).
  
  This can be solved with multiple solutions, it's hard to tell what is the best for you. Here are some alternatives.
  
  **Persistent Volume - ReadOnlyMany**
  
  > I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image.
  
  Yes, you could create a [Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) with [access modes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes):
  
  ```
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  
  ```
  
  then (by an automatic process)
  
  1. Load your data onto the volume with a [Job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) using access mode `ReadWriteOnce`
  2. Load the volumes to your pods/services using access mode `ReadOnlyMany`
  
  and keep the volume immutable, you create a new when you need to change content. You can also use multiple volumes this way.
  
  **CDN**
  
  > our read-only data assets (similar to a game's asset files).
  
  This inherently also sounds what typically is served by [Content Deliver Networks](https://en.wikipedia.org/wiki/Content_delivery_network). But it depends on your use case if this is a good solution or not.
  
  **Buckets**
  
  Data for e.g. CDN is usually stored by "buckets" at cloud providers, that may also be an option, then your applications load data using HTTP typically. [Minio](https://min.io/) may be an alternative if you want to do this in your cluster.
  
  > **Comments:**
  > **username_0**: But does this provide a volume "file" that I can then take and plug into another cluster, or do I have to repeat the "create volume, load volume" steps every time I want to deploy the data?  
  > **username_2**: PVC is only within cluster. CDN is "global". And buckets is also widely accessible.  
  
  ---
  
  ### Answer by username_3 on 2024-12-18. Score: 0
  
  This may not have been available when the question was asked, but a relatively new option is to use "Image Volumes":
  
  <https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/>
  
  Basically, package your assets into an image that can be distributed using the same channels as your application images, and mount that image into your application containers at runtime.
  
  I'm looking at using this technique to inject client-specific branding assets (logos, some CSS tweaks) into a standard image which would be shared by multiple clients.
  
  
  ---
  
  