Answers: <p>Personally I don't see any reason for which an <a href="https://devops.stackexchange.com/questions/452/what-is-an-artifact-repository">Artefact Repository</a> - the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts. </p>
  
  <p>The artefact size <em>might</em> have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.</p>
  
  <p>As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:</p>
  
  <ul>
  <li>are produced from some sort of specification which supports change versioning (equivalent to software source code)</li>
  <li>are built via a repeatable and automatable process</li>
  <li>are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)</li>
  </ul>
  
  <p>Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-09-14T02:41:47.497Z
Answers: Perfect answer.  I store all my heavy models in `git lfs` and pull them when necessary [serverless paradigm] :)
Answers: Dawny33
Answers: CC BY-SA 3.0
Answers: 2017-09-14T07:38:50.773Z
Answers: @Dawny33 but would you now consider moving away from git lfs?
Answers: Ta Mu
Answers: CC BY-SA 3.0
Answers: 2017-09-14T08:14:48.993Z
Answers: @J.Doe So far so good with lfs. Would probably move if I find a really good better alternative.
Answers: Dawny33
Answers: CC BY-SA 3.0
Answers: 2017-09-14T08:16:34.530Z
Answers: then I do not get why you say the answer about is "perfect" while it suggests using an artefact repository?! @Dawny33
Answers: Ta Mu
Answers: CC BY-SA 3.0
Answers: 2017-09-14T08:37:21.473Z
Answers: @J.Doe By the linked answer, I thought using `git lfs` is akin to the concept of using an artefact repo.  If I'm wrong, then my bad :D
Answers: Dawny33
Answers: CC BY-SA 3.0
Answers: 2017-09-14T15:31:13.720Z
Answers: @J.Doe are you somehow suggesting that using `git lfs` and a dedicated artifact repository are not equally devopsy? "cuz they both can be used...
Answers: Dan Cornilescu
Answers: CC BY-SA 3.0
Answers: 2017-09-15T07:11:36.227Z
Answers: @DanCornilescu hehe gotcha! Why then is that not listed as valid alternative in your answer? ;)
Answers: Ta Mu
Answers: CC BY-SA 3.0
Answers: 2017-09-15T07:34:11.053Z
Answers: @J.Doe I'm not familiar with `git lfs`, I just read about it when I saw your comment. Also I didn't mean to list actual tools in the family.
Answers: Dan Cornilescu
Answers: CC BY-SA 4.0
Answers: 2018-06-20T21:06:53.347Z
Answers: [DVC](https://dvc.org) can be considered as a better alternative to `git-lfs`
Answers: Shcheklein
Answers: CC BY-SA 3.0
Answers: 2017-09-13T16:07:45.937Z
Answers: 2017-09-13T16:12:58.623Z
Answers: 2017-09-13T16:12:58.623Z
Answers: Dan Cornilescu
Answers: <p>I would recommend taking a look at <a href="https://dvc.org" rel="nofollow noreferrer">DVC</a> - an open source version control system for data science projects.</p>
  
  <p>One of the basic things that it perfectly handles is managing data files (along with code) - inputs, outputs (models), intermediate results. Semantically it's similar to <code>git-lfs</code> but unlike <code>git-lfs</code> it is capable of managing files like 100GB and what is more important it does not rely on proprietary storage/format. It's completely open-source and is compatible with any network storage as a server to keep data files - S3, GCP cloud storage, SSH, FTP, etc.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-04-09T09:53:41.330Z
Answers: DVC has some nice features, but I don't believe Git LFS uses proprietary storage - it's an open source project.
Answers: RichVel
Answers: CC BY-SA 4.0
Answers: 2018-06-20T21:03:28.073Z
Answers: 2018-06-20T21:03:28.073Z
Answers: Shcheklein
Body: <p>Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.</p>
  <p>With the rise of data science there is a new type of artefact - monolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can be many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, such blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.</p>
  <p>What are best practices to address this new challenge using DevOps methods - if even possible?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-09-13T12:24:41.423Z
Comments: I don't see the difference between a large blob and an uberjar in java context. Same practices applies, the size of an artefact has few reasons to come into play.
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-09-13T13:39:28.503Z
Comments: Hi - I thought with uber-jars from 2gb upwards you would tell them the microservice architecture story, or?.. But model blobs just start there, 8gb will be soon not rare.
Comments: Ta Mu
Comments: CC BY-SA 3.0
Comments: 2017-09-13T14:53:16.457Z
Comments: I just mean a dB snapshot of 350Go is not a different asset than a 5Mo jar, it has to be stored somewhere anyway and artifact repository can handle that
Comments: Tensibai
Comments: CC BY-SA 3.0
Comments: 2017-09-13T15:40:52.013Z
Comments: I agree - just because the resulting program is large doesn't mean it isn't still compiled, versioned and stored like everything else (albeit with perhaps a few storage challenges), so I fail to see how this "brings organizations back to the pre-CI age" If an organization thinks that, I'm not sure that they actually understand DevOps/CI.
Comments: James Shewey
Tags: <continuous-integration><continuous-delivery><automation><methodology>
Title: Data Science pipelines and monolithic model blobs
UserDisplayName: Ta Mu
ThreadText: # Data Science pipelines and monolithic model blobs
  
  (Asked by: username_0 on 2017-09-13)
  
  Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.
  
  With the rise of data science there is a new type of artefact - monolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can be many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, such blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.
  
  What are best practices to address this new challenge using DevOps methods - if even possible?
  
  **Tags:** continuous-integration, continuous-delivery, automation, methodology
  
  > **Comments:**
  > **username_1**: I don't see the difference between a large blob and an uberjar in java context. Same practices applies, the size of an artefact has few reasons to come into play.  
  > **username_0**: Hi - I thought with uber-jars from 2gb upwards you would tell them the microservice architecture story, or?.. But model blobs just start there, 8gb will be soon not rare.  
  > **username_1**: I just mean a dB snapshot of 350Go is not a different asset than a 5Mo jar, it has to be stored somewhere anyway and artifact repository can handle that  
  > **username_2**: I agree - just because the resulting program is large doesn't mean it isn't still compiled, versioned and stored like everything else (albeit with perhaps a few storage challenges), so I fail to see how this "brings organizations back to the pre-CI age" If an organization thinks that, I'm not sure that they actually understand DevOps/CI.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-09-13. Score: 9 | Accepted answer
  
  Personally I don't see any reason for which an [Artefact Repository](https://devops.stackexchange.com/questions/452/what-is-an-artifact-repository) - the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts.
  
  The artefact size *might* have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.
  
  As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:
  
  * are produced from some sort of specification which supports change versioning (equivalent to software source code)
  * are built via a repeatable and automatable process
  * are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)
  
  Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.
  
  > **Comments:**
  > **username_4**: Perfect answer.  I store all my heavy models in `git lfs` and pull them when necessary [serverless paradigm] :)  
  > **username_0**: @username_4 but would you now consider moving away from git lfs?  
  > **username_4**: @J.Doe So far so good with lfs. Would probably move if I find a really good better alternative.  
  > **username_0**: then I do not get why you say the answer about is "perfect" while it suggests using an artefact repository?! @username_4  
  > **username_4**: @J.Doe By the linked answer, I thought using `git lfs` is akin to the concept of using an artefact repo.  If I'm wrong, then my bad :D  
  > **username_3**: @J.Doe are you somehow suggesting that using `git lfs` and a dedicated artifact repository are not equally devopsy? "cuz they both can be used...  
  > **username_0**: @username_3 hehe gotcha! Why then is that not listed as valid alternative in your answer? ;)  
  > **username_3**: @J.Doe I'm not familiar with `git lfs`, I just read about it when I saw your comment. Also I didn't mean to list actual tools in the family.  
  > **username_5**: [DVC](https://dvc.org) can be considered as a better alternative to `git-lfs`  
  
  ---
  
  ### Answer by username_5 on 2018-06-20. Score: 4
  
  I would recommend taking a look at [DVC](https://dvc.org) - an open source version control system for data science projects.
  
  One of the basic things that it perfectly handles is managing data files (along with code) - inputs, outputs (models), intermediate results. Semantically it's similar to `git-lfs` but unlike `git-lfs` it is capable of managing files like 100GB and what is more important it does not rely on proprietary storage/format. It's completely open-source and is compatible with any network storage as a server to keep data files - S3, GCP cloud storage, SSH, FTP, etc.
  
  > **Comments:**
  > **username_6**: DVC has some nice features, but I don't believe Git LFS uses proprietary storage - it's an open source project.  
  
  ---
  
  