Answers.Body: <p>Using GitHub versions and tags is certainly one way to go. However, there are more sophisticated techniques and tools you might want to consider:</p>
  <ul>
  <li><p><strong>Data Version Control (DVC)</strong>: DVC is a popular open-source tool for versioning data and preprocessing code, specifically designed for machine learning projects. It provides a Git-like interface to manage both data and code, allowing you to link specific versions of your data and preprocessing pipeline with corresponding model versions. You can learn more about DVC at <a href="https://dvc.org/" rel="nofollow noreferrer">https://dvc.org/</a>.</p>
  </li>
  <li><p><strong>MLflow</strong>: MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It includes tools for tracking experiments, packaging code into reproducible runs, and sharing and deploying models. With MLflow, you can track different versions of your preprocessing code, data, and models, and switch between them when needed. You can learn more about MLflow at <a href="https://mlflow.org/" rel="nofollow noreferrer">https://mlflow.org/</a>.</p>
  </li>
  <li><p><strong>Docker</strong>: Another way to ensure consistency in your preprocessing is by using Docker containers. You can create a Docker image with the required dependencies and preprocessing code, and then use this image for both training and inference. This ensures that the same code and environment are used at each stage. You can then version the Docker images using tags, and store them in a container registry such as Docker Hub or Google Container Registry. You can learn more about Docker at <a href="https://www.docker.com/" rel="nofollow noreferrer">https://www.docker.com/</a>.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-23T17:37:34.150Z
Answers.LastActivityDate: 2023-03-23T17:37:34.150Z
Answers.UserDisplayName: Hans-Peter Schrei
Answers.Body: <p>Experiment tracking is so important, it's very hard to work together as a team on models without it. Sounds like you'd both want to track your data lineage and track your models. Here you can see my comparison of yolov5 and yolov8 models, everything is stored for all of these models; hyperparameters, metrics, confusion matrix, images, dependencies..  I didn't store the code, but it's very easy to do. <a href="https://www.comet.com/kristenkehrer/dogs-and-cats/view/new/panels" rel="nofollow noreferrer">https://www.comet.com/kristenkehrer/dogs-and-cats/view/new/panels</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-23T20:12:45.333Z
Answers.LastActivityDate: 2023-03-23T20:12:45.333Z
Answers.UserDisplayName: Kristen Kehrer
Body: <p>We have been working on a deep learning problem for a few iterations now.  We've been tweaking the preprocessing as we go.  We've also been training models as we go.</p>
  <p>The people using the models for inference need to do the same preprocessing on their data as we used on our training data.</p>
  <p>Is there any standard way of 'versioning' the preprocessing code so that it can be linked to the models that used that version?  I could use GitHub versions or tags, and maybe that's the best way.  Or is there some other technique that is widely used?</p>
  
Tags: <data-preprocessing>
Title: Linking preprocessing with models
UserDisplayName: Paul Reiners
ThreadText: # Linking preprocessing with models
  
  (Asked by: username_0 on 2023-03-21)
  
  We have been working on a deep learning problem for a few iterations now. We've been tweaking the preprocessing as we go. We've also been training models as we go.
  
  The people using the models for inference need to do the same preprocessing on their data as we used on our training data.
  
  Is there any standard way of 'versioning' the preprocessing code so that it can be linked to the models that used that version? I could use GitHub versions or tags, and maybe that's the best way. Or is there some other technique that is widely used?
  
  **Tags:** data-preprocessing
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-03-23. Score: 1 | Accepted answer
  
  Using GitHub versions and tags is certainly one way to go. However, there are more sophisticated techniques and tools you might want to consider:
  
  * **Data Version Control (DVC)**: DVC is a popular open-source tool for versioning data and preprocessing code, specifically designed for machine learning projects. It provides a Git-like interface to manage both data and code, allowing you to link specific versions of your data and preprocessing pipeline with corresponding model versions. You can learn more about DVC at <https://dvc.org/>.
  * **MLflow**: MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It includes tools for tracking experiments, packaging code into reproducible runs, and sharing and deploying models. With MLflow, you can track different versions of your preprocessing code, data, and models, and switch between them when needed. You can learn more about MLflow at <https://mlflow.org/>.
  * **Docker**: Another way to ensure consistency in your preprocessing is by using Docker containers. You can create a Docker image with the required dependencies and preprocessing code, and then use this image for both training and inference. This ensures that the same code and environment are used at each stage. You can then version the Docker images using tags, and store them in a container registry such as Docker Hub or Google Container Registry. You can learn more about Docker at <https://www.docker.com/>.
  
  
  ---
  
  ### Answer by username_2 on 2023-03-23. Score: 0
  
  Experiment tracking is so important, it's very hard to work together as a team on models without it. Sounds like you'd both want to track your data lineage and track your models. Here you can see my comparison of yolov5 and yolov8 models, everything is stored for all of these models; hyperparameters, metrics, confusion matrix, images, dependencies.. I didn't store the code, but it's very easy to do. <https://www.comet.com/username_2/dogs-and-cats/view/new/panels>
  
  
  ---
  
  