Answers: <ul>
  <li>I believe you’re on the right track.</li>
  <li>The best practice for CI/CD tools is to make your <strong>build and test stages agnostic</strong> to the CI/CD platform by using <strong>bash, PowerShell, or Python</strong> scripts.</li>
  <li>Then, you can use <strong>Jenkins or GitHub Actions</strong> to <strong>orchestrate</strong> everything.</li>
  <li>Both tools allow you to configure runs with different <strong>parameters</strong> and trigger them based on <strong>events</strong> (using <strong>hooks</strong> or <strong>plugins</strong>).</li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2024-08-16T15:06:39.603Z
Answers: 2024-08-16T15:06:39.603Z
Answers: YassineLbk
Body: <p>Our team has implemented some algorithms that run on robots. We capture robot sensor data over several runs per day. Whenever there is major changes in algorithm, we run it in simulation mode over past captured sensor data. We have written python scripts to:</p>
  <ol>
  <li>Run algorithm over past robot runs data one after another</li>
  <li>Analyse / visualize the algorithm performance by plotting various graphs in jpg files.</li>
  </ol>
  <p>Now we want to automate it further. Here are new requirements:</p>
  <ol>
  <li>Organize the past data</li>
  <li>Run latest or select code commit of the system / algorithm against some input or randomly selected data. (preferrably reusing python scripts we already have)</li>
  <li>Store the performance metric of the run in the database.</li>
  <li>Check if there are considerable degradation in the performance.</li>
  <li>Visualize the performance with different custom visualizations / graphs / plots. (preferrably reusing python scripts we already have)</li>
  <li>Given old data IDs and commit IDs, fetch corresponding run results and provide analysis / visualization.</li>
  </ol>
  <p>I was thinking to implement some client server app from scratch. For example server side (say a minimal django app) system can expose some REST APIs to</p>
  <ol>
  <li>Accept request (containing dataset ID and code commit ID) to checkout the commit and run simulation against specified data</li>
  <li>Persist the run result in database and graphs images in file system</li>
  <li>Return old run performance data and graphs image links given say run ID, commit ID or dataset ID for comparison across different runs / commits / datasets</li>
  </ol>
  <p>And then we can have some web app build from scratch that can consume these REST endpoints.</p>
  <p>But I felt there must be some existing framework to achieve this. However, a quick google did not lead me to anything.
  I have following doubts:</p>
  <p>Q1. Is there any tool to achieve this?<br />
  Q2. Does this usecase fit somewhere in dev ops lifecycle? If yes where?<br />
  Q3. How this use case is implemented in industry?</p>
  <p>I have following guess: Since we have simulation run and performance analysis / visualization scripts ready, we can reuse them somehow to fit in CI/CD pipeline. For example, we can implement points 2 and 3 using some CI/CD tools like Jenkins or Github actions. They can checkout and build specified commit and then run our python scripts to run the simulations and performance analysis and visualization.
  Requirements 1, 4, 5 and 6 can be implemented from scratch and can work independently from CI/CD tool used. I feel this will have advantage that we will use CI/CD tools for what they are best at: checking out and building app (on ad hoc demand or on every commit) while still allowing us to use our existing python scripts thereby not limiting our customization for analysis and visualization.</p>
  <p>Now my question is:</p>
  <p>Q4. Does above make sense? Or we should do it all either in some CI/CD tool or from scratch?</p>
  
Tags: <jenkins><continuous-integration><cicd><github>
Title: How to evaluate and compare algorithm / app performance across different past datasets and code commits
UserDisplayName: Mahesha999
ThreadText: # How to evaluate and compare algorithm / app performance across different past datasets and code commits
  
  (Asked by: username_0 on 2024-08-02)
  
  Our team has implemented some algorithms that run on robots. We capture robot sensor data over several runs per day. Whenever there is major changes in algorithm, we run it in simulation mode over past captured sensor data. We have written python scripts to:
  
  1. Run algorithm over past robot runs data one after another
  2. Analyse / visualize the algorithm performance by plotting various graphs in jpg files.
  
  Now we want to automate it further. Here are new requirements:
  
  1. Organize the past data
  2. Run latest or select code commit of the system / algorithm against some input or randomly selected data. (preferrably reusing python scripts we already have)
  3. Store the performance metric of the run in the database.
  4. Check if there are considerable degradation in the performance.
  5. Visualize the performance with different custom visualizations / graphs / plots. (preferrably reusing python scripts we already have)
  6. Given old data IDs and commit IDs, fetch corresponding run results and provide analysis / visualization.
  
  I was thinking to implement some client server app from scratch. For example server side (say a minimal django app) system can expose some REST APIs to
  
  1. Accept request (containing dataset ID and code commit ID) to checkout the commit and run simulation against specified data
  2. Persist the run result in database and graphs images in file system
  3. Return old run performance data and graphs image links given say run ID, commit ID or dataset ID for comparison across different runs / commits / datasets
  
  And then we can have some web app build from scratch that can consume these REST endpoints.
  
  But I felt there must be some existing framework to achieve this. However, a quick google did not lead me to anything.
  I have following doubts:
  
  Q1. Is there any tool to achieve this?  
  Q2. Does this usecase fit somewhere in dev ops lifecycle? If yes where?  
  Q3. How this use case is implemented in industry?
  
  I have following guess: Since we have simulation run and performance analysis / visualization scripts ready, we can reuse them somehow to fit in CI/CD pipeline. For example, we can implement points 2 and 3 using some CI/CD tools like Jenkins or Github actions. They can checkout and build specified commit and then run our python scripts to run the simulations and performance analysis and visualization.
  Requirements 1, 4, 5 and 6 can be implemented from scratch and can work independently from CI/CD tool used. I feel this will have advantage that we will use CI/CD tools for what they are best at: checking out and building app (on ad hoc demand or on every commit) while still allowing us to use our existing python scripts thereby not limiting our customization for analysis and visualization.
  
  Now my question is:
  
  Q4. Does above make sense? Or we should do it all either in some CI/CD tool or from scratch?
  
  **Tags:** jenkins, continuous-integration, cicd, github
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-16. Score: 0
  
  * I believe you’re on the right track.
  * The best practice for CI/CD tools is to make your **build and test stages agnostic** to the CI/CD platform by using **bash, PowerShell, or Python** scripts.
  * Then, you can use **Jenkins or GitHub Actions** to **orchestrate** everything.
  * Both tools allow you to configure runs with different **parameters** and trigger them based on **events** (using **hooks** or **plugins**).
  
  
  ---
  
  