Answers: <p>Sadly nobody answered, so I will provide feedback about how i did it so far.</p>
  
  <p>Graphite was not a good solution for me due to its retention policy. Performance tests sent data to the database every millisecond, and due to retention Graphite did not store all the results but only the last one per the period, meaning if you sent 100 data points in the last minute, the first 99 would be dropped, the 100th one would be stored. This fact was found by me somewhere on some forum. I do not claim I have setup/configured/used graphite correctly, yet the result was several days of debugging a bug which appeared an actual feature, lots of stress and questioning if my head and hands grow out of my shoulders.</p>
  
  <p>Next option was to use InfluxDB. Sadly didn't work for the following reason:<br>
   - If running InfluxDB in a Linux docker container, I was not able to bind mount it to a specific folder on host's OS outside of docker installation folder.
  A solution would be to try and use docker named volume. Maybe that would have worked, but I was short on time and patience.</p>
  
  <p>The final solution was to use simply a Postgres DB. Created a timetable with all the metrics and a timestamp field. As a result, Locust writes the final session statistics into a CSV file. CSV file is parsed by a python script and parsed data is inserted into the table. Grafana then connects to the DB and quite nicely visualizes the results using a simple query.</p>
  
  <p>Next possible question would be: </p>
  
  <blockquote>
    <p>how to store the results of every single request to gain better
    precision for the results of my performance tests?</p>
  </blockquote>
  
  <p>My personal suggestion would be to use influxDB with a consistent storage saved in a named volume if you are using Windows host. There the smallest time period can go down to milliseconds and data retrieval was very easily setup by manually writing a simple select query. Yet, please do not take my word for it and try this out, because i haven't done this, so my thoughts are nothing but a theory.</p>
  
  <p>Happy performance testing.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-04-09T09:19:06.763Z
Answers: >- If running InfluxDB in a Linux docker container, I was not able to bind mount it to a >specific folder on host's OS outside of docker installation folder.
  use -docker run -p 8086:8086 \
        -v influxdb:/var/lib/influxdb \
        influxdb to bind directories or even better create a data volume: https://towardsdatascience.com/get-system-metrics-for-5-min-with-docker-telegraf-influxdb-and-grafana-97cfd957f0ac
Answers: mglauche
Answers: CC BY-SA 4.0
Answers: 2018-07-06T13:56:09.667Z
Answers: 2018-07-06T13:56:09.667Z
Answers: GrimSmiler
Body: <h1>Intro</h1>
  
  <p>I am a tester and need to analyze the results of automated tests (response times and etc.)</p>
  
  <h1>Current situation</h1>
  
  <p>1) I run performance tests using <a href="https://locust.io/" rel="nofollow noreferrer">locust.io</a><br>
  2) During test execution, response times of successful results are pushed to <a href="https://graphiteapp.org/" rel="nofollow noreferrer">Graphite</a> database<br>
  3) I visualize the results in <a href="https://grafana.com/" rel="nofollow noreferrer">Grafana</a> </p>
  
  <h1>Problem</h1>
  
  <p>I tried to calculate the average using Grafana to see if it matches the results generated by Locust. It doesn't. </p>
  
  <p>After a while, i tried to figure out what could be a problem. Graphite has <a href="http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-schemas-conf" rel="nofollow noreferrer">data retention</a>. As a result, the smallest time period for a data point is 1 second.</p>
  
  <p>My situation is that I could have several data points with a time difference of several milliseconds.</p>
  
  <p>I thought I could use something like <a href="https://prometheus.io/" rel="nofollow noreferrer">Prometheus</a> as a substitute for graphite, but it doesn't seem to have an import of data. A solution would be installing <a href="https://github.com/prometheus/graphite_exporter" rel="nofollow noreferrer">Graphite Exporter</a>, such that i can imitate "pushing" data to db. But I think installing this extension and setting everything up is an overkill. There should be a much simpler solution.</p>
  
  <h1>Question</h1>
  
  <p>Could someone suggest a solution from personal experience? I need a db/storage to which I can write in a format <code>query_name responce_time timestamp</code>, and then use this storage as a data source for visualizing a report in Grafana.</p>
  
  <p>Thanks in advance!</p>
  
Tags: <testing><databases><performance><prometheus><grafana>
Title: What is a good storage for storing performance results and visualizing in grafana
UserDisplayName: GrimSmiler
ThreadText: # What is a good storage for storing performance results and visualizing in grafana
  
  (Asked by: username_0 on 2018-06-18)
  
  # Intro
  
  I am a tester and need to analyze the results of automated tests (response times and etc.)
  
  # Current situation
  
  1) I run performance tests using [locust.io](https://locust.io/)  
  2) During test execution, response times of successful results are pushed to [Graphite](https://graphiteapp.org/) database  
  3) I visualize the results in [Grafana](https://grafana.com/)
  
  # Problem
  
  I tried to calculate the average using Grafana to see if it matches the results generated by Locust. It doesn't.
  
  After a while, i tried to figure out what could be a problem. Graphite has [data retention](http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-schemas-conf). As a result, the smallest time period for a data point is 1 second.
  
  My situation is that I could have several data points with a time difference of several milliseconds.
  
  I thought I could use something like [Prometheus](https://prometheus.io/) as a substitute for graphite, but it doesn't seem to have an import of data. A solution would be installing [Graphite Exporter](https://github.com/prometheus/graphite_exporter), such that i can imitate "pushing" data to db. But I think installing this extension and setting everything up is an overkill. There should be a much simpler solution.
  
  # Question
  
  Could someone suggest a solution from personal experience? I need a db/storage to which I can write in a format `query_name responce_time timestamp`, and then use this storage as a data source for visualizing a report in Grafana.
  
  Thanks in advance!
  
  **Tags:** testing, databases, performance, prometheus, grafana
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2018-07-06. Score: 1
  
  Sadly nobody answered, so I will provide feedback about how i did it so far.
  
  Graphite was not a good solution for me due to its retention policy. Performance tests sent data to the database every millisecond, and due to retention Graphite did not store all the results but only the last one per the period, meaning if you sent 100 data points in the last minute, the first 99 would be dropped, the 100th one would be stored. This fact was found by me somewhere on some forum. I do not claim I have setup/configured/used graphite correctly, yet the result was several days of debugging a bug which appeared an actual feature, lots of stress and questioning if my head and hands grow out of my shoulders.
  
  Next option was to use InfluxDB. Sadly didn't work for the following reason:  
  - If running InfluxDB in a Linux docker container, I was not able to bind mount it to a specific folder on host's OS outside of docker installation folder.
  A solution would be to try and use docker named volume. Maybe that would have worked, but I was short on time and patience.
  
  The final solution was to use simply a Postgres DB. Created a timetable with all the metrics and a timestamp field. As a result, Locust writes the final session statistics into a CSV file. CSV file is parsed by a python script and parsed data is inserted into the table. Grafana then connects to the DB and quite nicely visualizes the results using a simple query.
  
  Next possible question would be:
  
  > how to store the results of every single request to gain better
  > precision for the results of my performance tests?
  
  My personal suggestion would be to use influxDB with a consistent storage saved in a named volume if you are using Windows host. There the smallest time period can go down to milliseconds and data retrieval was very easily setup by manually writing a simple select query. Yet, please do not take my word for it and try this out, because i haven't done this, so my thoughts are nothing but a theory.
  
  Happy performance testing.
  
  > **Comments:**
  > **username_1**: >- If running InfluxDB in a Linux docker container, I was not able to bind mount it to a >specific folder on host's OS outside of docker installation folder. use -docker run -p 8086:8086 \       -v influxdb:/var/lib/influxdb \       influxdb to bind directories or even better create a data volume: https://towardsdatascience.com/get-system-metrics-for-5-min-with-docker-telegraf-influxdb-and-grafana-97cfd957f0ac  
  
  ---
  
  