Answers.Body: <p>The only thing I could imagine is that Laptop memory is exceeding RAM and that there is a lot of swapping going on. The 180 kHz sampling needs about 1/3rd more memory. If this is the case try (as test only) to set the LTSA setting to 1 sec.
  Unfortunately, I do not use Triton, so I only can give general advice.
  Closing unused programs the need memories could also help.</p>
  <p>Edit: I may need to dig into Triton, to see if it loads the whole file for processing then 10 min 180 kHz is 5/3 more data than 2 min @ 125 kHz. Anyhow, my experience with Matlab is that when it goes really slow, then usually there are memory issues.</p>
  <p>Edit2: It seems that Triton loads only data that are needed (here 5 sec) so only difference is sampling 180 vs 125 kHz or 1/3 more.</p>
  <p>Edit3: while data are sampled as 16 bit Matlab uses 64 bit as standard. So, a 5 s data set @ 125 kHz will have 1.25 MB on disk and 5 MB in Matlab. @ 180 kHz data will be 7.2 MB. This is not a big issue for a 16GB PC one would think, but &gt;8 GB are already used by system. To really know if PC is running out of memory is to have Memory tab of task manager open and to follow development of memory usage. Especially, if processing is CPU intensive and Matlab defers Garbage Collection (to free used memory), memory can grow as function of time.</p>
  <p>Edit4: It may not be the case here, but the presence of NaN of Inf (e.g. divide by 0) will also slow down processing</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T18:42:35.273Z
Answers.Text: I don't *think* its a memory problem because my laptop and the virtual machine are both 16 GB and using Task Manager I can see it's only using 8.7/54% of the memory (with just MATLAB open). 
  
  If I try that test with a 1 sec average, what would I be looking for?
Answers.UserDisplayName: selene
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T20:00:43.953Z
Answers.Text: It does load the whole file before doing the averaging (readaudio(filename), that's what I was worried might be causing the slow down. I was thinking I could probably modify it to speed it up but need to find the time to do that :) And before going through all that effort wanted to see if that would make sense as the problem.
Answers.UserDisplayName: selene
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-05T19:30:17.780Z
Answers.Text: I was incorrect - it is NOT loading in the whole file, only the samples necessary for the 5 seconds to be averaged. I dug in the code to try to change that and realized that is already how it is being done.
Answers.UserDisplayName: selene
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-06T05:00:19.047Z
Answers.Text: Added more edits to answer to address Matlab
Answers.UserDisplayName: WMXZ
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T03:42:34.863Z
Answers.Text: Thank you @WMXZ! I have been running with task manager open watching the memory status and that is why I don't think that is it because it is very steady at just over 8 GB. I'll take a screenshot of it tomorrow and add.
Answers.UserDisplayName: selene
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T18:33:21.010Z
Answers.LastActivityDate: 2022-08-06T04:59:46.523Z
Answers.LastEditDate: 2022-08-06T04:59:46.523Z
Answers.UserDisplayName: WMXZ
Answers.Body: <p>I second @WMXZ. It is likely a memory issue. Do you know what your bit size is? I believe that Triton uses the function audioread, and the standard number format in Matlab is a 64-bit float a.k.a. double (8 bytes), so you need 4 to 8 times as much memory as the file occupies on disk.</p>
  <p>I know this isn't really possible in the Triton interface, but if you start digging into the code you can set both your necessary bit size and the data type you want the wav file to be imported as.  That could speed things up.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-05T19:29:43.960Z
Answers.Text: Recordings are 16-bit. So, it could be a memory issue even though it doesn't appear its maxing out the memory, but maybe its maxed out within Matlab or ??
  
  added the bit size and computer specs to my question.
Answers.UserDisplayName: selene
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-06T10:10:52.643Z
Answers.Text: Yeah, I have this issue with matlab on my laptop all the time, but not on the desktops that have more powerful RAM. For comparison, I run LTSAs in Triton on half hour files, 96 kHz, spanning 2-3 months on a 50% duty cycle. It generally takes 36 hours or more to run on the high ram computers.
Answers.UserDisplayName: etgriffiths
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-05T11:52:25.773Z
Answers.LastActivityDate: 2022-08-05T11:52:25.773Z
Answers.UserDisplayName: etgriffiths
Body: <p>I am generated LTSAs (long-term spectral averages) using program Triton. I have done this lots in the past, but for my most recent dataset, the LTSA creation is taking WAY longer than ever before... like DAYS longer. I am not working with that long of a dataset or a very different sample rate than used before. But, this is the longest duration of individual .wav files I've every worked with. I'm wondering if the time it takes to read in each .wav is slowing it down? <strong>So, my question is, does anyone know why is my LTSA creation so slow?</strong></p>
  <p>Current data set - 10 min wav files, nearly except for 15-30 sec gaps between files for file writing, 180 kHz sample rate, 16 bit, LTSA settings 5 sec time average, 100 Hz frequency average. 8 weeks of data, about 7900 individual files. This is taking over 42 hours to run. Output size is about 800 MB.</p>
  <p>Previous data sets that were faster - 2 min wav or flac files, continuous, 125 kHz sample rate, 16 bit, same LTSA settings (5 sec, 100 Hz), typically 4-6 weeks of data, 20,000+ files, would run in 8-12 hours.</p>
  <p>Data are on an external hard drive directly connected to my laptop. Trying this on a virtual machine directly connected to our server (where a back up of the data are stored) is even slower</p>
  <p>I'm using the latest version of Triton on Github at <a href="https://github.com/MarineBioAcousticsRC/Triton" rel="noreferrer">https://github.com/MarineBioAcousticsRC/Triton</a>. I have a 16 GB Windows 10 machine, i7 processor. It is a different computer than I created the previous LTSAs on, but both were 16 GB RAM and i7's.</p>
  
Tags: <ltsa><triton><data-processing>
Title: Very slow creation of LTSAs from 10-min wav files using Triton
ThreadText: # Very slow creation of LTSAs from 10-min wav files using Triton
  
  (Asked by: username_0 on 2022-08-04)
  
  I am generated LTSAs (long-term spectral averages) using program Triton. I have done this lots in the past, but for my most recent dataset, the LTSA creation is taking WAY longer than ever before... like DAYS longer. I am not working with that long of a dataset or a very different sample rate than used before. But, this is the longest duration of individual .wav files I've every worked with. I'm wondering if the time it takes to read in each .wav is slowing it down? **So, my question is, does anyone know why is my LTSA creation so slow?**
  
  Current data set - 10 min wav files, nearly except for 15-30 sec gaps between files for file writing, 180 kHz sample rate, 16 bit, LTSA settings 5 sec time average, 100 Hz frequency average. 8 weeks of data, about 7900 individual files. This is taking over 42 hours to run. Output size is about 800 MB.
  
  Previous data sets that were faster - 2 min wav or flac files, continuous, 125 kHz sample rate, 16 bit, same LTSA settings (5 sec, 100 Hz), typically 4-6 weeks of data, 20,000+ files, would run in 8-12 hours.
  
  Data are on an external hard drive directly connected to my laptop. Trying this on a virtual machine directly connected to our server (where a back up of the data are stored) is even slower
  
  I'm using the latest version of Triton on Github at <https://github.com/MarineBioAcousticsRC/Triton>. I have a 16 GB Windows 10 machine, i7 processor. It is a different computer than I created the previous LTSAs on, but both were 16 GB RAM and i7's.
  
  **Tags:** ltsa, triton, data-processing
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-08-04. Score: 3
  
  The only thing I could imagine is that Laptop memory is exceeding RAM and that there is a lot of swapping going on. The 180 kHz sampling needs about 1/3rd more memory. If this is the case try (as test only) to set the LTSA setting to 1 sec.
  Unfortunately, I do not use Triton, so I only can give general advice.
  Closing unused programs the need memories could also help.
  
  Edit: I may need to dig into Triton, to see if it loads the whole file for processing then 10 min 180 kHz is 5/3 more data than 2 min @ 125 kHz. Anyhow, my experience with Matlab is that when it goes really slow, then usually there are memory issues.
  
  Edit2: It seems that Triton loads only data that are needed (here 5 sec) so only difference is sampling 180 vs 125 kHz or 1/3 more.
  
  Edit3: while data are sampled as 16 bit Matlab uses 64 bit as standard. So, a 5 s data set @ 125 kHz will have 1.25 MB on disk and 5 MB in Matlab. @ 180 kHz data will be 7.2 MB. This is not a big issue for a 16GB PC one would think, but >8 GB are already used by system. To really know if PC is running out of memory is to have Memory tab of task manager open and to follow development of memory usage. Especially, if processing is CPU intensive and Matlab defers Garbage Collection (to free used memory), memory can grow as function of time.
  
  Edit4: It may not be the case here, but the presence of NaN of Inf (e.g. divide by 0) will also slow down processing
  
  > **Comments:**
  > **username_0**: I don't *think* its a memory problem because my laptop and the virtual machine are both 16 GB and using Task Manager I can see it's only using 8.7/54% of the memory (with just MATLAB open).   If I try that test with a 1 sec average, what would I be looking for?  
  > **username_0**: It does load the whole file before doing the averaging (readaudio(filename), that's what I was worried might be causing the slow down. I was thinking I could probably modify it to speed it up but need to find the time to do that :) And before going through all that effort wanted to see if that would make sense as the problem.  
  > **username_0**: I was incorrect - it is NOT loading in the whole file, only the samples necessary for the 5 seconds to be averaged. I dug in the code to try to change that and realized that is already how it is being done.  
  > **username_1**: Added more edits to answer to address Matlab  
  > **username_0**: Thank you @username_1! I have been running with task manager open watching the memory status and that is why I don't think that is it because it is very steady at just over 8 GB. I'll take a screenshot of it tomorrow and add.  
  
  ---
  
  ### Answer by username_2 on 2022-08-05. Score: 2
  
  I second @username_1. It is likely a memory issue. Do you know what your bit size is? I believe that Triton uses the function audioread, and the standard number format in Matlab is a 64-bit float a.k.a. double (8 bytes), so you need 4 to 8 times as much memory as the file occupies on disk.
  
  I know this isn't really possible in the Triton interface, but if you start digging into the code you can set both your necessary bit size and the data type you want the wav file to be imported as. That could speed things up.
  
  > **Comments:**
  > **username_0**: Recordings are 16-bit. So, it could be a memory issue even though it doesn't appear its maxing out the memory, but maybe its maxed out within Matlab or ??  added the bit size and computer specs to my question.  
  > **username_2**: Yeah, I have this issue with matlab on my laptop all the time, but not on the desktops that have more powerful RAM. For comparison, I run LTSAs in Triton on half hour files, 96 kHz, spanning 2-3 months on a 50% duty cycle. It generally takes 36 hours or more to run on the high ram computers.  
  
  ---
  
  