Answers.Body: <p>I know, the OP is a year old, but here is my answer:</p>
  <p>I guess your parameter settings are suboptimal.</p>
  <p>I just tried PAMGuide on a dataset I had on my disk</p>
  <pre><code>120 files, 30 s each, 192 kHz sampling
  </code></pre>
  <p>was running PAMguide (with Matlab) with default configuration</p>
  <pre><code>PSD, 1 s window, no time average, PlotType Both
  </code></pre>
  <p>processing lasted &gt;900 s, plot never showed up and matlab effectively froze PC (16GB laptop). Had to power off and on again to continue.</p>
  <p>Changed parameter in PAMguide to</p>
  <pre><code>PSD, 1024 samples window, 10s time average, PlotType Both
  </code></pre>
  <p>and all worked fine</p>
  <pre><code>Analysis complete in 119.5887 s.
  Plotting... done in 1.8633 s.
  Computing noise level statistics... done in 0.048294 s.
  Plotting... done in 1.0546 s. 
  </code></pre>
  <p>Suggest, to reduce processing window to a much lower level.</p>
  <p>The 1 s window gives you 1Hz resolution and in my case 192000 frequency bins, but screens have typically only about 1000 pixels.</p>
  <p>Same is with time averaging: if you have a day of data to visualize, then with 1 s average you would have 86400 samples, but screens are very likely less than 2000 pixels wide, so 10 to 30 s averaging brings your data to a reasonable size.</p>
  <p>OK, your may have to play with the parameters, but start first with the smallest window and the longest time averaging. In my case in order to get PSD statistics I needed to reduce time averaging to 2.5 s, which needed nearly the same amount of time for analysis and plotting</p>
  <pre><code>Analysis complete in 122.0571 s.
  Plotting... done in 0.58137 s.
  Computing noise level statistics... done in 0.10046 s.
  Plotting... done in 0.88013 s.
  Writing concatenated output array...done in 0.72288 s.
  </code></pre>
  <p>When reducing the analysis window, make sure you always have averaging window specified, otherwise you will get far too many time samples and will blow-up your memory.</p>
  <p>Comment to your crash: very likely was the page file (the file where virtual memory is off-loaded) was too small and exhausted, generating a crash.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-24T07:34:41.233Z
Answers.LastActivityDate: 2023-10-24T07:34:41.233Z
Answers.UserDisplayName: WMXZ
Answers.Body: <p>I'm not used to PAM, but MATLAB has some functionalities to work with large data sets, like datastore and tall arrays. You can check available options on this <a href="https://www.mathworks.com/help/matlab/large-files-and-big-data.html" rel="nofollow noreferrer">support page</a>.
  Also, since the whole process may take a while, it may be useful to spend a few more seconds when running setting up a <a href="https://www.mathworks.com/help/parallel-computing/" rel="nofollow noreferrer">parallel processing pool</a>, which will speed up the process in the long run. A simple way and example is when using <code>parfor</code> (<a href="https://www.mathworks.com/help/parallel-computing/parallel-for-loops-parfor.html" rel="nofollow noreferrer">a parallel <code>for</code></a>).</p>
  <p>However, it was mentioned in the comment that &quot;The first file is analyzed in about 20 seconds and the last file that is analyzed before it crashes takes about 400 seconds&quot;. For what I understood, it doesn't seem you need to load all the files to analyse the data.
  So maybe in this case it's just about rearranging the programming logic to load a file, analyse it, save results, clear memory, repeat.</p>
  <p>(This &quot;answer&quot; should be a simple comment, but I don't have the reputation needed yet. Just hope it helps.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-23T22:51:30.973Z
Answers.LastActivityDate: 2023-10-23T22:51:30.973Z
Answers.UserDisplayName: Phxuibs
Body: <p>I am trying to do soundscape analysis (LTSA, PSDs, TOLs) using PAMGuide in MATLAB. However, after loading about 50-100 files/455 files (depending on my frequency range), MATLAB crashes. Given that the files are very large (about 300MB), I suspect a memory issue. Does anyone have a similar issue and solution?</p>
  <p>Thanks in advance!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-17T16:31:39.957Z
Comments.Text: Welcome @MelanieSmith. Could you give us a bit of info about how much memory your computer has? I have not had this issue with PAMguide. You can look at your memory usage using the Windows Task Manger > Performance tab and see how your memory is changing as the files load.
Comments.UserDisplayName: selene
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-25T00:52:39.223Z
Comments.Text: Hi Selene, my computer has 16 GB of memory, which gave me an error for array size. I then changed my settings so MATLAB can use virtual memory. MATLAB memory usage increases as I load more files and crashes around 40-50 GB. As I load more and more files, the time it takes to analyze those files also increases. The first file is analyzed in about 20 seconds and the last file that is analyzed before it crashes takes about 400 seconds. All the files are the same length, size, etc.
Comments.UserDisplayName: Melanie Smith
Tags: <soundscape><ltsa><matlab>
Title: MATLAB Crashed When Using PAMGuide for Large Datasets
ThreadText: # MATLAB Crashed When Using PAMGuide for Large Datasets
  
  (Asked by: username_0 on 2022-10-17)
  
  I am trying to do soundscape analysis (LTSA, PSDs, TOLs) using PAMGuide in MATLAB. However, after loading about 50-100 files/455 files (depending on my frequency range), MATLAB crashes. Given that the files are very large (about 300MB), I suspect a memory issue. Does anyone have a similar issue and solution?
  
  Thanks in advance!
  
  **Tags:** soundscape, ltsa, matlab
  
  > **Comments:**
  > **username_1**: Welcome @MelanieSmith. Could you give us a bit of info about how much memory your computer has? I have not had this issue with PAMguide. You can look at your memory usage using the Windows Task Manger > Performance tab and see how your memory is changing as the files load.  
  > **username_0**: Hi Selene, my computer has 16 GB of memory, which gave me an error for array size. I then changed my settings so MATLAB can use virtual memory. MATLAB memory usage increases as I load more files and crashes around 40-50 GB. As I load more and more files, the time it takes to analyze those files also increases. The first file is analyzed in about 20 seconds and the last file that is analyzed before it crashes takes about 400 seconds. All the files are the same length, size, etc.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-10-24. Score: 2
  
  I know, the OP is a year old, but here is my answer:
  I guess your parameter settings are suboptimal.
  I just tried PAMGuide on a dataset I had on my disk
  120 files, 30 s each, 192 kHz sampling
  
  was running PAMguide (with Matlab) with default configuration
  PSD, 1 s window, no time average, PlotType Both
  
  processing lasted >900 s, plot never showed up and matlab effectively froze PC (16GB laptop). Had to power off and on again to continue.
  Changed parameter in PAMguide to
  PSD, 1024 samples window, 10s time average, PlotType Both
  
  and all worked fine
  Analysis complete in 119.5887 s.
  Plotting... done in 1.8633 s.
  Computing noise level statistics... done in 0.048294 s.
  Plotting... done in 1.0546 s. 
  
  Suggest, to reduce processing window to a much lower level.
  The 1 s window gives you 1Hz resolution and in my case 192000 frequency bins, but screens have typically only about 1000 pixels.
  Same is with time averaging: if you have a day of data to visualize, then with 1 s average you would have 86400 samples, but screens are very likely less than 2000 pixels wide, so 10 to 30 s averaging brings your data to a reasonable size.
  OK, your may have to play with the parameters, but start first with the smallest window and the longest time averaging. In my case in order to get PSD statistics I needed to reduce time averaging to 2.5 s, which needed nearly the same amount of time for analysis and plotting
  Analysis complete in 122.0571 s.
  Plotting... done in 0.58137 s.
  Computing noise level statistics... done in 0.10046 s.
  Plotting... done in 0.88013 s.
  Writing concatenated output array...done in 0.72288 s.
  
  When reducing the analysis window, make sure you always have averaging window specified, otherwise you will get far too many time samples and will blow-up your memory.
  Comment to your crash: very likely was the page file (the file where virtual memory is off-loaded) was too small and exhausted, generating a crash.
  
  
  ---
  
  ### Answer by username_3 on 2023-10-23. Score: 1
  
  I'm not used to PAM, but MATLAB has some functionalities to work with large data sets, like datastore and tall arrays. You can check available options on this [support page](https://www.mathworks.com/help/matlab/large-files-and-big-data.html).
  Also, since the whole process may take a while, it may be useful to spend a few more seconds when running setting up a [parallel processing pool](https://www.mathworks.com/help/parallel-computing/), which will speed up the process in the long run. A simple way and example is when using `parfor` ([a parallel `for`](https://www.mathworks.com/help/parallel-computing/parallel-for-loops-parfor.html)).
  
  However, it was mentioned in the comment that "The first file is analyzed in about 20 seconds and the last file that is analyzed before it crashes takes about 400 seconds". For what I understood, it doesn't seem you need to load all the files to analyse the data.
  So maybe in this case it's just about rearranging the programming logic to load a file, analyse it, save results, clear memory, repeat.
  
  (This "answer" should be a simple comment, but I don't have the reputation needed yet. Just hope it helps.)
  
  
  ---
  
  