I have a model that computes a time varying covariance matrix for time series data. The data has about 40 features per time point, with a few hundred thousand time points. The forward pass and loss calculation are quite quick, ~1 or 2 seconds, but the backwards pass takes upwards of 5 minutes for each minibatch.
I’ve tried using the profiler to diagnose the issue, but it doesn’t seem to be overly helpful. I’ve uploaded the exported profile to dropbox here 25.
I’m struggling to glean any useful information from this profile, could someone take a look at it and see if I’m missing anything, or whether it indicates something I’m doing fundamentally incorrectly?