Filtering a dataset to get a more uniform distribution for neural network training
I'm looking into using artificial neural networks (ANN) to predict the reaction rates in my fluid instead of solving the full system of stiff ODEs. Some people from my lab have already done some work on that so I don't start from scratch but I am having problems with my applications. One of them I think relates to the quality of my dataset for training. We usually extract training data from CFD simulations that are either 1D/2D/3D. No matter what, we end up with a multidimensional array of data to feed to the neural network. To give you an idea of the size of the problem, I am looking into training 8 nets with 10 inputs and 1 output for each. I feel like a training set of about 100,000 points would be reasonable but the problem is that these 100,000 points need to cover a specific region of my multi-dimensional space. 650
Simply extracting the whole flowfield from my simulations is not very good for 2 reasons:
* For each snapshot, only a small portion of the points lie in the region where I need a high sampling to make sure my training is accurate
* As I compile snapshots together, I end up with many near-duplicate points which (I believe) have a negative effect on my ANN training by a) biasing the training by putting more weight on these regions b) adding unnecessary points.
So I've been trying to filter the points I record before including them in my training set. As I see it, that involves checking whether a new point is within a certain n-dimensional radius of every point of my dataset. This brute force approach, which barring a few tricks scales like n^2, works so-so for extracting 10,000 points out of 100,000 (takes 30 min say) but breaks down as I increase the sizes and numbers of the snapshots... Clearly, there must be a more clever way of doing this, but I am not sure in which direction to start looking. I first tried with python and could move to FORTRAN to speed things up but I feel like I should look for a better strategy first. Is my only hope some kind of k-d tree? I have little to no experience with them and the problem that I see is that my tree will grow as I build my dataset and this can only increase the complexity. Would a python k-d tree library suit my need? Should I move to FORTRAN given the size of my problem? Any advice is appreciated, thank you ! 650