How to find the gradient when a black box I/O function is involved in evaluation of the loss? I am trying to learn a neural network $NN_\pi$ to minimize the loss function
$$  L_{\pi} = || Y_{true} - F(X_{true},    NN_{\pi}(X_{true}) ) ||^2 $$
where $F$ is a black box (I/O) function (we only have access to the output given an input to $F$, but we have no way to compute the gradient of $F$), and $X_{true},Y_{true}$ are training data.
The issue is we cannot backpropagate as we cannot find gradient of $L$ (as $F$ is an oracle function, we cannot differentiate it).
Can it be possible to learn $NN_{\pi}$ without knowing $F$?  Or, can we find the gradient with any other methods?
In my application, $X \in \mathbb{R}^2$, $NN_\pi(X) \in \mathbb{R}$, and $F(\cdots),Y \in \mathbb{R}^2$, but I am interested in general solutions.  None of these variables will be in a high dimensional space.

It is always possible to reframe model-fitting problems as reinforcement learning problems. The actions are the choice of model parameters and the reward is the negative of the error (here loss) function.
A possible implementation would be to use baseline comparison policy-gradient (aka the REINFORCE algorithm) . This requires that the likelihood of an action (here choice of neural-net parameters) is a differential function of its parameters.
A possible method

*

*Sample the weights/parameters to be used in the neural-net from a convenient distribution. An independent normal distribution for each parameter would be a convenient choice for this action distribution.$w_i\sim\left(\mu_i,\sigma^2\right)$.

*Use those parameters in the neural-net with the inputs of an example drawn from the training set and the weights drawn from the action distribution i.e. calculate $F\left(X,NN\left(X;\mathbf{w}\right)\right)$

*Calculate the loss function $L$ for the sampled example.

*Perform updates for all the parameters $\Delta \mu_i=\alpha\left(b-L \right)\left(w_i-\mu_i \right)$

*And repeat.

There are three hyper-parameters to choose with this method. 1) The variance of the parameters tried ($\sigma^2$): It is possible to make this a learnable parameter as well. But this can make choosing the learning rate trickier. It's probably safest to go small. Also, it is important to scale the inputs so that they have roughly the same range. 2) The learning rate ($\alpha$): important in all stochastic gradient algorithms. To ensure convergence, a small value is usually best but doesn't have to be too small if the variance is also small. 3) The baseline that the observed loss is compared to ($b$): It is known that the mean of $L$ isn't optimal but some fixed value near to the expected mean is the safest choice.
Note while this is possible, it may not be very efficient. This would be a problem if the neural-net is very complex and takes a long time to calculate for each new set of parameters.
