NNs with ODEs have been done successfully, e.g. https://github.com/rtqichen/torchdiffeq/ 1
In general, there is nothing to keep you from using the lower-level functions in PyTorch or mixing them with others.
The optimizers are impemented using them, see the code in torch/optim/.
When benchmarking, one has to be careful about the asynchronous nature of things. If your results are too surprising, chances are that there is room for refinement of your measurement methodology.
Best regards
Thomas