The cuda runtime error is often misleading and can be the cause of different errors.
What you can do is let your network run on the cpu for debugging. This gives usually a better error behaviour and you can find the line which actually lead to the error.