Solved by jmaronas in post #4 
              
              
                Yes they can. If you take a simple problem and compute the derivatives with back propagation, you will see that the gradients are scaled by the value of your cost error. 
Now imagine, if your cost is CE and 100*CE, in a convex optimization problem you will reach the same local minimum for the parame…