Solved by tom in post #4 
              
              
                Now you tricked yourself with too much linearity. 
While relu is nonlinear globally (else we would famously not have the universal representation property) it is linear in a neighborhood of almost every input (if you pardon the mathematics lingo), rendering your network linear in a neighborhood of y…