KFrank:

I’d like to expand on this a little. It is true that Linear layers introduce
no quadratic terms. Also the ReLU activation function is piecewise
linear. But many non-linear activation functions do have quadratic (and
higher-order) terms in their expansions, so they do introduce quadratic
terms into the overall function computed by the network.
(For example, pytorch’s ELU (“exponential-linear unit”) has a regime
where the quadratic term dominates.)


Yeah, but the thing is, usual networks won’t generate quadratic and linear “features” at once, so polynomials are approximated with something like additive models.