The issue is not over fitting. In fact linear regression is extremely exploitable with this method. See <a href="http:&#x2F;&#x2F;karpathy.github.io&#x2F;2015&#x2F;03&#x2F;30&#x2F;breaking-convnets&#x2F;" rel="nofollow">http:&#x2F;&#x2F;karpathy.github.io&#x2F;2015&#x2F;03&#x2F;30&#x2F;breaking-convnets&#x2F;</a>