Solved by Wafaa_Wardah in post #2 
              
              
                I found the solution (Thank you DeepLizard) 
I needed to flatten my conv1 output properly before passing into the fc1 layer.