Answers.Body: <p>Yeah the server is probably misconfigured. Craft relies on a 404 happening to do some of what it does, but the status code should be changed before the response is returned.</p>
  
  <p>I don't have enough information to tell you how it is misconfigured. Would depend on the webserver (Nginx/Apache/whatever) and we'd need to see the specific config.</p>
  
  <p>There also is no <code>/sitemap.xml</code> on the site.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T20:31:06.853Z
Answers.LastActivityDate: 2019-12-09T20:31:06.853Z
Answers.UserDisplayName: andrew.welch
Body: <p>I'm trying to crawl a site, I've tried using Screaming Frog and <a href="https://www.internetmarketingninjas.com/seo-tools/google-sitemap-generator/" rel="nofollow noreferrer">https://www.internetmarketingninjas.com/seo-tools/google-sitemap-generator/</a> and both return the same results when I try a crawl.  It finds the home page, but it then will give 404's for all the links it finds on the page.  If I got to the site manually I don't get 404's.</p>
  
  <p>I'm new to Craft CMS so I don't know much (yet).  Inherited this site not too long ago.</p>
  
  <p>Site is: <a href="https://highparknaturecentre.com/" rel="nofollow noreferrer">https://highparknaturecentre.com/</a></p>
  
  <p>In addition, I also noticed that If I go to <a href="https://highparknaturecentre.com/about" rel="nofollow noreferrer">https://highparknaturecentre.com/about</a> the server returns http status 404 but returns the body content.  Could this be a server issue?</p>
  
Tags: <seo>
Title: Trying to crawl a site
ThreadText: # Trying to crawl a site
  
  (Asked by: username_0 on 2019-12-09)
  
  I'm trying to crawl a site, I've tried using Screaming Frog and <https://www.internetmarketingninjas.com/seo-tools/google-sitemap-generator/> and both return the same results when I try a crawl. It finds the home page, but it then will give 404's for all the links it finds on the page. If I got to the site manually I don't get 404's.
  
  I'm new to Craft CMS so I don't know much (yet). Inherited this site not too long ago.
  
  Site is: <https://highparknaturecentre.com/>
  
  In addition, I also noticed that If I go to <https://highparknaturecentre.com/about> the server returns http status 404 but returns the body content. Could this be a server issue?
  
  **Tags:** seo
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-09. Score: 2
  
  Yeah the server is probably misconfigured. Craft relies on a 404 happening to do some of what it does, but the status code should be changed before the response is returned.
  
  I don't have enough information to tell you how it is misconfigured. Would depend on the webserver (Nginx/Apache/whatever) and we'd need to see the specific config.
  
  There also is no `/sitemap.xml` on the site.
  
  
  ---
  
  