Access denied by robots.txt

[vswebdesign]

I'm trying to run a link checker prior to going live next week. I keep getting an Access Denied by Robots.txt error. I had my robots.txt file set up to block everything while I was working on it. I just recently changed it back to the defaults:

Sitemap: http://garlicfestival.3dcartstores.com/sitemap.xml

# Disallow all crawlers access to certain pages.
User-agent: *
Disallow: /checkout.asp
Disallow: /add_cart.asp
Disallow: /view_cart.asp
Disallow: /error.asp
Disallow: /shipquote.asp
Disallow: /rssfeed.asp
Disallow: /mobile/


Anyone know why it is still getting blocked? Does it take a while for the new robots.txt to work?

[thecartdesigner]

Not sure if its an issue, but you can try clearing your sitemap cache and then make sure you also clear your cache in your store settings as well.

Here are the instructions for clearing your sitemap cache:

https://support.3dcart.com/Knowledge...sitemaps-cache

[vswebdesign]

Thanks for the link. However, there is no sitemap_cache.txt file. There is a sitemap_1.xml. Is this the same thing, but in XML format?

[elightbox]

There should be a sitemap_cache.txt in your /web folder. Just delete it.

You can also test your robots.txt here:
Robots.txt Checker

[vswebdesign]

Alas, there is no such file in my /web directory.

[thecartdesigner]

I think it has changed a bit with version 6+ as I also do not show the file in ftp anymore either. Go (Marketing-->SEO Tools) 3rd section "Sitemap", (Update Sitemap)