By default, Magento uses the following directives for robots.txt
User-agent: ... Disallow: /checkout/ Disallow: /customer/ ...
But if you check those pages, meta values have INDEX,FOLLOW.
Don’t you think the value should be NOINDEX,NOFOLLOW instead?
submitted by /u/MagePsycho
[link] [comments]