I need some help please!

I’m crawling a Magento site and there are thousands of URLs that contain ?store=3&version=(random number)&signature=(long number)&timestamp= Why are there so many of these URLs? They also seem to be 301s, non-indexable, and redirected. What is happening?!?! Whatever it is, it’s completely eating away the crawl budget.

The crawl is still going, currently at 690,981 URLs and only 13% done with the crawl! I agree I can disallow the URLs in the robots.txt, I just wish we knew why all of these URLs are being created in Magento. Is it a button they can click to stop them from being created? There are thousands of these timestamp URLs for each page. They aren’t indexed which is good, but they are also 301s.

submitted by /u/LoWkEyPyRaT
[link] [comments]