Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
-
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this:
User-Agent: rogerbot Crawl-Delay: 5
Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop.
Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
-
Hello Dave. Thanks for your reply. We are aware this is not affecting us being temporary and exclusive to the MOZ bot so that's why we are worried about the data-set issues.
As I mentioned most of our excluded content are products, we can't be certain that MOZ has every keyword and that the ones discovered are being weighted correctly.
Understandably Shopify might never make robots.txt available so it would be nice for MOZ to identify the web as a shop hosted on Shopify (a moz.txt file) and apply a rate limiting, at the very least allow the user to control the crawl parameters from our control panels for those SaaS apps that block these core functions.
Hope MOZ and Shopify one day have a coffee and find a way to figure this out. But meanwhile, Is there any way to request crawls in specific folders? something like "domain.com/products/*****"
-
hey, Dave from the Help Team here.
The 430 error seems to be a result of shopify blocking our bot from accessing those pages temporarily. We have seen instances where this clears up after the second crawl, so keep your eye out for your weekly campaign update email in the meantime.
The good news is, that your human visitors will still be able to access your pages to do their shopping, phew!
Thanks so much for letting us know. We'll track this issue and look into a fix. I'm sorry I don't have better news for you at this time.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Bar doesn't show any data and keeps asking me to log in when actually I'm logged in.
Hi all, I've been using Moz Bar for years. It ran well until about three weeks ago. It suddenly failed to show the DA and PA of sites that I open after I log in. And it keeps asking me to log in when I did. I tried to uninstall the Mozbar extension and reinstalled it several times. Nothing worked. I also tried to uninstall Chrome and clear the cookies, still, nothing changed. Did anyone experience this? How do you solve it and make it run on the track? Any information will be appreciated. [admin edited support category]
Moz Bar | | Bennie223310 -
Need to solve "Oops our crawlers were unable to access" url for new campaign
I'm putting the url designfirstkitchenandbath.com and getting the "oops! our crawlers were unable to access the site. Since this site is a potential client, which shows up online, I can't get access to fix the code, plus while I can write a little html I don't feel comfortable working with hard, live code on someonelse's site. Anyone have a simple solution?
Moz Bar | | alisacromer0 -
How to upload the bulk Keywords with Tags in MOZ Rank Tracker Tool?
Trying to upload multiple keywords at a time with their different Tags. But here i can upload the keyword one by one also i am not able to associate tags with the keyword.
Moz Bar | | _nitman2 -
Moz crawler only crawls one page?!
Hello there, I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled. We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working. Hopefully someone can point us at the right direction. Thanks in advance, Jeremy
Moz Bar | | mediaxplain.nl0 -
Learn how to use MozBar to analyze competitors' schema markup. Get you Daily SEO Fix!
Schema markup helps Google and Bing identify what your (and your competitors') website pages are all about and as a result, helps search engines better index them. Better indexing can lead to better rankings for relevant keywords and phrases and herein lies the opportunity. In today's Daily SEO FiX, Jordan shows you how to use MozBar to analyze the schema markup of the competition and optimize your own site and pages for rich snippets. If you don't have MozBar, no worries! You may download it for free here. This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Moz Bar | | kellyjcoop1 -
Moz reporting for C-Blocking
Hey Mozers, I see Moz has a reporting tool for C-blocking and for november I had 330. Does this mean 330 Ip addresses came from the same location in the month of november?
Moz Bar | | rpaiva1 -
How do you stop Moz crawling a page?
Hello, I have a contact form which generates thousands of duplicate crawl errors. I'm going to use to block Google indexing these pages. Will this also block MOZ from crawling these pages and displaying the error? Thanks!
Moz Bar | | Seaward-Group0
