Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are backlinks from the Episode Description field on Spotify "follow" or "no follow"?
Are backlinks from the Episode Description field on Spotify "follow" or "no follow"?!
Link Building | | FLLocal0 -
Why Google Search Console Data is different from Moz Data for my website
Hi All I am running a website, I have been using Moz since Feb 2021. Kindly go through these pics My question is why Moz is showing 2K plus backlinks while Google search console is showing just 1253 backlinks. Why fewer links in the google search console is less? How can I increase Google search console backlinks? Also, Moz is showing 90+ DA backlinks but those websites are not showing by the Google search console. What should I do to let google consider them? m0JGjBQ.jpg Cmp8ei3.png
Link Building | | ssubodhsingh0 -
Back link from site with DA of 72 to a website domain. Clicking on the link redirects to our website not the attended one.
Hi,
Link Building | | JIMBO16
I've ran a back link check and discovered a good back link to a site which then gets redirected back to my company's website. I have a feeling that an old SEO agency has purchased a small website which has a decent link back from a relevant organisation with a high Domain authority and then redirects the domain to our website to get the link juice. What are your thought on this? Is this really bad practise and possibly damaging? Thanks, Jim0 -
Sponsored posts against Google guidelines?
I'm a bit confused. Every blog I try to outreach on always give me a quote for a sponsored post. Isn't this against Google guidelines because you paying for a link technically even though your paying for a post? Do you guys buy sponsored posts? Should this be avoided? How do you outreach to a blog that offers sponsored posts? They can smell that you want a link from a mile away and give you a nice fat quote.
Link Building | | The_Kiwi_Man1 -
What is the best way to remove a link that redirects to a spammy site?
I've got a new client and I'm trying to clean up their backlinks. There are several links that all redirect to this spam site http://www.expert-lender.com. All of the websites appear to be real, i.e. http://www.sammorganhomes.com/ but the actual links i.e http://www.sammorganhomes.com/wp-fav/backup/supplement/semitruckleasing.html are in a sub-directory and redirect to the spam site. I don't know if these links were from the previous SEO company or if they paid someone to create these or if these sites have been hacked. Can anyone tell me what is going on here and what should I tell these site owners?
Link Building | | Total-Design-Shop0 -
Will Google penalize for backlinks on non-indexed pages?
Have a client who has a bunch of links from spammy sites that aren't even indexed by Google. Will Google penalize (i.e. Penguin) for backlinks from spammy sites that aren't indexed? I'm guessing "Yes, Google still crawls non-indexed sites and may penalize the sites linked to from the banned site." Anyone have any data or experience with this? Anyone have a non-indexed or banned site? Does Google still crawl it?
Link Building | | AdamThompson0 -
Could google recognize urls in data-attributes?
Hi, could google's crawler recognize the following example as a link and will be crawl the url? any experience? ty for your help
Link Building | | censeo0 -
How does Google interpret articles or prepositions in languages where it's attached to the (key)word?
Hi, All! This is for any foreign language SEOs where articles or prepostitions such as "the" "to" "in" or anything else are actually part of the word they are modifying and not a separate word, as in English: How does Google understand those words on-page and in anchor text? If you want to optimize for the word "house", and your content/anchor text says "the house" or "in the house" (again, all one word) - what does Google count that as? Does it count toward "house"? Does it count toward "in the house" only? Does it count toward "house" but not as much as if you had just put "house"? I end up sometimes writing slightly grammatically-off content because I want to optimize for the keyphrase - but is that necessary? Obviously different languages might be different, but you can probably project a little from one to the others. Thanks in advance!
Link Building | | debi_zyx0