Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Bing Indexation and handling of X-ROBOTS tag or AngularJS
-
Hi MozCommunity,
I have been tearing my hair out trying to figure out why BING wont index a test site we're running.
We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing.We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz
However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site.
In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header.
With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag.
However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive.
I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing.
I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages.
I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content.
Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages.
Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl.
Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests?
Thanks in advance for your assistance....
-
Thank you for the update Kavit.
-
Hi Everett and Fellow Mozzers,
I have been away overseas so wasn't able to put up an update.
Eventually, managed to get a hold of someone at BING within the tech team who told me that the reason that they didn't index the pages was simply because of popularity.
It isn't enough to have unique content, design and structure on your site, it is also vital to have traffic, links and mentions as external signals.
We also got word that dynamic sites and pre-render content will be acceptable for BING so we're resting easier at night these days.
Development on the site continues as per schedule and we will be launching the proper site this year on a highly authoritative domain which should yield very different results to the test we put together.
Hopefully, this will help someone else who is on a similar pathway.
Everett, I would like to thank you again for taking the time to read, reply and help us with our analysis.
Thanks!
-
Hi Everett,
Thank you for the analysis and deeper insights.
I did make the changes to the test pages bar the design template.
We added the unique titles, meta descriptions and meta keywords.
We added completely unique content to all three pages with no other instances of this content appearing on the web at all.
The pages are now also interlinked and also linked from the top of the homepage so none of them are orphan pages.
sitemaps have been updated and resubmitted.
The latest version has been out a week so far, but no response from BING as yet.
Thanks,
Kavit.
-
Hello Kavit,
I would suggest putting unique Title tags, meta descriptions and content on those pages. They are very thin as it is, and all of the content is boilerplate.
There are 57,100,000 results on Bing for: "Search for an Australian Business, Government Department or Person" which is the content on the home page you shared.
There are 60,600 results on Bing for: ""There was a table set out under a tree in front of the house, and the March Hare and the Hatter were having tea at it" which is the content on this page: http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/no-metatag.html .
And so on. I can see why Bing wouldn't want to add yet another thin, duplicate, orphan page to their index. My advice would be to build out those test pages with a design template and to put original content, title tags and meta descriptions on all of them. Then repeat your test.
-
Hi Everett,
Thank you for taking the time out to read and respond.
The URL we have setup for testing is: wp-seospike-weblbl.naws-sensis.com.au
We have 3 control pages (all flat HTML pages) that we setup and put online for bing to crawl:
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/no-metatag.html - no robots metatag and allowed to crawl and index.
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/metatag.html - page with a noindex metatag not to be crawled and indexed
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/metatag-header.html - X-Robots meta tag NOINDEX
http://wp-seospike-weblbl.naws-sensis.com.au - homepage with no robots exclusion
Ideally, I expected the homepage and the no-metatag page to be indexed at least.
I am familiar with the builtvisible documentation that they've put out.
My main pain point is that even the flat HTML pages are getting ignored, so I can't even test the deeper AngularJS developed pages since my control group is not delivering results as it should.
a site command on the above domain on bing shows no results.
Thanks again!
-
Is there any chance of getting a URL for the domain in question?
Have you read this yet?
https://builtvisible.com/javascript-framework-seo/What are the URLs like that you're asking Bing to index? Which is closest?
Hashbang
http://www.IWishJSFramworkWebsitesWouldGoAway/#!Escaped Fragment
http://www.IWishJSFramworkWebsitesWouldGoAway/?escaped_fragment=Base URL using Angular's $location service to construct URLs without the #! via the HTML5 History API http://www.IWishJSFramworkWebsitesWouldGoAway/
I know this doesn't answer your question, but hopefully it will get the discussion started.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
NO Meta description pulling through in SERP with react website - Requesting Indexing & Submitting to Google with no luck
Hi there, A year ago I launched a website using react, which has caused Google to not read my meta descriptions. I've submitted the sitemap and there was no change in the SERP. Then, I tried "Fetch and Render" and request indexing for the homepage, which did work, however I have over 300 pages and I can't do that for every one. I have requested a fetch, render and index for "this url and linked pages," and while Google's cache has updated, the SERP listing has not. I looked in the Index Coverage report for the new GSC and it says the urls and valid and indexable, and yet there's still no meta description. I realize that Google doesn't have to index all pages, and that Google may not also take your meta description, but I want to make sure I do my due diligence in making the website crawlable. My main questions are: If Google didn't reindex ANYTHING when I submitted the sitemap, what might be wrong with my sitemap? Is submitting each url manually bad, and if so, why? Am I simply jumping the gun since it's only been a week since I requested indexing for the main url and all the linked urls? Any other suggestions?
Web Design | | DigitalMarketingSEO1 -
Problems preventing Wordpress attachment pages from being indexed and from being seen as duplicate content.
Hi According to a Moz Crawl, it looks like the Wordpress attachment pages from all image uploads are being indexed and seen as duplicate content..or..is it the Yoast sitemap causing it? I see 2 options in SEO Yoast: Redirect attachment URLs to parent post URL. Media...Meta Robots: noindex, follow I set it to (1) initially which didn't resolve the problem. Then I set it to option (2) so that all images won't be indexed but search engines would still associate those images with their relevant posts and pages. However, I understand what both of these options (1) and (2) mean, but because I chose option 2, will that mean all of the images on the website won't stand a chance of being indexed in search engines and Google Images etc? As far as duplicate content goes, search engines can get confused and there are 2 ways for search engines
Web Design | | SEOguy1
to reach the correct page content destination. But when eg Google makes the wrong choice a portion of traffic drops off (is lost hence errors) which then leaves the searcher frustrated, and this affects the seo and ranking of the site which worsens with time. My goal here is - I would like all of the web images to be indexed by Google, and for all of the image attachment pages to not be indexed at all (Moz shows the image attachment pages as duplicates and the referring site causing this is the sitemap url which Yoast creates) ; that sitemap url has been submitted to the search engines already and I will resubmit once I can resolve the attachment pages issues.. Please can you advise. Thanks.0 -
Is placing an H1 tag below a slideshow a bad practice?
Hello All, It is to my understanding that it is a best practice to have a single H1 headline that corresponds to your title tag at the top of your page, above your content for the best On-Site Optimization. We are developing a site that uses a big slideshow with text on each slide, and are concerned that placing the H1 Headline below this will be a bad practice. Would a better option be to have the slide show text on the image and place no alt tags on the slides, so that the crawlers accessing the page overlook this and see the H1 below the slider first? We need to maintain the slider for design purposes, but would like the site to optimized. A similar example to what the slider will look like is as follows: http://www.boviskyle.com/ However, we look to have optimized, "10x" content below the slider with a solid H1 headline as well. Thank You!
Web Design | | Armen-SEO0 -
Using H tags and its maximum Limits
hi..
Web Design | | funclub247
I want to Know what is a Maximum limit of using H tags in One Page : for Eg : I Know That I Can use Only One H1 Tag per Page, What about Other H tag Limit..
h1 - 1 time Maximum
h2 - ..?
h3 - ..?
h4 - ..?
h5 - ..?
h6 - ..?
h7 - ..?
.....
i want to target more than 30 key word using H tag as a header of the paragraph...1 -
Other tags inside an H1 tag
So I have a situation with the website I'm currently redesigning where the H1 titles are supposed to mix colors per the current brand strategy. The branding crew is adamant that this has to be done so there is no use in saying "just don't do it". To accomplish this I'm wrapping the words that need to be the other color in a . Additionally, some pages have a "sub text" as part of the title, floated to the right and in a smaller font but with the same multi color treatment. I'm wondering if the sub text should be in an H2 and positioned to the right or if it would be beneficial to have the text in the H1 as well. An example of what I'm talking about would be something like this: "Big Shoes for Big Guys - Nike Shoes" In that, the "Big Shoes" and "Nike" would be one color and the "for Big Guys" and "Shoes" would be another. I can imagine having the "Nike Shoes" as part of the H1 would be a good idea in some respect but I'm not certain of that. In order to make that happen I can only think of one way to do it: -H1-
Web Design | | EscaladeSports
Big Shoes
-span- for Big Guys -/span-
-div- Nike
-span- Shoes -/span-
-/div-
-/H1- So that brings me back to the original concern, do search engines care about tags inside the H1? The only other way to accomplish the color changes that I can think of would be to have a fairly large chunk of javascript setup to go through H1's to colorize them using the span tags. That is unless GoogleBot has started to execute javascript while crawling the sites now...1 -
Duplicate H1 tag IF it holds SAME text?
Hello people, I know that majority of SEO gurus (?) claim that H1 tag should only be used once per page. In the landing page design I'm working with, we actually need to repeat our core message stated in H1 & H2 - at the bottom of the page. Now the question is: Can that in any way cause any ranking penalty from big G? In my eyes that is not attempt to over optimize page as it contains SAME info as the H1 & H2 at the top of the page. Confusing, so I'm hope that some SEO gurus here will share some light on this. Thanks in advance!
Web Design | | RetroOnline0 -
How do search engines interpret <hgroup>...</hgroup> tags?
Hi there. I'm building an HTML 5 site and through research of new HTML 5 elements I've seen little conclusive information about the interpretation of the new <hgroup>tag, in terms of SEO application and interpretation. In particular does Google interpret the nested heading tags as individual elements or does it combine them into one entity? For example, say I have: <hgroup> # Article Heading ## Article Sub-heading </hgroup> How is this interpreted by Google and what would be some good SEO best practices regarding the <hgroup>tag in HTML5: Is it interpretted as a single tag (" Article Heading: Article Sub-heading ") or two separate heading tags (one and one )? Also, how much does the ordering of the tags matter (say for example I wanted something like the following for visual purposes? <hgroup> ## Article Sub-heading # Article Heading </hgroup> One last thing: is it safe to assume that it is indeed OK to have multiple tags on a single page (as referenced by Matt Cutts a while back in a Webmaster Video)? Thanks! </hgroup> </hgroup>
Web Design | | LMDNYC2 -
How to Add canonical tags on .ASPX pages?
What is the proper way (or is it possible) to add canonical tags on website pages that end in .aspx? If you add a canonical tag to the Master Page it will put that exact canonical tag on every page, which is bad. Is there a different version of the tag to put on individual pages? And one to put on the home page without the Master Page error?
Web Design | | Ryan-Bradley0