Feb 20, 2007
I have been all bitter and twisted the past few weeks about Google putting some of my pages into the supplemental index.And not just my rubbish pages, some of my best content has gone supplemental. After chewing on the problem for a couple of weeks, I have it figured out.
It's the trailing slash on my URLs, or lack thereof.
As odd as this may sound, www.example.com/page/ is not the same as www.example.com/page - in Google's eyes they are 2 different pages.
Above: This basket of crap off my desk pretty well sums up what I think of supplemental results
Supplemental-o-meter
One of my many supplemental pages is my spam-o-meter.- I know for a fact that the spam-o-meter drives a decent amount of traffic to this site
- It's a good bit of link bait - that is, it has accumulated a respectable amount of links that I didn't have to ask for (just like Google recommends)
- It's a PR6 page, respectable in anyone's book
- has been online for almost a year so we aren't talking brand new or sandboxed
- Ranks ok in the serps
- Is not duplicated anywhere
- Has no outbounds on the page
- I haven't accumulated any links to this page from dubious sources
I'm pretty sure this page should not be lumped into the category of low quality and forgotten pages that is the supplemental index - or I'll really have to start questioning how much I know about this SEO game.
Want proof?
Check out the following URLs (sorry, I can't direct link them as my AJAX script will get all clever and fudge the URLs into something else).http://www.ragepank.com/spam-o-meter/ (PR6)
http://www.ragepank.com/spam-o-meter (PR0)
My point is that the URL with the trailing slash is PR6 and the other is PR0. If that doesn't sway you, see if you can notice anything different in the following images, taken from a Google site:ragepank.com query...
No slash - and in the supplementals
With the slash and it's all sweet
I am really careful at maintaining consistent link structure - I always use the trailing slash on my sites because I like it to feel consistent. But people who link to your site, are they as consistent? Unsurprisingly, people don't care about your trailing slash when they link to you - they go with what works. Googlebot follows these inbound links and indexes them - just like it's supposed to do.
What doe it all mean?
For me, having pages in the supplemental index is not good for credibility on a SEO related site. So now I have found out what the problem is.Obviously, I want the result without the slash gone, so doing a 301 redirect is the obvious answer.
Does nobody else give a shit?
I think most people would agree this situation of having a version in the main index and a version in the supplemental index is not desireable. A 301 redirect would definitely fix the problem, and I actually did this in a prior version of my CMS.At the time, I noticed many automated tools would break when accessing my site - they would often trip up on the 301 redirect. On top of that, most sites didn't seem to care about trailing slashes, so I decided I didn't need to care either and removed the redirect code.
Now when I take a look around, I see that the good sites do care about trailing slashes.
See what happens when you go to http://www.seobook.com/glossary (note no trailing slash).
And notice how the Wikipedia shows a clear difference between a slash and no slash
I'm going to go with what Aaron Wall and Wikipedia do. 301 redirects on the no-slash pages it is!
Do you have pages in the Supplemental Index?
Enter the following query into Google.site:www.example.com *** -sljktf
www.example.com should be replaced by your own site of course. This should bring up a list of the supplemental pages on your site.
Conclusions
I would humbly suggest you take a careful look at your site, if you use folder based URLs. I thought I knew a thing or two about how to prevent duplicate content, and I should have trusted my instincts instead of following the sheep.I'll update this post when the changes are made - I fully expect Google to remove my supplemental results within a couple of weeks.
23 Comments
Hi Martin,
Yes, this is recent (20 Feb 07). Must add the dates somewhere on the page...
Without knowing the URL of your client's site it's hard to comment directly (feel free to email me the URL if you want another opinion).
But what you are saying makes sense.
I have been a fan of good internal linking for ages. I first noticed that by using good internal navigation, you could often get the same PR as your homepage for a 30 odd page site.
My internal navigation always points to the slash version - I'm pretty good at keeping this consistent. But the problem arises when people start linking in from outside (which you want). In my case, it's a very real possibility that the spam-o-meter page will accumulate more link juice from outside links than from internal links. In that case, Google might display the noslash version in the SERPs, which is not what I want.
By allowing both versions to work, they compete with each other. One at maybe 70% strength, and the other at 30%. By 301ing them together, you get a single page at full strength.
This part of on-page SEO I like to call "technical SEO". Trailing slashes, just like frames, flash, querystrings and javascript, are technical obstacles that prevent you from ranking well. Having good technical SEO doesn't make you rank well - it simply prevents you from ranking poorly. The job of ranking well is still left up to content and links.
I think a site with good technical SEO and good content SEO needs less links to rank well.
I'd like to hear how you get on with the changes you made. I'd be pretty confident you will see an improvement here, even if not straight to page one. And keep an eye on the PR as well - you might find your pages getting better internal PR at the next foolbar update because of this change.
Harvey.
we had a site before in that we had approx 18,000 pages listed in google but due to some circumstances we have to close our old site and we created a new site with all the same datas and links that we used in our old site.
Within 2 months near about 7000 pages were added of our new site by google search engine, now a days we are suffering a trouble that most of the pages are having supplemental results.
1. How to remove supplemental result from my new site ?
2. If we add or remove some of the contain on the supplimental pages, can supplemental results will be
removed & will it be removed forever or it will be removed for sometimes.
3. How should i improve my Page rank in google ?
reply will be highly appreciated
Hi Mike,
The link in your comment doesn't work for me right now, but there are two things I find that can put your pages into the supplemental index.
1. Not enough authority
You say you have moved all the content to a new domain. Perhaps this new domain does not have the link authority of the old domain? Google may not be taking the new domain seriously until it has better incoming links, or has aged a little.
2. The other thing I find causes supplemental results is content that looks too similar to other content. Make sure every page has it's own title, meta description and H1 heading. Make sure the pages don't look too "machine generated" as Google is getting better at picking up on this.
I'll try your link again later, or if you are referring to a different site, you can email it to me and I'll take a look in more detail.
Harvey.
Patrick - Apr 11, 2007
Hi Mike,
it's April 10 now, and I was wondering if you had seen a difference as a consequence of the trailing slashes. Does it *really* lift pages out of the index ?
I am a bit new at this.. and I am confused: did you 301 redirect the pages without trailing slashes to the pages with a trailing slash or vice versa ?
I also do not really understand the logic of this. I know trailing slashes help in a somewhat lower loading time and server load, but why would google favour the pages with trailing slashes over those without ? You refer to being systematic with internal linking.. and would appreciate some additional insights here
Thanks so much
Patrick
Patrick - Apr 11, 2007
Oops.. I meant Harvey, not Mike. Sorry about that !
Hi Patrick,
I hadn't actually checked on those supplemental pages since writing this article, but have just had a look now.
Guess what - it worked, though I never really had any doubts about this.
Previously, there were supplemental results on the first page of a site:ragepank.com query. Now, I don't see any supplementals (at least not in the first 10 pages).
The case on the site is a little bit special, as it was not a matter of upgrading pages from the supplemental index to the main index. What has happened here is the supplemental pages have been deleted from all indexes, but the 301 redirect lets me keep any link juice that these pages may have. This will increase the PR of the pages with the slash.
I 301ed the no-slash version to the slash version. It doesn't really matter which way you choose to go - consistency is the key here.
Google follows links. If you link to the no-slash version, Google will follow it. If you link to the slash version, Google will follow that instead. If you link to both, then Google will follow (and maybe index) both versions, which leads to problems as discussed above. By adding the 301 redirect, this solves the problem, but it's nice to just link to the correct version anyway throughout your site.
When I say systematic linking, this just means choosing slash or no-slash and making sure all your internal links point to that version. Just a matter of keeping a tidy house.
Hope this answers your questions.
Harvey.
Patrick - Apr 12, 2007
Thanks very much ! It was very useful advice.
Patrick - Apr 12, 2007
I do have another (related) question though. Although I understand the function of a 301 redirect, I am not sure I understand the role of the trailing slash completely. Would it matter if noone had linked to the slashed version and would it matter if in the website itself.. there is no use of the slashed version at all. In that case you would be consistent simply by not using it! Or is it that Google just gives more priority to a slashed url as opposed to a non slashed one ?
I'm not a Google engineer, so you can never be 100% certain of these things.
But I can't think of a single reason why Google would prefer one over the other. So I don't believe there is any priority given to slashed URLs.
What is important is to be consistent within your own site. I happen to prefer the trailing slash, but that's just me.
Patrick - Apr 13, 2007
Great ! Thank you.
Patrick - Apr 13, 2007
As some feedback I can share some of our own experiences with supplementals. It is (in our experience) very much the case that a few good external or even internal links can lift a page out of the supplemental (as Michael Martinez claimed a while ago). I totally agree with that one.
Funny thing is.. look at some of the www.stumbleupon.com profile pages. Take a url which has a lot of reviews and check the reviewers. Some are in supplemental some are not. There is no rhyme or reason (some are active some are not, some have many fans = inbound internal links; some do not etc) Very odd.
As also from our own experiences where we have linked from 2 pages: home page and a page directly linked from the homepage (with low PR.. still toolbar PR.. is not worth anything anymore anyway.. and besides.. it would have the same impact on all links). OK now here is the weird thing: some pages consistently rank quite well and others are in/out of the ranking which goes parallel to in/out of the index altogether.. or in/out of the supplemental. No more external links.. nothing changes.. just time.
Now here's another funny one: some of them have a subdomain and others have a subdirectory. It SEEMS.. that the ones with a subdomain go in/out of the index altogether.. but if they are IN.. they rank in the top 5 on their main keyword (name of a participant). The subdirectory go in/out supplemental.. and even if they are out of supplemental they rank in the 80-100 range.. far from the top 5.
However others, have consistently ranked the same way. They are not older, not linked more/less.. nothing.. all grouped together as a list of 15-20 links.
Supplementals DO RANK high for niche keyphrases in fact.. just when things become more competitive they are almost invisble.
A final comment: it seems that when the site as a whole gets more "trust" (whatever that may be exactly) allocated this also impacts SOME supplemental pages... irrespective of the fact that they got more/less inbound (external) links. In other words: more trust also impacts all pages on the site as a whole.
This is just OUR experience.. with OUR growth pattern.
What code you use for 301 redirections?
In PHP, this is the standard code for a 301 redirect...
header("HTTP/1.1 301 Moved Permanently");
header("Location: http://www.domain.com/");
exit();
Although it wasn't stated explicitly in your entry, I bet the no-slash version of your pages were being treated as duplicated content.
Since Google does treat the slash and no-slash versions as different locations (who knows why), I think it's likely they see both pages as having the exact same content, and the second one being indexed gets a lower score and supplemental index placement.
As you mention though, so long as you are consistent with the methodology of ensuring all of your links use either the slash or no-slash internally you should be good on internal links.
Secondarily I have always chosen to use .htaccess files for 301 redirects. Most of my sites are written in PHP though. Is there a reason for choosing to use the header() tag inside the page instead of using a .htaccess file?
Thanks for the post Harvey!! So links to your sites without a trailing slash are useless? or if you have a 301 redirect on your website does this fix things up?
Cheers
I have the same kind of problem. http://www.qualitycodes.com has a page rank of 2 and http://www.qualitycodes.com/index.php has a page rank 1. http://www.qualitycodes.com/index.php is only used in internal links
I know this is very automated sounding to say, but I actually really love the name ragepank. I cant stand when people leave automated posts that say "hey great article", but I am not even righting this comment to get a link, I just want to let you know the name rocks. CHeers!
Any word on the use of slashes *within* keyphrases?
example: "computer/network consultant"
Would this come up in a search for both:
"computer consultant"
"network consultant"
?
Don't feel bad. Almost nobody really gets this issue. You read about a "Duplicate Content Penalty". But Google can even sort our multiple versions of links within the same site - Joomla is very bad at creating them.
This is from a Feb 2009 Speech given by Matt Cutts about the new Canonical Link Element (Which should help). But right in the speech Cutts says Google may not always use it. Nice!
Each of these URL's are viewed as unique:
* www.example.com
* example.com
* www.example.com/
* example.com/
* www.example.com/index.html
* www.example.com/Home.aspx
* example.com/Home.aspx
I've repooted here "http://www.internetmarketingiq.com/tutorials/83-canonical-link-element-avoiding-duplicat-content.html"
This is why I choose to use an .html for my articles in CMS platforms, so people will link to a consistent page.
Thanks alot for the post. I was just checking my backlinks in seopro(dot)com.au and didn't use the backslash. It showed 0 links! I added the backslash and it then showed 22.
Then in yahoo it showed 23(without the backslash)!! What the #$@#$??
Now what I really want to know is:
1.Should I seo for both with and without the backslash?
2 How important is the www. as well does it matter if I use it or not?? Any help would be appreciated.
Much thanks for this post, it's exactly what I was looking for!
Pretty good information on the trailing slashes. Most of the SEO's or Webdevelopers know that Google will consider them as seperate pages.
Martin Kelley - Feb 24, 2007
I wish I could see a date on this post, but I'm assuming that it's recent considering you're talking about Google's Supplemental results. I've been banging my head on the wall the last few weeks figuring out why a particular page didn't show up in Google's top-100 for it term. It's a real page on a semi-obscure topic and my subjective opinion is that it's better than all but the top five Google links. The site itself has okay PR (5) considering its topic and gets lots of Google hits, just not for certain pages.
Today I went into Google Webmaster Tools looking for more clues and stumbled across something that amazes me in the "internal links" description. The site has a static sidebar on every page that has the top dozen links (including the one not hitting). When the sidebar's link has a trailing slash the Internal Links count is in the mid-40s, accounting for most pages on the site. When the sidebar's link omits that trailing slash the Internal Links count is ONE.
Could this be why certain pages aren't showing. I've tested by searching on key phrases from various pages and it seems like those pages linked in the sidebar with trailing slashes are in the top-ten list for their terms while those without are often not even in the top-100. Amazing!
Needless to say there are trailing slashes everywhere. We'll see where my test phrases are in a week or two. I'm right now retrofitting my other sites with trailing slashes too.