Does a spider follow links on a 404 page?

<< The CAPTCHA to end all CAPTCHAs Renaming pages = lost links and traffic >>

Apr 20, 2007

I have been reading an article by Lisa Barone (bruceclay.com) on nice 404 pages. This interested me as I have only just got around to improving my own 404 page.

I agree with the concept of the article - a nice 404 page is your last chance to save a lost visitor. Nothing is less attractive than a default browser page saying the page does not exist.

Lisa suggests the following items are the most important...

An apology for the error (even if it user error)
A prominent search box
A link to your site map
A link to your home page
Links to the other main areas of your site

I'd agree with all of that, even though I'm missing some of these. Some other things I consider important are...

Some branding - at least a logo
A minimum of images - the last thing you want is something on your 404 page raising a 404 error
Some basic styling - Times new roman font with blue / purple links screams of laziness
The words 404 prominently on the page. Geeks like to know when they have hit a 404.

It should also be said that a 301 should be used instead of a 404 when a page has moved. Users don't care or need to know about a page moving if they can be redirected to the new location automatically.

Some interesting points

A couple of things in the article bothered me. Firstly, Lisa suggests that search engines follow links on 404 pages.

If you leave the spiders to find a default 404 page you're throwing a roadblock in front of them that they have no way to get over. Search engines can't hit the back button or use the other advanced features of your Web site. All they can do is follow links. If they come across a bad link and you don't give them anywhere else to go, they'll leave your site.

Hmm. I haven't read the HTTP spec in that much detail, but I never considered a 404 page as something a spider would follow. I assumed the spider read the 404 header and ignored the body content (which is for the benefit of the user).
I'd wholeheartedly suggest including lots of relevant links on a 404 page, but this is for the user, not the search engine.

Custom 404 pages are an important part of SEO, and keeping your visitors happy.

Something I also find funny is 404 pages that don't return a 404 response. Google ends up indexing your 404 error, because it looks like a normal page! Surely not good form.

<< SEO Articles index < The CAPTCHA to end all CAPTCHAs | Renaming pages = lost links and traffic >

Tags: usability 404 404 error 404 "404 error" usability

Comments

Lynn Bishop - Jun 8, 2007

We had a bit of fun with our 404 page, you might be interested in taking a look.

http://www.canzdesign.com/contnt

Regards Lynn from www.canzdesign.com

Anthony - Aug 30, 2007

Lynn, that page is a bit wordy IMHO. I say keep it simple to help your users find what they are looking for.

-OldSchool

Robert - Sep 27, 2007

I agree with your statement that the custom error page is for the user and most likely has no bearing on how the search engines index your site. For one thing - you need to add a "nofollow" in the meta tag section to prevent the spider from indexing the error page - as far as i have read, once a spider hits a "nofollow" it doesn't go any further beyond that page.

here is an example of AboutNewJersey.com's error page - http://www.AboutNewJersey.com/errortst.htm

Daniel Murray - Apr 4, 2008

Good things to do with a 404 page:

- Redirect it to your homepage.
- Redirect it to a product page.
- Redirect to your sitemap.
- Have a search bar on it.
- Have a link on it to a few main pages so if the search engines end up there, they're not left at a dead-end.

Dan ;-)

Goran Candrlic - Aug 22, 2008

- logo
- excuse for not having that content
- homepage link
- having questions: email us

nothing more nothing less

otherwise users might think that this is a regular page.

considering crawling... no need... website should be crawled from other sources

onsite sitemap and xml sitemap should and must provide enough for spiders