Does a spider follow links on a 404 page?

I have been reading an article by Lisa Barone (bruceclay.com) on nice 404 pages. This interested me as I have only just got around to improving my own 404 page.

I agree with the concept of the article - a nice 404 page is your last chance to save a lost visitor. Nothing is less attractive than a default browser page saying the page does not exist.

Lisa suggests the following items are the most important...
  • An apology for the error (even if it user error)
  • A prominent search box
  • A link to your site map
  • A link to your home page
  • Links to the other main areas of your site


I'd agree with all of that, even though I'm missing some of these. Some other things I consider important are...
  • Some branding - at least a logo
  • A minimum of images - the last thing you want is something on your 404 page raising a 404 error
  • Some basic styling - Times new roman font with blue / purple links screams of laziness
  • The words 404 prominently on the page. Geeks like to know when they have hit a 404.


It should also be said that a 301 should be used instead of a 404 when a page has moved. Users don't care or need to know about a page moving if they can be redirected to the new location automatically.

Some interesting points

A couple of things in the article bothered me. Firstly, Lisa suggests that search engines follow links on 404 pages.
If you leave the spiders to find a default 404 page you're throwing a roadblock in front of them that they have no way to get over. Search engines can't hit the back button or use the other advanced features of your Web site. All they can do is follow links. If they come across a bad link and you don't give them anywhere else to go, they'll leave your site.

Hmm. I haven't read the HTTP spec in that much detail, but I never considered a 404 page as something a spider would follow. I assumed the spider read the 404 header and ignored the body content (which is for the benefit of the user).
I'd wholeheartedly suggest including lots of relevant links on a 404 page, but this is for the user, not the search engine.

Custom 404 pages are an important part of SEO, and keeping your visitors happy.

Something I also find funny is 404 pages that don't return a 404 response. Google ends up indexing your 404 error, because it looks like a normal page! Surely not good form.
Digg StumbleUpon del.icio.us technorati blinklist furl reddit sphinn

Tags: usability404404 error404 "404 error" usability