10 Reasons Search Engines Might Not Index Your Webpage

by Simon on October 1, 2011

Websites become listed on search engine results if, and only if, the search engine spiders are indexing them. If they pass right by your pages, and don’t even know they exist, then you’re simply not going to get listed.

Do this simple check. Go to http://www.google.com and type in site:yourdomain.com. You should see a list of the pages from your web domain that Google has in their index. If there are no pages listed you have either made a mistake typing your domain or Google does not have ANY pages off your web domain in their index. If there are less pages indexed than you had expected here are some reasons why.

You May be Inadvertently Blocking Google

1. We regularly help customers who have either a robots.txt file saying disallow robots or some other block on the search engines.
2. Your pages are ‘hidden’ in an HTML frameset.

You Don’t Want Certain Pages Indexed

Some pages are just not meant to be indexed. This may be as you plan it however it’s useful to understand what spiders do and don’t index.  Search engine spiders don’t index:

1. Pages only accessible by using a search form i.e. there are no links to the page from other pages on your website
2. Pages that require a log in
3. Pages that require visitors to submit a form
4. Pages that redirect to a different URL before showing content

While some pages are not meant to be indexed, other pages are and might be missed. Pages that spiders often ignore include:

5. Pages with too many outgoing links – try and keep below 100 (see Google’s Webmaster Guidelines)
6. Pages with complex URLs – these often give spiders an error result
7. Pages which are more than three clicks from the home page, often described as “deep pages”

Other factors may prevent your web pages from being indexed by search engines. These factors include:

8. Broken links from your site
9. A webpage that exceeds 105K
10. A slow loading time or a down server.

Finally, if your page is a flash page, search engine spiders struggle to index it as it may have limited content they can scan. As you’re optimizing your website and specific web pages, pay attention to these factors. The goal, of course is to be indexed and to achieve first page search engine results.

Finally Google recommend submitting a Sitemap using Google Webmaster Tools. Google uses your Sitemap to learn about the structure of your site and to increase our coverage of your webpages.

Did you enjoy this post?

If you enjoyed this post, here's what you can do next:

1. Share this post with your contacts and friends.

2. Join 783 people who receive our posts via:

3. Click to scroll down and leave a comment and tell us what you think.

has written 57 posts on Oxford Digital Marketing.

Simon Wallace-Jones started his Digital Marketing work in the late 80's implementing solutions for companies like DEC, Sun and Tektronix. In 1996 he co-founded the pioneering company Relayware.com which developed one of the world's first web marketing automation solutions which is still in use today by leading brands like Sony, Lexmark and Lenovo. In 2008 he co-founded Oxford Digital Marketing to help smaller businesses take their use of Digital Marketing to a higher level.

{ 0 comments… add one now }

Leave a Comment

Previous post:

Next post: