SEO Information

Search Bots, Crawlers, and Spiders


If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/

MORE RESOURCES:
Unable to open RSS Feed $XMLfilename with error HTTP ERROR: 404, exiting 		RELATED ARTICLES
"Web Content Management System fr Window": Search Engine Typos
Oops! I meant "web content management system for windows." Do search engines understand consumer search engine typos? Typing something so close to what you are looking for, like typing "web content management system fr window" instead of "web content management system for windows" may not seem like a big deal, but search engine bloopers alter consumer searches more than we know.
Why is Search Engine Optimisation Expensive?
Many of our customers (http://www.internet-marketing-australia.
How MSN and Yahoo Sells Your Traffic
Yes, it really happens. Now you might find it hard to believe butyou will understand after I explain.
Crash Course in Getting a #1 Google Ranking
First, here's the rundown of some of the terminology I'm going to use -Inbound Links - Links coming into your siteOutbound Links - Links leaving your siteCross Links - Links that you have "traded" with another site (ie, they've got a link from their site to your and you've got a link from your site to theirs)PR (Page Ranking) - Google's measure of how "important" your site isSEO Is Not DeadOk, now lets talk about what you really want to hear - how to get those coveted 1-10 ranks for your keywords. Remember this - SEO is not dead.
2 Powerful Ways To Capitalize on Your Search Engine Traffic
Many marketers know that search engine marketing is among one ofthe best methods to get visitors to their websites. It's aprofitable way to reach new subscribers and new customers.
Link Popularity - Basic Overview
There are many techniques that SEM/SEO experts use to optimize Web sites. One of the more important techniques is "Link Popularity.
Get Personal With Google
Anyone who's been on the net any time at all knows that Google has been the search engine of choice for many for a while now. Whether that continues remains to be seen, but for now, with MSN still in development and Yahoo going through some growing pains, many still feel that Google is the leader.
The Search Engine Optimization Game Is Getting Tricky
For more than 4 years now I have been modifying my work from home based web sites trying to optimize for the major search engines, Especially Goggle. Today with much disappointment I have come to the realization that search engine traffic alone will not make me rich.
Copywriters: Make Friends with Search Engine Optimization
I don't know about you, but I felt a lot of internal resistance when it was first suggested to me that I write 'for' the search engines. Years of training and habit, and a lot of pride, turned me against the idea of adjusting or changing my copy to suit 'spiders'.
Soliciting Search Engines
As your guide operator through the web, search engines are invaluable when used effectively.You don't have to be able to create a search engine to use it, and their interface is designed with that in mind.
Increase Web Site Sales with a SEO Proposal - Part 1
You can easily get confused by all the search engine optimization companies and SEO experts that offer SEO services. It's hard to know who to trust or what should be included in a SEO proposal.
Search Engine Marketing 101: What Search Engines See When They Visit Your Web Site
If you have a Web site, have you ever wondered what a search engine sees when it visits your site to add the site to its index? Do you know that it doesn't see the beautiful graphics or the fancy Web design? Do you know that it only sees the source code, or the "skeleton" of your Web site?Do you realize that knowing this little tidbit of information and doing something about it can make a huge difference in your search engine rankings and, ultimately, the success of your online business?One very important thing that you need to remember is: the search engines like simplicity. The simpler your Web site is, the easier it is for the engine to determine what your Web site is about.
What In The World Is This Google Sandbox Theory Thing? And How Do I Beat It?
Ok, so over the past month or so I've been collecting various search engine optimization questions from all of you. Today, I'm going to answer what was the most frequently asked question over the past month.
A Real Example of Search Engine Optimization (SEO) Success
The term, Search Engine Optimization (SEO), refers to a set of techniques by which web sites and web pages are constructed for maximum recognition and ranking by search engines such as Google, Yahoo, and MSN Search. Using the right techniques can guarantee top listing positions for keywords and keyphrases that are related to a site.
Monitor and Increase Your Search Engine Visibility with the DIY SEO Tools
In this three part article, you'll find many tools that any webmaster can use to monitor your site's search engine position, and use to increase the visibility of your site in major search engines like Google, Yahoo and MSN.URL Trendshttp://www.
10 Ways To Indirectly Get To The Top Of Search
There are millions of web sites trying to get listed inthe top 20 spots of the major search engines. Thatamounts to a lot of competition! I say if you can'tget listed at the top, indirectly get to the top.
The Smart Way To Get Listed On Google!
So you want to get listed on Google? And you want a GOOD listing on Google? "But my website is not popular and is too new to be noticed by Google," you say?Well, I have a solution to your problem.There are couple ways to get listed on Google.
The SEO Game - Do You Play It?
Most people do not think of SEO as a game. Let me take 5 minutes of your time and explain how SEO is simply a game of Chess.
Google Slavery...Old Habits Die Hard
For the first few months after Yahoo decided to go their own way with natural search (and MSN decided to get serious about the search business), the search results provided by those two could only be described as bizarre. Enough time has now passed that the dust has somewhat settled and there are three main (from a traffic standpoint) sites for quality natural searches.
Beyond Search Engines
Some webmasters report that search engines account for 75% or more of their total website traffic. However, it's important not to become too dependent on search engines for new business.

home | site map | contact us