Determining the Ideal Search Terms for Crawler-Based Search

by Stacy Williams

Jul 01, 2001

A good search engine optimization campaign targets all the different types of search engines. Today we'll cover true, crawler-based search engines such as AltaVista, Excite, Google, Hotbot and Lycos. True search engines are defined as using automated programs called crawlers (or spiders, or robots) that follow links from page to page across the web -- all day, every day. They index the content of every web page they come across into their search engine's gigantic database. This database is the pool of sites that the search engine looks through when you run a search.

Search engines also have complex algorithms that determine which site gets ranked first when you do a search, which site gets ranked twelve millionth, and what order the other sites fall into in between. These algorithms are proprietary (different for each engine) and top secret. There are probably only a handful of engineers at each search engine that truly know what it takes to be ranked #1. But there are a lot of commonalities as to what they look for that we marketers can work with.

The first step is also, by far, the most important. Unfortunately, it's also often rushed through, or treated as an afterthought. This critical step is determining the ideal search terms for your site. Search terms are the words and phrases that your target audience types into a search engine when looking for a site like yours. They are the terms that you'll target and incorporate into your site. Note that I've said "words and phrases" - you almost never want to use single-word terms. They're usually too broad and will pull up sometimes millions of other sites you've got to battle for position. Longer phrases of multiple words that are highly targeted work best.

There are three things to look for when determining your ideal search terms:

1. Frequency of usage: You want to use terms that people are actually running searches on. It doesn't do you any good to optimize a site for a term that no one is using. The question then becomes, how do you know what people are searching on? At this point in time, there are only two tools available. Both have a nominal fee associated with them, and neither one is perfect, but they're a lot better than using your best guess. I prefer to use GoTo.com's Search Suggestion Tool. You can type in a search term and see how many people ran searches using that term the previous month. You can also get suggestions for other terms that might be appropriate. GoTo.com's figures include those of its partner network, so they represent around 75% of web searches. Another tool is Wordtracker, an online tool that you can subscribe to from up to a day ($6) to a year ($199). They also have a free 30-day trial, though it doesn't show you nearly all of Wordtracker's capabilities.
2. Number of competitors: You want to use search terms that aren't terribly competitive. That is, all things being equal, if you've got one search term that pulls up 80,000 sites total, and another that pulls up 750,000 sites, you'll want to use the first term. The fewer sites you've got to battle for position, the better your chances for getting ranked high. Of course, you've got to balance frequency of usage with this. Often, terms that people are using in searches a lot tend to pull up a huge number of competitive sites. And terms that people hardly ever type into the engines pull up a smaller number of competitive sites. But usually you can find a handful of gems - search terms that people are using often but that don't pull up too many other sites.
3. Relevance: You've, obviously, got to make sure that your search terms are relevant to your site's content. Not only will this help drive more targeted traffic, but it'll make the process of incorporating the term into the site easier. It can be frustrating, awkward, and counterproductive to try to force a search term such as "Britney Spears" (high frequency of usage) into a page about a driving school for teens, for example.

I find that putting all the search terms into a spreadsheet, along with the frequency of usage and number of competitors (I use an average across several engines) allows me to analyze the data easily. It's fairly simple to compute an index of the two figures and quickly determine which terms are best, at least from a mathematical point of view. Then the task becomes factoring in relevance, which can only be done by hand.

If you want to optimize 20 pages on your site, you'll want to select 20 search terms to work with. It used to be that you could optimize your home page for 20 different terms, but that's no longer the case. The search engines are looking for "themes" - that is, they like to see the same search term used in multiple places on a single page. Next time, I'll list the nine places on a web page where the search engines look for search terms, and we'll also discuss design issues that can potentially sabotage your efforts (those crawlers are pretty limited in what they can read and index).

About the Author:

Stacy Williams is founder and president of Prominent Placement, Inc., which offers strategic search engine optimization to its clients. Prior to founding Prominent Placement, Stacy spent nine years at the progressive advertising agency Kilgannon McReynolds, Inc., most recently as Director of Marketing and Internet Marketing Guru.

With an extensive background in Internet marketing, Stacy offers her clients cutting edge solutions that are always rooted in sound marketing strategy.