A great deal has been written about how to get visitors to your web site (including some of the papers on this site). But once you get them to your site – how do you show them what they want? For a small site, the answer is simple: good navigation. Show the visitors clearly phrased, intelligently placed links, in a clean directory structure, and they'll figure it out.

However once you get past a few hundred subjects or products, a web site needs an internal search engine.

There are two basic types of search engines:

Algorithm based. You dump all your data into a database, write the mathematical logic for searching, convert the logic to computer code, and voila! Problem: the dirty little secret of algorithm-based search is that it doesn't work all that well. Why? Because it's an attempt at artificial intelligence (AI), and artificial intelligence is still 20 years (or more) from perfection.

Hierarchical. Humans catalog your information, services, and products, and then you organize it in neat database tables. Problem: it's incredibly labor-intensive – and unless your catalogers are really well tuned to your visitors' mindset, it doesn't work any better than algorithms. (Cataloging a book site by the Library of Congress cataloging system is a fast-track to failure.)

At Hastings we combine both methods. Here's our process:

1. Set up a purely algorithm-based search engine on your site, writing the logic according to our best guesses about your vistors and their wants.

2. Wait until we have some web site logs accumulated, showing the clickstreams of frustrated visitors as they wander around lost, plus a stack of email complaints from annoyed visitors.

3. Readjust the search logic and the searchable sections of your database. (If we're working off well-established company databases, we add new tables, rather than altering the basic structure. No reason to drive your systems administrators crazy.)

4. Add a thesaurus. This is a separate table (or fields) where your personnel can add keywords that aren't in your database. For example: your company sells 5 varieties of Gamay wine. But the web site clickstreams show that 400 visitors a day are searching for "beaujolais", and then leaving the site when they can't find any. Since most of those visitors would be happy with Gamay, that being from the same grape, your database manager can add "beaujolais=gamay" to the thesaurus, and all 400 visitors will be directed to your Gamay wines.

5. Set up a simple log analyzer that allows your marketing people to keep an eye on search trends.

6. Teach them how to make changes to the thesaurus.

Some of what we do to accomplish this may look like rocket science*, but the reason for doing it is very simple: a retail store that wants to make a profit shows the customers their full line of merchandise – and does its best to put that merchandise in front of the customers' noses. Web site search needs to do the same thing.

If you want to see the rocket science, here's an introductory paper by one of our principals: http://www.hastingsresearch.com/net/02-dkr-ir-intro.shtml