Search Engine Woes...

I have a custom designed search engine on my site. It was built using a lot of JavaScript and an offline database powered by some custom designed data management software. Lately, I've come to the realization that the overhead required for this undertaking is a bit more than I can manage. In trying to find an alternative to my own search engine, I've been looking into the feasibility of third-party site search services. By the way, the image for this post is the Internet Explorer mascot, Inori Aizawa.

Google: One search service that I looked at was Google. Google has a very good search form that can be easily modified to fit into my existing site with only a few minor changes. My existing search engine has a dedicated search page and the Google service has the same option. When adapting the Google service to my site, I decided to not link to the dedicated search page provided by Google. The Google-provided search page is customizable, but, not to the extent that it will look like the rest of the anime ratings site.

The Google search form links to a Google-supplied results page. The results page has more customization options, but, it still won't look like the rest of the anime ratings site. Google offers the option of sending results to a page not hosted by Google. Unfortunately, my web host doesn't allow such things (at least not for free or even cheap). I've tested the code and my web host stops the page from loading.

The interface issues aside, the only major drawback to the Google search service is that I can't control how the anime ratings site is indexed. Keywords that should return something don't seem to work (i.e. Bubblegum doesn't return Bubblegum Crisis). And if new content is added, I have no real way of knowing when or even if the new pages are accessible via the search form. In fact, I don't even know if the current pages are accessible via the search form.

Another related issue is that I really don't know what portion of my web pages Google's web crawler will focus on. Some of the test result pages do not meet my expectations. My search engine focuses on titles and, to a lesser extent, characters. The Google search results are more generic.

Yahoo: I tested a Yahoo search form and came up with very similar results as the Google search form. In the case of Yahoo, however, their site explorer utility allows me to know whether or not my pages are indexed.

But, like Google, keywords that I expect to return a given result do not return the results that I expect. Unlike the Google search form, the Yahoo search form can only be limited to one domain. Even so, the Yahoo filter often returns more results more often than the Google search form. Unfortunately, the Yahoo search results often include pages from other sites. The Yahoo site filter is domain-specific not directory-specific. And, because my site is a sub-directory of a larger domain, the results include other unrelated sites that share the same domain.

After several days of testing and tweaking, I'm sure that if I was willing to pay good money, the problems that I've experienced with third-party search services could be made to go away.

Update

Since the demise of the Verizon hosted site in September 2010, I've moved to newer accommodations here at Blogger and now I have an integrated search bar right on the page, so, all of my previous efforts with regard to a custom built search solution have been largely in vain.

And, below, Internet Explorer mascot, Inori Aizawa, a girl who lives in a post-apocalyptic wasteland that is the Internet, appears in a video... Enjoy.