Lately, I’ve been reading and testing out some of the most discussed real time search engines and for all those of you who wish to stay up-to-date with the latest hot topics, shared links or just what’s being said about your company or other subjects, I think these can be helpful.

So what are real time search engines?

Well basically, imagine a search engine such as Google or Bing that only provides results that come from social media networks. These networks are filled with content that is constantly being updated, thus providing search results that vary as new posts appear. Unlike conventional search engines that provide relevancy-based results that are pretty much always the same, you get information on what is being said about your searched keyword in real time.

Currently, in the race for the “Best real time search engine” title, most of the engines available concentrate on Twitter posts and the difference between each one lies in the organization and filtering the overwhelming data that is being shared on this network.

Here is a couple of the ones I’ve given a look at:

Topsy concentrates on displaying the most shared links about a given subject. It indicates how many tweets the link has gotten and by clicking on the number of tweets displayed, you will be able to dig deeper and view the list of related tweets.

Collecta is simple and updates automatically as new posts appear, without having to launch your search over. Click on a result to read the whole thing and quick links enable you to share the post easily. However, the constant monitoring for your searched keyword can sometimes slow your browser down a bit.

CrowdEye is more complex with a screen filled with a lot of information. After launching a search, you not only get the latest tweets about your query, but also a whole bunch of other information related to your search, such as: a breakdown on the tweet volume per hour, the most popular links and related queries. This might be the most complete – or most complex for some users- real time search engine for the twitterverse today.

Wowd provides results that update live, as topics become more popular on the entire Web rather than in Twitter posts. It does not rely on crawlers or page rankings, but rather depends on users installing a browser app on their PC. As users browse the Web with the app installed, the visited pages are added to the public pages that appear in the Wowd results. As more and more people visit the same pages, they become more popular and get a better ranking in the Wowd results.

Of course, as the buzz around real time search engines grow, there are more and more engines available. Twitter offers their version that pretty much does the trick if you don’t need any other fancy feature than just getting the tweets as they get posted. Yahoo might also be teaming up with OneRiot in order to offer their version of real time searching. One thing’s for sure, real time search engines are getting a lot of attention. Even Google and Bing have announced the implementation of Twitter search in their engines.

Will it become the new way of doing online search? Will crawler-based search engines lose popularity to this new type of hype-based searching? Is popular information better than relevancy-based information?

What’s your opinion?

Normally, by using a search engine, you expect all the relevant content to be shown to you quickly. But as the web gets bigger and there is an ever-growing number of web pages, search engines cannot index them all. One search engine can index and search a great deal of pages, but why not combine the power of all search engines and search an even greater number of pages? Well, we can with meta-searching.

Meta-search engines are powerful tools that can drastically increase the speed and effectiveness of your internet searches. Unlike stand-alone search engines, meta-search engines do not have their own index for pages. They search other search engines and aggregate the results into a single list. Each Meta-search engine has its own relevancy algorithm to classify web pages after pulling them from the other engines. Some keep the result lists separated while others merge them, but all of them will provide you with much more content.

Meta-searches are very valuable for quick and broad researches, but if you want a more specific search, opting for a specialized search engine (such as http://www.wine-searcher.com/ for wine searches or http://us.imdb.com/ for movie searches) may be the best choice. If you do not, you can use the meta-search to identify other keywords relating to the subject you are interested in. This will help you specify your search later on.

Although meta-searches have a lot of advantages, they also have their disadvantages. They may collect data from much more sources, but the results may not be as interesting. Since they do not have direct access to the stand-alone engine’s database, they do not know how the pages were ranked originally. Therefore the results are said to be less relevant. Another disadvantage of the meta-search would be the unavailability to use search syntax. Although some meta-search engines give you the opportunity to use advanced syntax (the “AND” keyword or double quotes for example), most cannot use the other search engine’s full power effectively.

Therefore, when searching the web, you should consider what type of search would better serve your needs. Here are a couple of Meta-Search engines you could try:

Software:

• Copernic Agent (http://www.copernic.com/en/products/agent/index.html)
• Orbiscope Metasearch (http://www.orbiscope.net/en/search.html)
• Practisearch (http://www.practisearch.com/)

Web based:

• http://www.ixquick.com/
• http://www.dogpile.com
• http://www.metacrawler.com
• http://www.mamma.com

Since the video community on internet is growing at an incredible speed, it’s normal to consider the creation of a search engine dedicated to videos that could be able to search every video on the internet to find the one that will best fit with your search. Here is how we like to think a video search engine should work.

First, when you go to a search engine, you land on a normal, Google-looking web site and you are prompted to enter keywords and click on search.

This is where the search could differ from a simple search engine. The engine should first locate video files. It could do this by going through an index and keep only the file types that represent videos (avi, mov or wmv for example.) But when files are embedded into web pages, it would have to search through the web page and get to the file.

The easy way to locate videos relevant to your query is to search the description and title of the page that contains the video. This is the way Google and YouTube work. Even though it is useful, it does not work as well with videos as it does with text.

But our video specific search engine would not stop there. It could delve deeper in the information contained on the page. It could first parse the spoken word just as plain audio search would do, looking for your keywords using speech recognition. It would then use another tool to go through the written information in the video because most are fingerprinted with information pages.

Then, after deciding if the video is relevant or not to your search, the search engine could retrieve details such as duration, frame rate, bit rate, audio channels, etc. by going through the video’s syndication format.

After all these operations, the search would be much more relevant to your keywords than a simple Google-like operation. You could get the information you need in a movie format quickly and effectively.

There is currently only one web service that tries to do something similar to what I have described. According to their web site, Blinkx (www.blinkx.com) uses search strategies resembling the ones mentioned above and it even gives you a preview right on the search page.

Would one be willing to use super specific search engines like the one described above for each type of search (video, pictures, etc.)? Maybe engines less specific or accurate that offer the possibility of searching all document types from one location are more popular because of the ease of access…Are good results and relevant content not better than ease of use? What do you think?

Right now, Google is the most popular site on the internet (based on alexa.com ratings) and provides solutions to questions for thousands of users every day. Google can now find pages related to what you are looking for in under a second, but how did search engines technology get to where it is now?

It seems that the first tool designed to search the web was created in 1990 by Alan Emtage, a student at McGill University in Montreal. He named his creation Archie. (standing for “archive” without the “v”) The program accumulated directory listings of all the files located on public anonymous FTP sites and, therefore, creating a searchable database of file names.

This led to two new search programs, named Veronica and Jughead, which used Gopher’s index system to obtain information on specific servers or on menu titles from Gopher listings.

In June 1993, the first web robot (or crawler) appeared. Matthew Gray created an index called “Wandex” by using a Perl-based World Wide Web Wanderer. The purpose of Wandex was simply to measure the size of the World Wide Web, but JumpStation was released in December 1993 and used a web form to give access to all the information in its index, but was limited to searching titles and headings due to limited resources.

The first major commercial WebCrawler was Lycos in 1994. Soon after, many search engines started to appear because of the popularity and usefulness of web searches. Amongst the most popular search engines of that time, you could find Northern Light, AltaVista and Yahoo! During the late 1990’s, many other search engines joined the market, but few rose to success.

Around 2000, Google’s search engine arose in popularity due to its superior search results. The reason Google was able to provide better results was the use of PageRank, a great innovation in the search engine sector. PageRang used iterative algorithms to rank web pages based on the number and PageRank of other web sites and pages linking there.

Years later, even though other competitors tried making their place in the search market, Google is still the dominant figure in web search. A lot of people have even started using the brand’s name as a synonym for web search (see googling).

Now, semantic search engines are starting to emerge, giving us the ability to further refine our searches by disambiguating queries in order to produce even more relevant searches. The first major search engine using this kind of technology is Bing, Microsoft Live Search’s successor. The battle is now on for global search domination, who will rise above, semantics or keyword search?