Search Advances

So, Twitter seems to be getting serious about its real time search capabilities. According to various reports, all of which seemed to have emerged from this source, Twitter’s new VP of Operations, Santosh Jayaram, has said that Twitter Search will soon be doing two things in addition to what it does now

• it will crawl the links that people tweet
• it will sort results by its reputation ranking system

The ranking algorithm is going to be very interesting, because unlike, say, Google’s search algorithm, this would have to work at two levels – one, similar to Google’s Page Rank to ascertain the site quality, and two, the reputation of the person sharing the link. So, it’d be interesting to see which one would come on top, assuming the same story – me, sharing a TechCrunch link, or Mike Arrington sharing a link to this blog. 😉 Mashable had earlier written about an alternate  Twitter search service called Tweefind that uses various parameters to rank a person. The eternal debate about what should make a better twitter rank just got more interesting. 🙂

RWW has connected the above happening to an interesting change that happened at Twitter recently – Twitter replacing tinyurl with bit.ly as the default url shortening service. According to an earlier article on the same site, bit.ly does more than just shorten a url, it “analyzes the page being linked to, pulls out the key concepts discussed on that page, and then provides real-time statistics about where the link is being shared and how many people are clicking on it.” Now, isn’t that interesting??!! When talking about the crawling of links, its hard not to think about the various services I’d written about earlier, (Krumlr, Fleck etc) which work on a delicious+twitter principle – use the delicious method of tagging and then share to twitter. I wonder, if at some stage, this is the kind of semantic association that Twitter would want to build on top of the crawling spiders, or will the machines take care of this too?

The impact of all this on Google remains to be seen. Google is also looking over its shoulder to another hyped up participant in the ring – Wolfram Alpha, which is yet to make a debut. But there are speculations that they are on top of that situation. Anyway, Google must be doing something, they always do, that’s what makes them so dangerous. Since it already indexes tweets, adding real time shouldn’t be a big deal. A greasemonkey script does that for me!! But with the addition of Search inside GMail, the possibilities of that + Google Profiles + Friend Connect (and Gtalk status sharing) in creating a human layer  on top of the existing search is interesting. Their Searchology event has brought out a lot of new stuff  –

• Search Options – a collection of tools that help you wean out the information you are really looking for, and view it in the way you want to. Essentially you can now tweak Google Search some more to your preferences.
• Wonder wheel – it clusters search information
• Rich Snippets – In addition to the info that currently gets displayed in a search item, there will be a line that sums up the result – eg. ratings for restaurants. It has asked publishers for their cooperation in adopting microformats to create this structured data.
• Google Squared – As per the post, it “doesn’t find webpages about your topic — instead, it automatically fetches and organizes facts from across the Internet.” Its description does remind me of a certain yet to be launched search engine 🙂
• Search will also indicate whether a site is optimised for mobile devices, and will consider location when delivering search results. (Google Suggest bringing in results from local places for say, restaurants)

Some excellent live coverage happened at Search Engine Land.

Meanwhile, a small detour for Microsoft and Facebook. Microsoft claimed recently that its going to become “more disruptive in search.” Facebook recently opened its stream API but also cut off the RSS feed for the updates. I used to make use of it in at least a couple of places. 🙁 It also acknowledged Indian users by making itself available in 6 Indian languages. I wonder where Facebook figures in these search battles. Does the opening of the stream API mean that we will soon have a real time status search mechanism? But how useful will that be when a lot of users prefer to keep their profiles walled (like FB itself)? But its interesting to note that many geeks also auto update their FB statuses with their twitter ones thanks to many available services. FB is quite an aggregator too, in its own way, so I wonder if we’ll get to see a search that shows Twitter + FB statuses, and the videos, pages, shared links and comments content on FB. Meanwhile, on real time, alerts now happen as pop ups. 😐

The last couple of days also saw new versions of a couple of existing players – One Riot now indexes and groups link shares on Twitter and Digg. It also allows you to dig further into the data- numbers, who shared it first etc  and then share it on the two services. Tweetmeme is launching an enhanced search version which lets you filter results by age, category, channel and also shows how many times result has been tweeted.

To me, real time is only one of the things that makes Twitter’s foray into search interesting. After all, when I search for real time links to a story on Twitter, I don’t think an Ad Sense like mechanism will work for revenue. So it is the combination of semantics, sentiment analysis, and real-time data that makes this Twitter development seem like a huge leap (when it happens). Google seems to be working on making more sense of data, than real time, or semantics. Can that be taken as actually walking the talk when they claim that search is still in its infancy and there’s a lot of room for existing and new players? Twitter and the new services don’t have the scale of indexed pages that Google has, and Google doesn’t have real time. For now, its interesting how all of these services actually work out complementing each other, as shown by the comparison here.

I have to admit, with all the connecting that was happening on Twitter, I was hoping that a revolutionary model (of revenue and web behaviour) would evolve. The current developments, though a lot of it is still conjecture, are not as over whelming as I’d hoped for. Its an organic evolution of sorts – semantic, real time, social web. Perhaps it is only the beginning.

until next time, the search is on…