Request Media Kit

Twitter Search Can Now Index 50 Times More Tweets Per Second

Twitter has changed its entire search architecture, and nobody noticed for weeks until they pointed it out. It must be successful right? It's supposed to be so powerful that it can index about 50 t...
Twitter Search Can Now Index 50 Times More Tweets Per Second
Written by Chris Crum
  • Twitter has changed its entire search architecture, and nobody noticed for weeks until they pointed it out. It must be successful right? It’s supposed to be so powerful that it can index about 50 times more tweets per second than Twitter is currently getting. That’s a lot. 

    Twitter is no longer using the technology it originally acquired from Summize, which became Twitter Search. Twitter’s Michael Busch explains the new technology:

    About 6 months ago, we decided to develop a new, modern search architecture that is based on a highly efficient inverted index instead of a relational database. Since we love Open Source here at Twitter we chose Lucene, a search engine library written in Java, as a starting point.

    Our demands on the new system are immense: With over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day (!) we already put a very high load on our machines. As we want the new system to last for several years, the goal was to support at least an order of magnitude more load.

    New Twitter Search Architecture (on the back end)

    Twitter says the new architecture will not only benefit the company, but users too. Users will notice a bigger index, which they say is twice as long, without making searches any slower. 

    Interestingly, Twitter also says the new tech will allow them to build "cool new features" faster and better. What kinds of features would you like to see Twitter add on to search? Share your thoughts with us. 

    Get the WebProNews newsletter delivered to your inbox

    Get the free daily newsletter read by decision makers

    Subscribe
    Advertise with Us

    Ready to get started?

    Get our media kit