[System Design Tech Case Study Pulse #8] Processing 2 Billion Daily Queries Using Unicorn and Apache Thrift : Facebook's Graph Search Secret Sauce

With detailed explanation and flow chart....

Oct 04, 2024

∙ Paid

Hi All,

Facebook's Graph Search is capable of processing 2 billion daily queries using Unicorn, a custom-built inverted index and retrieval system, and Apache Thrift for efficient inter-service communication. This sophisticated system forms the backbone of Facebook's ability to provide real-time, personalized search results across its vast social graph.

Let me dive deep into how this system works, exploring the key components, technologies, and processes that enable such massive-scale, low-latency graph search capabilities.

Learn how to Design Facebook Newsfeed

[System Design Tech Case Study Pulse #2] How Lyft Handles 2x Traffic Spikes during Peak Hours with Auto scaling Infrastructure..

We will continue to add a growing amount of system design, projects and ML/AI content. Ignito ( this publication) urgently needs you and your support (else Ignito will shut down). If you like Ignito publication and my work please support with some ( even a small amount is good) help/donation : Link

System Overview

Before we delve into the search architecture, let us look at some key metrics of Facebook's Graph Search system:

- Daily search queries: 2 billion+
- Peak queries per second: 100,000+
- Indexed entities: Trillions (users, posts, pages, etc.)
- Edge types in the graph: 100,000+
- Average query latency: < 100ms
- Unicorn servers: 1000+
- Data centers: 10+
- Daily index updates: Billions
- Query types supported: Keyword, structured, natural language
- Languages supported: 100+
- System availability: 99.99%
- Index size: Petabytes of data
- Thrift RPC calls per query: 50+ on average