---
title: "[Google Lecture] Finding needles in a 20 TB haystack, 200 million times per day"
url: "https://osherove.com/blog/2004/5/9/google-lecture-finding-needles-in-a-20-tb-haystack-200-milli.md"
canonical: "https://osherove.com/blog/2004/5/9/google-lecture-finding-needles-in-a-20-tb-haystack-200-milli.html"
published: "2004-05-09T18:45:00"
updated: "2004-05-09T18:45:00"
tags: ["Cool Articles"]
author: "Roy Osherove"
description: "Finding needles in a 20 TB haystack, 200 million times per day. Dr Eran Gabber, Google Tel Aviv University, Tel Aviv Monday - 17/5/2004, 12:00 - 14:00, Schreiber - 309 Abstract: Google faces two large technical…"
usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)."
license: "https://creativecommons.org/licenses/by/4.0/"
---

# \[Google Lecture\] Finding needles in a 20 TB haystack, 200 million times per day

By Roy Osherove May 9, 2004 In [Cool Articles](https://osherove.com/blog/category/cool-articles)

**Historical article** · Originally published May 9, 2004; technical details may be outdated.

Finding needles in a 20 TB haystack, 200 million times per day.

                           Dr Eran Gabber, Google

Tel Aviv University, Tel Aviv

Monday - 17/5/2004, 12:00 - 14:00, Schreiber - 309

Abstract:

Google faces two large technical challenges: ensuring that our search results are as relevant as possible, and serving hundreds of millions of queries in a fraction of a second at a reasonable cost. To solve the first problem, we perform an offline matrix computation to produce PageRank, a query independent measure of page reputation, and combine it with more traditional query-specific scoring. To solve the distributed computing problem, we use tens of thousands of commodity PCs and highly fault-tolerant software. I will discuss some details of these solutions, and also share some interesting statistical tidbits about search and the web.

Now **this** looks promising!

## More in Cool Articles

[All posts in Cool Articles](https://osherove.com/blog/category/cool-articles)

- Jun 4, 2004

    ### [Wintellect Web-exclusive articles](https://osherove.com/blog/2004/6/4/wintellect-web-exclusive-articles.html)

- May 20, 2004

    ### [Missed the Google lecture, but here's a short summary](https://osherove.com/blog/2004/5/20/missed-the-google-lecture-but-heres-a-short-summary.html)

- Apr 24, 2004

    ### [Good article about ReaderWriterLock and Multi Threading](https://osherove.com/blog/2004/4/24/good-article-about-readerwriterlock-and-multi-threading.html)

- Apr 13, 2004

    ### [Code DOM articles galore](https://osherove.com/blog/2004/4/13/code-dom-articles-galore.html)


---

## About the author

**Roy Osherove**: Senior Solutions Architect at AWS working on AI for production and AI-driven development; author of The Art of Unit Testing and Elastic Leadership, with about 20 years of consulting and training on unit testing, TDD, team leadership and continuous delivery.

### Sites

- [RobotPaper](https://robotpaper.ai/): This blog: AI patterns and anti-patterns, AI and coding agents, and engineering leadership.
- [osherove.com](https://osherove.com): Roy's personal site. Author of The Art of Unit Testing and Elastic Leadership; consulting and training on unit testing, TDD, team leadership and continuous delivery, plus about 20 years of blog posts.
- [The Art of Unit Testing](https://www.artofunittesting.com): The Art of Unit Testing, 3rd Edition (Manning): readable, maintainable, trustworthy unit tests, plus TDD workshops in C#, JavaScript, Java and Kotlin.
- [5whys.com](https://www.5whys.com): Roy's writing on software team leadership: self-organizing teams, surviving chaos, learning mode and the Elastic Leadership model.
- [Pipeline Driven](https://pipelinedriven.org): Continuous delivery and DevOps practices: build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps.
- [Elastic Leadership](https://www.elasticleadership.com): Elastic Leadership, Roy's framework for software team leaders (survival, learning and self-organizing modes).

### Social

- X: https://x.com/royosherove
- LinkedIn: https://www.linkedin.com/in/osherove/

### Contact

- Contact form: https://forms.gle/jQSx5KoXvrVA1JCK7

### License

Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page). License: https://creativecommons.org/licenses/by/4.0/