Anatomy of a search engine

cowphysicistInternet and Web Development

Dec 4, 2013 (3 years and 9 months ago)

122 views

Anatomy of a search engine


Design criteria of a search engine


Architecture


Data structures

Step
-
1: Crawling the web


Google

has

a

fast

distributed

crawling

system


Each

crawler

keeps

roughly

300

connection

open

at

once


Google

can

crawl

over

100

web

pages

per

second

using

four

crawlers

at

peak

speeds

(roughly

600
K

per

second

of

data)


Each

crawler

maintains

a

its

own

DNS

cache


The

crawler

uses

asynchronous

IO

and

a

number

of

queues

Ref: http://www.elibrary.icrisat.org/Google%20Search/how%20google%20works.htm

Step2: Indexing the Web (1)


Parsing


Any

parser

which

is

designed

to

run

on

the

entire

Web

must

handle

a

huge

array

of

possible

errors
.

These

range

from

typos

in

HTML

tags

to

kilobytes

of

zeros

in

the

middle

of

a

tag,

non
-
ASCII

characters,

HTML

tags

nested

hundreds

deep,

and

a

great

variety

of

other

errors

that

challenge

anyone's

imagination

to

come

up

with

equally

creative

ones
.


Use

flex

to

generate

a

lexical

analyzer

for

maximum

speed

URL Server

Store Server

Crawler

Crawler

Crawler

Repository

Indexer

Barrel
s

Indexer

Indexer

Sorter

Step2: Indexing the Web (2)

Indexing Documents into Barrels



After each document is parsed, it is encoded into a number of barrels
.



Every word is converted into a wordID by using an in
-
memory hash table
--

the lexicon
.



New additions to the lexicon hash table are logged to a file
.



The words in the current document are translated into hit lists
.



The words are written into the forward barrels
.



For parallelization, indexer writes a log to a file, instead of sharing the lexicon

Sorting



Takes each of the forward barrels
.



Sorts it by wordI
D to produce an inverted barrel
.



Parallelize the sorting phase
.



Subdivides the barrels into baskets to load into main memory because the barrels don’t fit into
memory
.



Sorts baskets and writes its contents into the inverted barrel

Searching (pseudo code)

1.
Parse

the

query

2.
Convert

words

into

wordIDs

3.
Seek

to

the

start

of

the

doclist

in

the

short

barrel

for

every

word

4.
Scan

through

the

doclists

until

there

is

a

document

that

matches

all

the

search

terms

5.
Compute

the

rank

of

that

document

for

the

query

6.
If

we

are

in

the

short

barrels

and

at

the

end

of

any

doclist,

seek

to

the

start

of

the

doclist

in

the

full

barrel

for

every

word

and

go

to

step

4

7.
If

we

are

not

at

the

end

of

any

doclist

go

to

step

4

8.
Sort

the

documents

that

have

matched

by

rank

and

return

the

top

k


Figure 4. Google Query Evaluation

Searching

The Ranking System



Every hit
-
list
includes

position, font and capitalization information
.



Factor in hits from anchor text and the PageRank of the document
.



Ranking function so that no particular factor can have too much influence




For a single word search

o

In order to rank a document, Google looks at that document’s hit list for a single word
query and computes an IR score combined with PageRank



For a multi
-
word search

o

Hits occurring close together in a document are weighted higher than hits occurring
far ap
art

Use of
Feedback



Google has a user
feedback mechanism because figuring out the right values for many
parameters is very difficult
.



When the ranking function is modified, this mechanism gives developers some idea of how
a change in the ranking function affects
the search results

Page Rank

Page Rank
:

Page Rank of any
node

shows: “how many important or popular nodes vote for the
given node”. This is a collective intelligence of nodes in graph
.


For given vertex/node
'
i
V
, let


'
i
V
IN
be the set of vertices/nodes that point to it
(predecessors), and let


'
i
V
OUT
be the s
et of vertices that vertex
'
i
V
points to (successors).
The score of vertex
'
i
V
can be defined as “Page et al. (1998)”:

















'
'
'
1
'
i
V
IN
j
j
j
i
V
OUT
V
S
N
V
S



(1)

Where:




'
i
V
S
Rank of vertex
i
V
.



'
j
V
S
=rank of vertex
'
j
V
, from which incoming link comes to word / vertex
'
i
V
.


N
Count of number of words/vertex in word graph of sentences.



damping factor ( “0.85” is used in “Page et al. (1998)”).

Google architecture


URL

server

sends

list

of

URLs

to

be

fetched

to

crawlers


StoreServer

compresses

and

stores

pages


Indexer

extracts

words,

their

pos
.
,

size,

capital
.


Anchors

cont
.
links

and

their

text


Sorter

generates



inverted

index


Searcher

uses

Lexicon,

II,

and

PR


Major data Structure

Major Data Structures
:

Google's data structures are
optimized so that a large document
collection can be crawled, indexed, and searched with little cost
. Google is designed to avoid disk
seeks whenever possible, and this has had a considerable influence on the design of the data

structures.

BigFiles
:
BigFiles are
virtual files spanning multiple file systems and are addressable by 64 bit
integers
.

Repository:

The repository contains the
full HTML of every web page
. Each page is
compressed
using zlib
. The repository requires
no other data structures to be used in order to access it. This
helps with data consistency and makes development much easier;

Document
Index:

The document index keeps
information about each document
. It is a
fixed
width ISAM (Index sequential access
mode) index, ordered by docID. The information stored in
each entry includes the current document status, a pointer into the repository, a document
checksum, and various statistics
.



Additionally,
there is a file which is used to convert URLs into docIDs
. I
t is a list of URL
checksums with their corresponding docIDs and is sorted by checksum. In order to find the docID
of a particular URL, the URL's checksum is computed and a binary search is performed on the
checksums file to find its docID.

Major data Structure

Lexicon:

The current lexicon
contains 14 million words

(though some rare words were not
added to the lexicon). It is
implemented in two parts
--

a list of the words (concatenated
together but separated by nulls) and a hash table of pointers
.

Hit Lists:

A hit list corresponds to a list of occurrences of a particular word in a particular
document including position, font, and capitaliz
ation information
. Hit lists account for most of
the space used in both the forward and the inverted indices.

Forward Index
:

The forward index is actually already partially sorted. It is stored in a number
of barrels (we used 64).
Each barrel holds a r
ange of wordID's. If a document contains words
that fall into a particular barrel, the docID is recorded into the barrel, followed by a list of
wordID's with hitlists which correspond to those words
.

Inverted Index:

The inverted index consists of the
same barrels as the forward index, except
that they have been processed by the sorter
.
For every valid wordID, the lexicon contains a
pointer into the barrel that wordID falls into.

It points to a doclist of docID's together with their
corresponding hit li
sts. This doclist represents all the occurrences of that word in all
documents.



Reference


[Cho

98
]

Junghoo

Cho,

Hector

Garcia
-
Molina,

Lawrence

Page
.

Efficient

Crawling

Through

URL

Ordering
.

Seventh

International

Web

Conference

(WWW

98
)
.

Brisbane,

Australia,

April

14
-
18
,

1998
.


[Page

98
]

Lawrence

Page,

Sergey

Brin,

Rajeev

Motwani,

Terry

Winograd
.

The

PageRank

Citation

Ranking
:

Bringing

Order

to

the

Web
.


[Chakrabarti

98
]

S
.
Chakrabarti,

B
.
Dom,

D
.
Gibson,

J
.
Kleinberg,

P
.

Raghavan

and

S
.

Rajagopalan
.

Automatic

Resource

Compilation

by

Analyzing

Hyperlink

Structure

and

Associated

Text
.

Seventh

International

Web

Conference

(WWW

98
)
.

Brisbane,

Australia,

April

14
-
18
,

1998
.


[Gravano

94
]

Luis

Gravano,

Hector

Garcia
-
Molina,

and

A
.

Tomasic
.

The

Effectiveness

of

GlOSS

for

the

Text
-
Database

Discovery

Problem
.

Proc
.

of

the

1994

ACM

SIGMOD

International

Conference

On

Management

Of

Data,

1994
.


Sergey

Brin

and

Lawrence

Page
;

The

Anatomy

of

a

Large
-
Scale

Hypertextual

Web

Search

Engine
;

Computer

Science

Department,

Stanford

University,

Stanford,

CA

94305