Building a scalable WWW search engine ... NOT in Perl! - Majestic-12

greenbeansneedlesΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 5 μήνες)

68 εμφανίσεις

Building a scalable

distributed

WWW search engine



NOT in Perl!







Presented by Alex Chudnovsky (http://www.majectic12.co.uk)

at Birmingham Perl Mongers User Group (http://birmingham.pm.org)

V1.0 27/07/05

Contents


1.
History

2.
Goals

3.
Architecture

4.
Implementation

5.
Why not Perl?

6.
Conclusions

7.
Credits

8.
Recommended reading

History

(of my work in area of information retrieval)


1.
First primitive pathetic stone
-
age search engine: 1000
documents in the “index” (1997, Perl)

2.
Second engine using proper inverted indexing for Jungle.com:
500,000 products indexed (Perl + Java, 2002)

3.
Current: 50,000,000 pages indexed with a lot more to go (to be
revealed, 2005)

Goals




1.
Build
a
distributed WWW search engine capable of dealing with at
least 1 bln web pages

based on principles of
SETI@Home

and D.NET

2.
See
to it that the

chosen language for implementation (more on this
later) fits purpose or more likely learn how to make it work

3.
Eventually make some money out of it



Architecture


1.
Data collection (crawling)

2.
Indexing: turning text into numbers

3.
Merging: turning indexed barrels into single searchable index

4.
Searching: locating documents for given keywords


Data collection (crawling)


Base

Issues URLs to crawl and
receives compressed pages

Distributed
c
rawlers


receive lists
of URLs to
crawl, crawl
them and
send back
compressed
data.

In the future will do
distributed indexing

Note: this stage is optional if you already have data to index, ie list of products with their descriptions

Crawler screenshot 1


Crawler screenshot 2


Crawler screenshot 3


Crawler screenshot 4


Crawler screenshot 5


Current Stats


Source:
http://www.majestic12.co.uk/projects/dsearch/stats.php

as of 27/07/05

Indexing


Indexing is a process of turning words into numbers and creating inverted index.

Data barrel

Doc #0: Birmingham Perl Mongers

Doc #1: Birmingham City

Doc #2: Perl City

Lexicon

(maps words to their numeric WordIDs)

Birmingham


0

Perl


1

Mongers


2

City


3


Inverted Index

(Each of the WordID has list of

(ideally sorted) DocIDs)

0

-
> 0, 1

1

-
> 0, 2

2

-
> 0,

3

-
> 1, 2

Note: if you use database then it make sense to have clustered index on WordID

Merging


Individual
indexed
barrels

Single
searchable
index

Note: this stage is not necessary if just one barrel is used as there will be no need to remap all Ids from local to their
global equivalents.

Searching


Searching is a process of finding documents that contain words from search query

Doc #0: Birmingham Perl Mongers

Doc #1: Birmingham City

Doc #2: Perl City

Lexicon

(maps words to their numeric WordIDs)

Birmingham


0

Perl


1

Mongers


2

City


3


Inverted Index

(lists DocIDs for each of the WordID)

0

-
> 0, 1

1

-
> 0, 2

2

-
> 0,

3

-
> 1, 2

Note: if you use database then it make sense to cluster on WordID

Search query:

“Birmingham Perl”

WordIDs: 0, 1

Intersection of DocIDs present in both lists
(implementation of boolean AND logic):

0 (Brum)

1 (Perl)

Result

0

0

Matched!

1

n/a

Not matched!

n/a

2

Not matched!

Search engine screenshot 1

Search engine screenshot 2

Implementation


1.
Microsoft .NET C# ported to Linux using Mono (
http://www.mono
-
project.com
)

2.
~90k lines of code (minimal copy/paste) written from scratch

3.
Low level of dependencies (SharpZipLib/SQLite/NPlot)



Why not Perl?

(using C
# instead)

1.
Not strong in GUI department

2.
Hard to deal with Multi
-
Threading and Asyncronous sockets

3.
OOP is more of a hack

4.
Lax compile
-
time checks

due to not being strictly typed

5.
Fear of performance bottlenecks forcing to use C++

6.
Hard to profile for performance analysis

7.
Managed memory lacks support for pointers
(?)

8.
Poor exceptions handling

9.
I wanted something new :)


Conclusions


Still work in progress, but some conclusions can be made already:

1.
Inverted indexing approach helps to achieve fast searches

2.
Its tough to build one


don’t try if you ain’t going to see it through!

3.
Crawler is one tough piece of code


6 months vs 2 months on searching

4.
.NET C# is a decent language suitable for heavy duty tasks like this


Credits


1.
R&D: Alex Chudnovsky <alexc@majestic12.co.uk>

2.
Pioneers*: FiddleAbout, dazza12, lazytom, Mordac, linuxbren,
Cyber911,
www.vanginkel.info
, Vari, ASB, SEOBy.org, arni, japonicus,
webstek.info | Pimpel, DimPrawn, Zyron, partys
-
bei
-
uns.de, jake, bull at
webmasterworld, nada, dodgy4, sri
-
heinz


* Volunteers running crawler and who crawled at least 1 mln URLs as of 27/07/05

Recommended reading


1.
“The Anatomy of a Large
-
Scale Hypertextual Web Search Engine”
Sergey Brin and Lawrence Page of Google (
http://www
-
db.stanford.edu/~backrub/google.html
)

2.
“Managing Gigabytes” Ian h. Witten et al ISBN 1
-
55860
-
570
-
3


Join!




Join the project

(unmetered broadband required!):



majestic12.co.uk


Your name could be
here!