Hubs and Authorities on the
world wide web
(most from Rao’s lecture slides)
Presentor: Lei Tang
Desiderata for link

based ranking
•
A page that is referenced by lot of important pages (has more
back links
) is more important (Authority)
–
A page referenced by a single important page may be more
important than that referenced by five unimportant pages
–
No links between competitive authorities(like Ford, Honda)
•
A page that references a lot of important pages is also important
(Hub)
•
Good authoritative pages (
authorities
) and good hub pages
(
hubs
) reinforce each other.
•
“Importance” can be propagated
–
Your importance is the weighted sum of the importance
conferred on you by the pages that refer to you
–
The importance you confer on a page may be proportional
to how many other pages you refer to (cite)
•
(Also what you say about them when you cite them!)
Different
Notions of
importance
Authority and Hub Pages (2)
•
Authorities and hubs related to the same query
tend to form a bipartite subgraph of the web
graph.
•
A web page can be a good authority and a
good hub.
hubs
authorities
Authority and Hub Pages (7)
Operation I: for each page p:
a(p) =
†††
栨焩
††††††††
焺q⡱Ⱐ瀩
E
Operation O: for each page p:
h(p) =
††††
愨焩
††††††††
焺q⡰Ⱐ焩
E
q
1
q
2
q
3
p
q
3
q
2
q
1
p
Authority and Hub Pages (8)
Matrix representation of operations I and O.
Let A be the adjacency
matrix of SG: entry (p, q) is
1 if p has a link to q, else the entry is 0.
Let A
T
be the transpose of A.
Let h
i
be vector of hub scores after i iterations.
Let a
i
be the vector of authority scores after i
iterations.
Operation I: a
i
= A
T
h
i

1
Operation O: h
i
= A a
i
1
1
i
T
i
i
T
i
h
AA
h
Aa
A
a
0
0
h
AA
h
a
A
A
a
i
T
i
i
T
i
Normalize after every multiplication
Authority and Hub Pages (11)
Example: Initialize all scores to 1.
1
st
Iteration:
I operation:
a(q
1
) = 1, a(q
2
) = a(q
3
) = 0,
a(p
1
) = 3, a(p
2
) = 2
O operation: h(q
1
) = 5,
h(q
2
) = 3, h(q
3
) = 5, h(p
1
) = 1, h(p
2
) = 0
Normalization: a(q
1
) = 0.267, a(q
2
) = a(q
3
) = 0,
a(p
1
) = 0.802, a(p
2
) = 0.535, h(q
1
) = 0.645,
h(q
2
) = 0.387, h(q
3
) = 0.645, h(p
1
) = 0.129, h(p
2
)
= 0
q
1
q
2
q
3
p
1
p
2
Authority and Hub Pages (12)
After 2 Iterations:
a(q
1
) = 0.061, a(q
2
) = a(q
3
) = 0, a(p
1
) = 0.791,
a(p
2
) = 0.609, h(q
1
) = 0.656, h(q
2
) = 0.371,
h(q
3
) = 0.656, h(p
1
) = 0.029, h(p
2
) = 0
After 5 Iterations:
a(q
1
) = a(q
2
) = a(q
3
) = 0,
a(p
1
) = 0.788, a(p
2
) = 0.615
h(q
1
) = 0.657, h(q
2
) = 0.369,
h(q
3
) = 0.657, h(p
1
) = h(p
2
) = 0
q
1
q
2
q
3
p
1
p
2
(why) Does the procedure converge?
0
0
2
1
2
0
1
)
(
x
M
x
x
M
Mx
x
AA
M
Mx
x
k
k
T
x
x
2
x
k
1
0
2
2
1
1
0
1
1
1
1
1
2
1
1
2
1
)
...
),
,...
,
(
]
ˆ
...
ˆ
ˆ
[
ˆ
ˆ
...
ˆ
ˆ
2
1
2
1
2
1
e
x
M
e
c
e
c
e
c
x
E
E
E
E
M
E
E
E
E
E
E
M
E
E
M
k
n
n
k
k
k
k
k
diag
e
e
e
n
n
n
The rate of convergence depends on the “eigen gap”
2
1
Authority and Hub Pages (3)
Main steps of the algorithm for finding good authorities
and hubs related to a query
q
.
1.
Submit
q
to a regular similarity

based search
engine. Let
S
be the set of top n pages returned
by the search engine. (
S
is called the
root set
and
n is often in the low hundreds).
2.
Expand
S
into a large set T (
base set
):
•
Add pages that are pointed to by any page in
S
.
•
Add pages that point to any page in
S
.
•
If a page has too many parent pages, only the first k
parent pages will be used for some k.
Authority and Hub Pages (4)
3. Find the subgraph SG of the web graph that is
induced by T.
S
T
Authority and Hub Pages (5)
Steps 2 and 3 can be made easy by
storing the link structure of the
Web in advance Link structure
table (during crawling)

Most search engines serve this
information now. (e.g. Google’s
link: search)
parent_url child_url
url1 url2
url1 url3
Authority and Hub Pages (6)
4.
Compute the authority score and hub score of
each web page in T based on the subgraph
SG(V,
E).
Given a page p, let
a(p)
be the authority score of p
h(p)
be the hub score of p
(
p, q
) be a directed edge in E from p to q.
Two basic operations:
•
Operation I: Update each a(p) as the sum of all
the hub scores of web pages that point to p.
•
Operation O: Update each h(p) as the sum of all
the authority scores of web pages pointed to by p.
Authority and Hub Pages (9)
After each iteration of applying Operations I
and O, normalize all authority and hub scores.
Repeat until the scores for each page
converge (the convergence is guaranteed).
5. Sort pages in descending authority scores.
6. Display the top authority pages.
V
q
q
a
p
a
p
a
2
)
(
)
(
)
(
V
q
q
h
p
h
p
h
2
)
(
)
(
)
(
Authority and Hub Pages (10)
Algorithm (summary)
submit q to a search engine to obtain the root
set S;
expand S into the base set T;
obtain the induced subgraph SG(V, E) using T;
initialize a(p) = h(p) = 1 for all p in V;
for each p in V until the scores converge
{ apply Operation I;
apply Operation O;
normalize a(p) and h(p);
}
return pages with top authority scores;
Handling “spam” links
Should all links be equally treated?
Two considerations:
•
Some links may be more
meaningful/important than other links.
•
Web site creators may trick the system to
make their pages more authoritative by
adding dummy pages pointing to their
cover pages (spamming).
Handling Spam Links (contd)
•
Transverse link:
links between pages with
different domain names.
Domain name:
the first level of the URL of a page.
•
Intrinsic link:
links between pages with the
same domain name.
Transverse links are more important than
intrinsic links.
Two ways to incorporate this:
1.
Use only transverse links and discard
intrinsic links.
2.
Give lower weights to intrinsic links.
Handling Spam Links (contd)
How to give lower weights to intrinsic
links?
In adjacency matrix A, entry (p, q) should
be assigned as follows:
•
If p has a transverse link to q, the entry
is 1.
•
If p has an intrinsic link to q, the entry is
c, where 0 < c < 1.
•
If p has no link to q, the entry is 0.
Considering link “context”
For a given link (p, q), let
V(p, q)
be the vicinity
(e.g.,
㔰5捨慲c捴c牳潦瑨攠汩湫n
•
If
V(p, q)
contains terms in the user query
(topic), then the link should be more useful
for identifying authoritative pages.
•
To incorporate this: In adjacency matrix
A
,
make the weight associated with link
(p, q)
to
be
1+n(p, q),
•
where n(p, q) is the number of terms in V(p, q) that appear
in the query.
•
Alternately, consider the “vector similarity” between
V(p,q) and the query Q
Evaluation
Sample experiments:
•
Rank based on large in

degree (or backlinks)
query: game
Rank in

degree URL
1 13
http://www.gotm.org
2 12
http://www.gamezero.com/team

0/
3 12
http://ngp.ngpc.state.ne.us/gp.html
4 12
http://www.ben2.ucla.edu/~permadi/
gamelink/gamelink.html
5 11
http://igolfto.net/
6 11
http://www.eduplace.com/geo/indexhi.html
•
Only pages 1, 2 and 4 are authoritative game pages.
Evaluation
Sample experiments (continued)
•
Rank based on large authority score.
query: game
Rank Authority URL
1 0.613
http://www.gotm.org
2 0.390
http://ad/doubleclick/net/jump/
gamefan

network.com/
3 0.342
http://www.d2realm.com/
4 0.324
http://www.counter

strike.net
5 0.324
http://tech

base.com/
6 0.306
http://www.e3zone.com
•
All pages are authoritative game pages.
Authority and Hub Pages (19)
Sample experiments (continued)
•
Rank based on large authority score.
query: free email
Rank Authority URL
1 0.525
http://mail.chek.com/
2 0.345
http://www.hotmail/com/
3 0.309
http://www.naplesnews.net
/
4 0.261
http://www.11mail.com/
5 0.254
http://www.dwp.net/
6 0.246
http://www.wptamail.com/
•
All pages are authoritative free email pages.
Tyranny of Majority
1
2
3
4
6
7
8
5
Which do
you
think are
Authoritative pages?
Which are good hubs?

intutively, we would say
that 4,8,5 will be authoritative
pages and 1,2,3,6,7 will be
hub pages.
BUT
The power iteration will show that
Only 4 and 5 have non

zero authorities
[.923 .382]
And only 1, 2 and 3 have non

zero hubs
[.5 .7 .5]
Tyranny of Majority (explained)
p1
p2
pm
p
q1
qn
q
m
n
Suppose h0 and a0 are all initialized to 1
2
2
1
2
2
1
1
1
)
(
)
(
)
(
)
(
n
m
n
q
a
n
m
m
p
a
normalized
n
q
a
m
p
a
2
2
1
2
2
1
)
(
)
(
n
m
n
q
h
n
m
m
p
h
i
i
2
2
2
2
2
2
2
2
2
2
2
)
(
)
(
)
(
)
(
m
n
p
a
q
a
n
m
n
q
a
n
m
m
p
a
0
)
(
)
(
k
k
k
m
n
p
a
q
a
m>n
Impact of Bridges..
1
2
3
4
6
7
8
5
When the graph is disconnected,
only 4 and 5 have non

zero authorities
[.923 .382]
And only 1, 2 and 3 have non

zero hubs
[.5 .7 .5]CV
9
When the components are bridged by adding one page (9)
the authorities change
only 4, 5 and 8 have non

zero authorities
[.853 .224 .47]
And 1, 2, 3, 6,7 and 9 will have non

zero hubs
[.39 .49 .39 .21 .21 .6]
Authority and Hub Pages (24)
Multiple Communities (continued)
•
How to retrieve pages from smaller communities?
A method for finding pages in nth largest community:
–
Identify the next largest community using the existing
algorithm.
–
Destroy this community by removing links associated
with pages having large authorities.
–
Reset all authority and hub values back to 1 and
calculate all authority and hub values again.
–
Repeat the above n
ㄠ1業敳湤n瑨攠湥硴慲来獴
捯浭畮楴cw楬i扥b瑨攠湴栠污牧敳琠捯浭畮楴c.
Multiple Clusters on “House”
Query: House (first community)
Authority and Hub Pages (26)
Query: House (second community)
More stable because
random surfer model
allows low prob edges
to every place.CV
Can be done
For base set too
Can be done
For full web too
Can be made stable with subspace

based
A/H values [see Ng. et al.; 2001]
See topic

specific
Page

rank idea..
Novel uses of Link Analysis
•
Link analysis algorithms
—
HITS, and
Pagerank
—
are not limited to hyperlinks

Citeseer/Cora use them for analyzing citations
(the link is through “citation”)

See the irony here
—
link analysis ideas originated from
citation analysis, and are now being applied for citation
analysis

Some new work on “keyword search on
databases” uses foreign

key links and link
analysis to decide which of the tuples matching
the keyword query are most important (the link is
through foreign keys)

[
Sudarshan et. Al. ICDE 2002
]

Keyword search on databases is useful to make
structured databases accessible to naïve users who don’t
know structured languages (such as SQL).
Enter the password to open this PDF file:
File name:

File size:

Title:

Author:

Subject:

Keywords:

Creation Date:

Modification Date:

Creator:

PDF Producer:

PDF Version:

Page Count:

Preparing document for printing…
0%
Comments 0
Log in to post a comment