Вы находитесь на странице: 1из 50

HRS JUDGING GUIDELINES

_______________________________________________________________________
Last Updated: 6/18/20136/18/2013 by Mia Spangenberg (mispan)

These guidelines contain confidential and proprietary information belonging to Microsoft Corporation.

The recipient understands and agrees that these materials and the information contained herein may not be used
or disclosed without the prior written consent of Microsoft Corporation.

1
Table of Contents
1 Overview ................................................................................................................................................................................... 3
2 Possible user intents .................................................................................................................................................................. 4
3 Landing Page relationship to possible user intents .................................................................................................................... 6
4 Selecting the Rating ............................................................................................................................................................... 911
5 Rating Options ..................................................................................................................................................................... 1113
5.1 Perfect ................................................................................................................................................................................. 1113
5.2 Excellent .............................................................................................................................................................................. 1214
5.3 Good .................................................................................................................................................................................... 1315
5.4 Fair ...................................................................................................................................................................................... 1416
5.5 Bad ...................................................................................................................................................................................... 1517
5.6 Detrimental ......................................................................................................................................................................... 1618
5.6.1 Adult Queries .................................................................................................................................................................. 1618
5.6.2 Child pornography .......................................................................................................................................................... 1618
5.7 No Judgment: Login ............................................................................................................................................................ 2022
5.8 No Judgment: Foreign ........................................................................................................................................................ 2022
5.9 No Judgment: Page Did Not Load (PDNL) ........................................................................................................................... 2123
5.9.1 Domain lookup failure .................................................................................................................................................... 2123
5.9.2 Search page on domain lookup failure, returned by the users ISP. .............................................................................. 2123
5.9.3 No error message, just a blank page .............................................................................................................................. 2123
5.9.4 LP hacked ........................................................................................................................................................................ 2224
5.9.5 Access denied ................................................................................................................................................................. 2224
5.9.6 Requested page cannot be found................................................................................................................................... 2325
5.9.7 Temporary server error or application crash.................................................................................................................. 2426
5.9.8 LP down for maintenance or is under construction ....................................................................................................... 2426
6 Query Considerations .......................................................................................................................................................... 2527
6.1 Misspellings, Punctuation, Capitalization, and Advanced Search Operators ...................................................................... 2527
6.2 URL Queries ......................................................................................................................................................................... 2628
6.3 Location-specific queries ..................................................................................................................................................... 2729
6.4 Personal Names .................................................................................................................................................................. 2830
7 Landing Page Considerations ............................................................................................................................................... 2932
7.1 General tips ......................................................................................................................................................................... 3032
7.2 Splash Pages and Entry Pages ............................................................................................................................................. 3133
7.3 Login pages ......................................................................................................................................................................... 3335
7.4 Spam, Phishing sites, and Parked Domains......................................................................................................................... 3436
7.5 Proxies and Frames ............................................................................................................................................................. 3638
7.5.1 Proxies ............................................................................................................................................................................ 3638
7.5.2 Frames ............................................................................................................................................................................ 3638
7.6 Freshness ............................................................................................................................................................................. 3841
7.7 Market................................................................................................................................................................................. 4142
7.7.1 Queries where market does not matter ......................................................................................................................... 4142
7.7.2 Queries where market does matter ............................................................................................................................... 4143
7.8 Language............................................................................................................................................................................. 4244
8 Mixed Rating Examples ....................................................................................................................................................... 4345
9 Glossary ............................................................................................................................................................................... 4951
10 Process Diagram .................................................................................................................................................................. 5052

2
1 Overview
When users enter words to search for (a query) into a web search engine such as Bing, Google, or Yahoo, they expect the search
engine to understand what they are looking for, and to return web documents that completely satisfy their requirements. With the
huge number of documents on the Web, the ambiguities of human language, and the usually short queries that users enter, this is a
hard problem.

As a judge working on the Human Relevance System (HRS) project, you will see queries that users type into search engines, and web
documents that search engines return for these queries. You will evaluate how satisfied users would be with each document, and so
help us evaluate and improve the quality of the results that we present to our users.

These guidelines describe how to accurately determine user intent and assess the relevance of a web document to that intent. While
the task may seem daunting, there are really only four steps:

1. Consider what the user could have had in mind (their intent) when they typed in the query; allow for misspellings or other
ambiguity.
2. Determine whether a given web document covers any of these intents, and if so, how well it satisfies them.
3. Give the document a rating on a five-point scale to indicate how useful it would be to most users who type in the query.
4. Where necessary, provide additional information to explain your rating.

The rest of the document explains how to perform each of these steps. Keep these basic tips in mind:

Ratings indicate how a search engine should sort results; it should be hard to sort results that have the same rating. If you
give two documents the same rating, it should be hard to say that one is clearly better and more useful than the other.
Ratings should be decided in the context of the target language and the target market; when encountering queries or
content in a foreign language or aimed at users in a different country, consider how likely it is that the user is looking for
such content, and how useful they would find that particular document.
You will often need to research possible intents associated with the query; however, you must not base your rating on how
the page is treated in search engines today; if search engines were perfect we wouldnt need human judges!
Make your judgment of relevance to the query strictly according to the guidelines and the content of the landing page.
o Dont be biased by a personal opinion unlikely to be shared by most web users.
o Dont change your ratings to get a particular rating distribution.
If you see any checkboxes alongside the rating options in the tool, choose all the ones that apply to your rating.
We have included many examples in these guidelines; you should use these to understand the concepts, and not expect an
example for every possible type of query or web site. The example intents weve listed for queries are not exhaustive; you
will probably be able to identify more.
Web sites frequently change; if you find that an example in the guidelines doesnt make sense, please check with your
manager the example may no longer be correct.
Dont rush! You will often need to research the possible user intents, and will need to think carefully before assigning a final
rating to a page. You should aim to meet the speed requirements suggested by your manager, but never sacrifice quality for
speed; we cannot use poor-quality ratings.
Dont worry! These guidelines are to help you understand the judging task; you will quickly become skilled at assessing
pages without continually referring to them but keep them handy, especially the Rating Matrix and the Process Diagram.

3
2 Possible user intents
Before you can decide how happy a user would be with a web document, you must determine what they were thinking
(what their intent was) when they typed the query into the search engine. Users with identical queries may not be looking
for exactly the same thing, and some intents may be more popular than others.
Most query intents can be classified into one or more of these three categories:

Navigational (N) Informational (I) Transactional (T)


Find a specific page Learn more about a topic. Complete a transaction
Usually a personal, company, or Query may contain words like how Involves interaction with a web resource, paid or free; query
brand name. to, what is, ways to, or review. may include words like buy, chat, register, or download

Likewise, the likelihood or popularity of an intent can be categorized into one of the five levels indicated below. An intent
that is most likely means that over 50% percent of users have a specific intent in mind. An intent that is very unlikely
indicates that less than 1% of users have that particular intent in mind for the query.

Most Likely
>50%

Very Likely
25-50%

Likely
How likely is this intent?
10-25%

Unlikely
1-10%

Very Unlikely
<1%

4
This chart illustrates the process of identifying the type of intent and the likelihood of that intent for several different kinds
of queries.

Query and possible intents Type Likelihood of this intent for this query
en-us, {facebook}
visit the Facebook site (N) Most likely (>50%)
learn about Facebook (I) Unlikely (1-10%)
go to the Facebook Help section (I)+(N) Unlikely (1-10%)
en-us, {Saturn moons}
find information on the satellites of the planet Saturn (I) Most likely (>50%)
en-us, {Saturn recall}
find information on product recalls by the Saturn car company (I) Most likely (>50%)
en-us, {2007 jetta wiper blades}
buy wiper blades for this specific car model (T) Most likely (>50%)
learn how to replace wiper blades for 2007 Jetta (I) Likely (10-25%)
en-us, {download boxee}
download the Boxee application program from the official web site (T)+(N) Most likely (>50%)
en-us, {gran torino}
find information on the movie, perhaps from the official site (I)+(N) Most likely (>50%)
visit the official web site of the movie (N) Very likely (25-50%)
buy a DVD of the movie (T) Likely (10-25%)
find information on the 1972-1976 Ford car (I) Unlikely (1-10%)
en-us, {nine inch nails}
find information on the rock band, perhaps from the official site (I)+(N) Most likely (>50%)
visit the official web site of the rock band (N) Very likely (25-50%)
watch Nine Inch Nails videos (T) Very likely (25-50%)
buy nails that are nine inches long (T) Unlikely (1-10%)
en-us, {diabetes}
find information on this disease (I) Most likely (>50%)
visit the site for the journal Diabetes (N) Unlikely (1-10%)
en-us, {touch cruise}
learn about this device, perhaps from the official product page (I)+(N) Most likely (>50%)
read user reviews (I) Very likely (25-50%)
buy the device online (less likely as its from 2008-2009) (T) Likely (10-25%)
learn about Templeton Tours cruises (I)+(N) Very Unlikely (<1%)
en-us, {Martha Stewart}
visit Marthas official web site (N) Most likely (>50%)
visit Marthas recipe page (N) Very likely (25-50%)
learn about Marthas prison sentence (I) Unlikely (1-10%)
buy Martha Stewart flowers online (N)+(T) Unlikely (1-10%)
buy Marthas Dinner at Home book (T) Unlikely (1-10%)
en-us, {Mikasa}
visit the glassware manufacturers web site (N) Most likely (>50%)
purchase Mikasa glassware products online (T) Very likely (25-50%)
visit the sporting goods manufacturers web site (N) Likely (10-25%)
purchase Mikasa sporting goods online (T) Likely (10-25%)
learn about the legal case of Mr. Scott Mikasa of Hawaii (I) Very Unlikely (<1%)
en-us, {cobalt}
learn about the element (N) Very likely (25-50%)
visit Cobalt company's web site (I)+(N) Likely (10-25%)
learn about the discontinued Chevy Cobalt, perhaps on Chevy's site (I)+(N) Unlikely (1-10%)
go to home page of Cobalt Boats (N)+(T) Unlikely (1-10%)
buy Kobalt tools (T) Very Unlikely (<1%)

5
3 Landing Page relationship to possible user intents
For each Landing Page (LP), consider these four questions:
1. Intent: Does the LP content address a possible intent for the query?
To help understand how the LP might be relevant to the query, you should check whether and where the query
terms appear in the LP title, content, and URL; note that a page can still be highly relevant even if they dont.
Useful keyboard shortcuts: Ctrl + F will do a find; F3 or Ctrl+G will find the next occurrence.
Compare the nature of the query with the LP; for example, a Wikipedia page on hiking boots does not address
a possible intent for a user searching for {cheap hiking boots}, even though all the words of the query may
appear on the page.
Evaluate the relevance of the main content, not the advertisements that may appear on the page.
You must assess LPs primarily by their content; however, the URL can give additional information to help
determine a rating; if the URL appears unrelated to the query, review the LP to see whether it is displaying
content from a different web site (see the description of frames and proxies) or is a spam site (see Section 7.4).
2. Scope: Does the range and depth of the LP content match what the user wants?
Assess each LP individually; even LPs on the same site can vary greatly in scope. For example, some pages on
Wikipedia.org are a couple of paragraphs long, while others are very detailed and span several pages.
Remember to focus on the users requirements:
More content is not necessarily better; the user may be looking for a specific piece of information.
Broader content may be more appropriate for broad queries; a user looking for information on cancer
will likely be happier with a page featuring information on the different types of cancer and their
treatment than one that focuses only on a single type of cancer..
3. Authority: Is the trustworthiness of the content on the LP appropriate to the expectations of the user?
LPs published by official/definitive web sites are the most authoritative and credible.
Official company LPs typically include a company logo; clicking on this logo takes the user to the
company home page. They also usually contain a contact us or about us link.
Official home pages of celebrities (e.g., actor, artist, author, or band) typically include the statement,
Official site of or Official home page of ; there may be many fan web sites, but there will
usually be only one (genuine) official home page.
The authority and reliability of content from non-official sources should always be evaluated carefully in the
context of the query. To determine whether a site is trustworthy, ask yourself:
Would you be comfortable downloading software from this site?
Would you be comfortable giving this site your credit card information?
Does the LP indicate that the information has been peer reviewed or provided by a credible source?
Is this a popular site? Popular retail sites like Amazon.com and eBay.com are more likely to invest in
providing safe transactions for their users, and to keep their pages free of excessive clutter.
Be especially wary of information generated by unknown users, such as:
content on sites that are not well-known or popular
content mills like http://ehow.com; see this listing for more examples
collaboratively created content on well-known sites such as http://wikipedia.org
user-generated content on sites such as http://answers.yahoo.com
user comments on blogs and articles, such as news postings on http://reuters.com
Consider the users expectations; dont assume that highly credible or official LPs best satisfy the intent:
Users may not be interested in only official pages; for example, the official Canon inkjet printer
cartridge page may not be the best result for a user looking for cheap cartridge refills. Similarly, the
best solution to a particular issue with Microsoft Word may not be on an official Microsoft LP.
Even content that is wrong or offensive may be relevant. For example, users looking for satire or
celebrity gossip may not care much about truthfulness.
4. Quality: Do the appearance and organization of the LP providing a satisfying experience?
In order to determine how satisfying an LP is, think about these questions:
Does the page contain excessive ads or distracting pop-ups?
Is the information organized in an easily digestible format? For some types of information, splitting
the content onto several pages is less desirable, but for queries that are broad and imply an intent for
a large volume of information, an LP that neatly organizes the content can be highly satisfying.

6
When you take all of these factors into consideration, you will be able to determine how satisfying a particular LP is for the
intent you have identified. A strongly satisfying page will closely match the users intent and requirements in scope and
authority, while a poorly satisfying result will be useful to almost no users.

How well does this LP satisfy this intent?

Obscene Content
Strongly Moderately Weakly Poorly
content inaccessible

On the following pages, we examine how well different LPs address the possible intents for a single query (well discuss how
to assign ratings later, including how to rate detrimental and inaccessible content).

en-us {English Premiership}


o By researching the query, we learn from Wikipedia that the Premiership is another name used for the Premier
League, the top-level English football (soccer) league.

http://www.premierleague.com satisfies the navigational intent strongly. the navigational intent strongly.

Intent Scope Authority Quality

This page addresses the The LP contains suitableThis site belongs to the Links are neatly organized
intent visit official web site of range and depth of content.
league itself (Barclays Bank is according to user interests.
the Premier League. the current sponsor), so its
the most official and reliable source.
http://en.wikipedia.org/wiki/Premier_League satisfies an informational intent strongly.

Intent Scope Authority Quality

This page addresses the The LP contains ? Wikipedia articles cannot be LP is well-organized with
intent learn all about the appropriate range and blindly trusted, but this article hyperlinks to different sections.
Premier League. depth of content. has references to credible sources.

7
http://news.bbc.co.uk/sport2/hi/football/eng_prem/default.stm satisfies an informational intent strongly.

Intent Scope Authority Quality

This page addresses the intent LP contains a good The authority is high, LP contains links to videos
find news about the Premier range and depth of since the BBC is a well-respected and articles featuring the
League. recent news. news source. latest news.
http://www.premiershiptickets.net satisfies a transactional intent moderately.

Intent Scope Authority Quality

This LP addresses the


LP has details on only a ? The LP belongs to a company LP is uncluttered by ads.
few teams (though it asks that sells tickets, but doesnt
transactional intent buy tickets users to call for other
for a Premier League match. appear to be officially linked to
teams); bookings are done the Premier League, so its
through an email form. authority is unknown.
http://www.englishclub.com does not satisfy any realistic query intent.

Intent Scope Authority Quality


There is no visible relationship There is no content None that can be seen. Excessive links lead to
between the content and a valid related to the query. further links.
intent.

8
4 Selecting the Rating
In this section we describe how to use the Rating Matrix to narrow down the rating options for an LP by considering both the
likelihood of the intent it addresses, and how well it satisfies that intent. You should read the full document to see examples
for each rating and to understand the variety of situations you may encounter.
To use the Rating Matrix, you should:
1. Identify what primary intent the LP addresses, and decide how likely it is that users typing in this query have this
particular intent in mind (row labels). Note that you are considering the intent addressed by the LP, not the main intent
of the query.
2. You should then determine how authoritatively and comprehensively the LP satisfies that intent (column labels).
Notice that the ratings for inaccessible, obscene, or irrelevant content do not change with likelihood of intent.

How well does this LP satisfy this intent?

Obscene Content
Strongly Moderately Weakly Poorly
content inaccessible

Most
No
Likely Excellent Good Fair Bad Detrimental
Judgment
>50% [Perfect]

Very
No
Likely Excellent Good Fair Bad Detrimental
Judgment
How likely is this intent?

25-50%

Likely No
Good Fair Bad Bad Detrimental
10-25% Judgment

Unlikely No
Fair Bad Bad Bad Detrimental
1-10% Judgment

Very
No
Unlikely Bad Bad Bad Bad Detrimental
Judgment
<1%

Table 1: The Rating Matrix

9
LP URL Intent addressed by LP Likelihood of Degree to which LP Rating Formatted Table
this intent satisfies this intent
en-us, {amazon}
o http://amazon.com visit the Amazon online store (N) Most likely Strongly Perfect Formatted: Indent: Left: 0.2"
http://en.wikipedia.org/wiki/A learn about the Amazon.com Unlikely Strongly Fair Formatted Table
mazon.com company (I)
Formatted: Indent: Left: 0.2"
http://en.wikipedia.org/wiki/A learn about the Amazon river (I) Unlikely Strongly Fair
Formatted: Indent: Left: 0.2"
mazon_River
http://www.amazon- learn about the Amazon river (I) Unlikely Weakly Bad Formatted: Indent: Left: 0.2"
rainforest.org/amazon- (about a trip down
river.html the river)
en-us, {beyonce knowles}
o http://www.beyonceonline.co current news on Beyonc Knowles (I) Most likely Strongly Perfect Formatted: Indent: Left: 0.2"
m and (official home page) Formatted Table
visit Beyonc Knowles home page
(N)
http://en.wikipedia.org/wiki/B find background information on Likely Strongly Good Formatted: Indent: Left: 0.2"
eyonc%C3%A9_Knowles Beyonc Knowles (I) (detailed and with
many citations)
http://www.people.com/peopl current news on Beyonc Knowles (I) Most likely Moderately Good Formatted: Indent: Left: 0.2"
e/beyonce_knowles (has photos and
relevant links)
http://www.nydailynews.com/ current news on Beyonc Knowles (I) Most likely Weakly Fair Formatted: Indent: Left: 0.2"
entertainment/beyonce-game- (Judged June 2013) (story on alawsuit
developer-100-million-suit- relating to Beyonc )
article-1.1373234
http://www.knowles.com visit Knowles acoustic components Very unlikely Doesnt matter Bad Formatted: Indent: Left: 0.2"
(N) (no effect on rating)
en-us, {crocs shoes}
o http://www.crocs.com/ buy Crocs shoes (T) Most likely Strongly Perfect Formatted: Indent: Left: 0.2"
and (official site with Formatted Table
information on Crocs shoes (I) online store)
http://www.shoebuy.com/croc buy Crocs shoes (T) Most likely Moderately Good Formatted: Indent: Left: 0.2"
s-shoes.htm (has a range of Crocs
shoes for sale, but
not on official site)
http://www.crocs.com/crocs- buy cheap Crocs shoes (T) Likely Strongly Good Formatted: Indent: Left: 0.2"
sale- (official discount
outlet/outlet,default,sc.html outlet)
http://www.shoes.com/en- buy Crocs mens Baja Realtree APC Unlikely Strongly Fair Formatted: Indent: Left: 0.2"
US/Product/91366- flip flops (T) Formatted: Font: +Body (Calibri), 10 pt
1023804/Crocs/Black_Black/M
ens+Baja+Realtree+APC.aspx
http://travel.ninemsn.com.au/ information on vacations involving Very unlikely Doesnt matter Bad Formatted: Indent: Left: 0.2"
holidaytype/weird/655271/cro crocodiles (I) (no effect on rating)
c-cuddling-in-africa
en-us, {MIX106.5}; multiple radio stations carry this branding, so we cannot determine a most likely intent.
o http://todaysbestmix.com visit the San Jose CA station official Likely Strongly Good Formatted: Indent: Left: 0.2"
LP (N) (official site) Formatted Table
http://www.kezr.com/pages/9 visit the San Jose CA station official Likely Moderately Fair
Formatted: Indent: Left: 0.2"
226030.php LP (N) (site music sub-page)
http://en.wikipedia.org/wiki/W history of the Cleveland OH station Unlikely Moderately Bad Formatted: Indent: Left: 0.2"
HLK (I)
http://www.mix106radio.com/ Visit the Boise ID station official LP Very unlikely Doesnt matter Bad Formatted: Indent: Left: 0.2"
pages/main (N) (no effect on rating)

10
Formatted: Indent: Left: 0", First line: 0"

5 Rating Options
5.1 Perfect
The LP is the definitive or official page that strongly satisfies the most likely intent.
The LP content is exactly what most users want, and should appear as the top search result.
Typically, only one LP is the Perfect result for a query. In rare cases, multiple official LPs equally satisfying the most likely
intent can get the same Perfect rating.
If no intent is most likely, no LP merits a Perfect. Thus, highly ambiguous or general queries such as {loans}, {blinds},
{vacuum cleaners}, {books}, {drugstore}, and {insurance} will not have a Perfect LP.

en-us, {apple}
o http://www.apple.com/ is Perfect; the most likely intent for this query is the company, rather than the fruit. This LP
strongly satisfies the most likely intent.
en-us, {migente}
o http://www.migente.com is Perfect; home page of the company that strongly satisfies the most likely intent.
en-us, {microsoft wired keyboard 600 software}
o http://www.microsoft.com/hardware/en-us/d/wired-keyboard-600 is Perfect; software download page from the
official Microsoft site that strongly satisfies the most likely intent.
o .
en-us, {v&a london}
o http://www.vam.ac.uk is Perfect; official web page of the museum.
en-us, {lg LGE960W}
o http://www.lg.com/us/cell-phones/lg-LGE960W-nexus-4-white is Perfect; official web page for this specific cellphone
model.
en-us, {drugstore.com}
o http://drugstore.com is Perfect; official web page strongly satisfying the most likely intent.
en-us, {a game}
o http://www.agame.com is Perfect; the query is ambiguous, but most users entering this query have this specific site
in mind (navigational intent), or want to play an online game (a transactional intent that this site satisfies).

11
5.2 Excellent
The LP strongly satisfies a very likely or most likely intent.
It closely matches the requirements of the query in scope, freshness, authority, market and language.
The LP content is almost exactly what users with a very likely or most likely intent wanted. Users with such an intent could
end their search here and move on.

en-us, {buy books}


o http://barnesandnoble.com/ is Excellent; well-known and credible site in the right market, strongly satisfies the most
likely intent (to buy books online).
o http://www.amazon.com/books-used-books-textbooks/b/?node=283155 is Excellent; for the same reasons.
en-us, {e-cards}
o http://www.americangreetings.com/ecards/index.pd is Excellent; credible LP strongly satisfying the most likely intent.
en-us, {Hayden Harnett Leather}
o http://haydenharnett.com/?action=home is Excellent; while it doesnt directly satisfy the intent, users can click on
tabs on the home page to choose a product line (most of them made from leather); there is no specific leather LP on
this official site, so there is no Perfect LP. LP strongly satisfies the most likely intent to purchase a leather product
from Hayden Harnett.
en-us, {redmond wa home listings}
o http://www.johnlscott.com/propertylist.aspx?GroupID=224950445 is Excellent; credible site exactly addressing what
the user wanted, with options to narrow down their search.
en-us, {msft stock}
o http://finance.yahoo.com/q?d=t&s=MSFT is Excellent; provides current and in-depth related information on an
authoritative source that strongly satisfies the most likely intent.
en-us, {call me by blondie}
o http://www.youtube.com/watch?v=aH3Q_CZy968 is Excellent; watching the video for the song is a very likely intent,
and this video is the most popular one available on YouTube, and is therefore is a strongly satisfying result.
en-us, {abortion rights}
o http://www.prochoice.org is Excellent; provides information supporting abortion rights, strongly satisfying a very
likely intent
o http://www.prolife.com is Excellent; provides information opposing legalized abortion, strongly satisfying another
very likely intent.
en-us, {2013 accord LX Sedan 2.4L 4-cyl 6-speed Manual - fuel economy}
o http://www.edmunds.com/honda/accord/2013/features-specs.html?style=101418836 is Excellent; has a specific
answer to the query for this car model, in the context of a detailed spec page on an authoritative site. The requested
information can easily be found at the top of the page. .
en-us, {darigold milkfat reduced fat milk}
o http://consumer.darigold.com/twopercent-Milkfat-Reduced-Fat-Milk.html is Excellent; one of three official Darigold
web pages for this specific product, a very likely intent.
o http://consumer.darigold.com/2percent-Milkfat-Reduced-Fat-Milk-Gallon.html is Excellent; one of three official
Darigold web pages for this specific product, a very likely intent.
o http://consumer.darigold.com/2percent-Milkfat-Reduced-Fat-Milk.html is Excellent; one of three official Darigold
web pages for this specific product, a very likely intent.
en-us, {the biggest loser}
o http://www.biggestloser.com is Excellent; it is very likely that users would wish to access this site affiliated with the
TV show to find fitness and diet-related information; the most likely intent is for the TV show.
en-us, {banana republic}
o http://bananarepublic.gap.com/browse/division.do?cid=5002 is Excellent; subpage for womens clothing on the
official Banana Republic site; strongly satisfies a very likely intent.
en-us, {guide to great gatsby}
o http://www.sparknotes.com/lit/gatsby is Excellent; well-known and authoritative guide for students organized to
allow users to easily navigate through the content; strongly satisfies most likely intent to find a guide to this book.
en-us, {cancer}
o http://www.webmd.com/cancer/default.htm is Excellent; LP contains many links, but this is expected since cancer is a
broad topic; strongly satisfies most likely intent to find information on an authoritative site.

12
5.3 Good
The LP moderately satisfies a very likely or most likely intent, or strongly satisfies a likely intent.
It may not closely match the requirements of the query in scope, freshness, authority, market or language.
The LP content is useful to many users, but most users wont be completely satisfied and would keep looking.

en-us, {the meaning of NATO }


o http://www.elook.org/dictionary/nato.html is Good; provides a basic definition of the term.
en-us, {another brick in the wall lyrics}
o http://www.lyricsfreak.com/p/pink+floyd/another+brick+in+the+wall+part+ii_20108776.html is Good; detailed
information, but the authority and credibility of the content isnt clear.
en-us, {josh turner}
o http://joshturner.com/josh is Good; artists biography. Narrower than query, but likely to be useful.
en-us, {hockey equipment}
o http://www.ebay.com/sch/i.html?_trksid=p5197.m570.l1313.TR0.TRC0&_nkw=hockey+equipment&_sacat=0&_from=R
40 is Good; auction site with reasonably extensive content for purchasing Hockey Equipment. Not strongly satisfying
as many products are second hand, which not all users will want.
en-us, {can you lead a normal life with epilepsy}
o http://www.fitsugar.com/Epilepsy-2331555 is Good; contains detailed, credible information on epilepsy, diagnosis,
treatments and lifestyle, and covers problems people with epilepsy face, but overall the page is broader in scope than
the query.
en-us, {this pain is just too real song lyrics}
o http://askville.amazon.com/song-Female-singer-lyrics-held-hand-years/AnswerViewer.do?requestId=8766220 is
Good; user posting of the lyrics to "My Immortal" by Evanescence, which contains the query terms (likely intent).
en-us, {number of chips I can drive with cmos op}
o http://en.wikipedia.org/wiki/Fan-out is Good; explains how to calculate the answer.
en-us, {Mapplethorpe images}
o http://www.culturekiosque.com/art/exhibiti/mapplethorpe.html is Good; content is relevant and not explicit.
en-us, {when is nystatin and triamcinolone acetonide cream usp prescribed}
o http://dailymed.nlm.nih.gov/dailymed/archives/fdaDrugInfo.cfm?archiveid=5323 is Good; provides clinical
information on these two creams targeted at medical professionals, not the general consumer (deeper than query).
en-us, {windows xp shuts down upon start-up error blue screen}
o http://www.techsupportforum.com/microsoft-support/windows-xp-support/180803-blue-screen-upon-start-up-stop-
0x0000007b.html is Good; has two brief postings on the problem for the version of Windows the user has asked for,
but is dated (2007) and has unclear credibility.
en-us, {victretomy}
o http://www.tour2india4health.com/vitrectomy-surgery-india.htm is Good; moderately satisfies the most likely
informational intent (assumes misspelled query); market is not important for this query, and the LP content is not
specific to India.
en-us, {palin fake pregnancy}
o http://bojack.org/2008/12/sarah_palins_fake_pregnancy_be.html is Good; content is not very credible, but credibility
is not a high priority for most gossip/rumor queries such as this one.
en-us {how to make sweet potato pie}
o http://www.ehow.com/how_6999_make-sweet-potato.html is Good; the content is appropriate in scope, but this site
is not an authoritative source for recipes.
en-us {cataract surgery}
o http://www.mayoclinic.com/health/cataract-surgery/MY00164 is Good; authoritative site, but content is surprisingly
sparse and only accessible via further links; LP moderately satisfies the most likely intent to find information on this
surgery.

13
5.4 Fair
The LP weakly satisfies a very likely or most likely intent, moderately satisfies a likely intent, or strongly satisfies an
unlikely intent.
It may not match the requirements of the query in scope, freshness, authority, market or language.
The LP is related to the query, but satisfies it so weakly that it is only useful to some users.

en-us, {Seattle}
o http://www.ihg.com/holidayinn/hotels/us/en/reservation/hotelsearchresult?destination=SEATTLE%2CWA is Fair; LP
lists Holiday Inn hotels in Seattle (too narrow).
en-us, {josh turner}
o http://joshturner.com/community is Fair; community subpage on the official site; unlikely intent.
en-us, {amberley character change}
o http://www.renderosity.com/mod/bcs/index.php?ViewProduct=74176 is Fair; describes the Amberley character for
3D rendering software; likely intent, but no content about change.
en-us, {Don Roth Associates}
o http://activerain.com/donroth is Fair; features some content about a Don Roth and the word Associates appears
on the page, but this is not specifically about Don Roth Associates.
en-us, {atlanta's fox 5 news}
o http://www.facebook.com/fox5atlanta is Fair; the station's Facebook page featuring brief updates, weaklysatisfies
the most likely "news" intent.
en-us, {whispering pines and georgetown FL}
o http://www.mapquest.com/maps?address=162++WHISPERING+PINES+RD&city=GEORGETOWN&state=FL&country=US
is Fair; this is a MapQuest site for a specific street location (unlikely intent); more probable user intents are
{Whispering Pines in Georgetown FL} or {junction of Whispering Pines Rd and Georgetown Denver Rd, FL}
en-us, {arquette wedding singer}
o http://www.nicksflickpicks.com/wsinger.html is Fair; LP is a movie review that only briefly mentions Alexis Arquette.
en-us, {celebrity pictures}
o http://celebpic.blogspot.com is Fair; this LP is text heavy and not solely focused on showcasing celebrity photos,
which weakly satisfies the most likely intent to see celebrity pictures.
en-us, {ebook web page}
o http://www.textbooks.com/Catalog/HO6/Web-Page-Management.php?&showAll=true&PG=2 is Fair; has limited
range of e-books on web page design.
en-us, {Disney channel}
o http://disney.go.com/games is Fair; games sub-page of official site (unlikely intent for this specific query; some users
would find this LP useful). The user has specified channel so would be more interested in content relating to shows.
en-us, {microsoft}
o http://microsoftsucks.org/ is Fair; this LP is a Microsoft complaint forum (unlikely intent).
en-us, {make a cake}
o http://chowhound.chow.com/topics/407976 is Fair; has one recipe for yellow cake (much narrower than the query).
en-us, {lg LGE960W reviews}
o http://www.lg.com/us/cell-phones/lg-LGE960W-nexus-4-white is Fair; official web page for this cellphone, but
content would only weakly satisfy users for this query.
en-us, {sextube.com}
o http://www.alexa.com/siteinfo/sextube.com is Fair; unlikely intent; note that while the LP contains a link to the adult-
oriented site in the query, the LP is focused on providing information on the sites traffic ranking.
zh-hk, {mad on you joey}
o http://lyric.coms.hk/thread-155-1-1.html is Fair; in Traditional Chinese. A likely intent is for {the lyrics to the song
mad about you by Joey Yung}; although the song has an English title, its sung in Cantonese, but users may want to
verify the lyrics.

14
5.5 Bad
The LP weakly satisfies a likely or unlikely intent, moderately satisfies an unlikely intent, poorly satisfies any intent, or
satisfies a very unlikely intent.
It may completely miss the requirements of the query in scope, freshness, authority, market or language.
The LP has so little relation to the query that it will be useful to almost no users.
A Bad rating is also warranted where the LP:
misses something that the user specifically asked for;
adds no value beyond telling the user that the URL were using is no longer correct, and that they should click a link to
visit the current location of the content.
uses spam techniques, or is a parked domain (details in Section 7.4);
misleadingly provides content from other sites (details in Section 00);
tries to affect the users computer in an unwanted way, for example by asking users to install an unknown special
program to view or download the content; these may contain malware.

en-us, {redmond wa apartments}


o http://www.redmondrents.com/ is Bad; LP is about apartments in Redmond OR, rather than Redmond WA.
en-us, {boston .com}
o http://www.bostonbyfoot.org/maps/ is Bad; Boston.com is a very popular site, so a page featuring information on
exploring Boston on foot poorly satisfies the query.
en-us, {2005 slk 55 0-60}
o http://www.slkworld.com/slk-r171-general-discussion/8775-2006-slk55-0-60-time.html is Bad; forum discussion of
the 0-60 time of a 2006 SLK55, but no useful content on the 2005 model that the user specifically asked for.
en-us, {shaun niven}
o http://www.linkedin.com/pub/sean-niven/15/957/632 is Bad; given the spelling of the name in the query and that
this person is based in South Africa, it is very unlikely that a user is looking for information on a Sean Niven.
en-us, {motelsaround pittsburgh}
o http://local.yahoo.com/results?p=Penn+Brewery&sortby=adistance&csz=Pittsburgh%2C+PA&cityfilt=pittsburgh+pa is
Bad; lists matches for {Penn Brewery} near Pittsburgh PA; no relevance to a search for motels.
en-us, {1226 s darlington ave tulsa ok}
o http://www.movingideas.org/tulsa-ok-local-movers/ is Bad; the query is for an address, while the LP contains
information on a moving company at a different address starting with 1226.
en-us, {irs code 121 irrevocable trust}
o http://www.taxalmanac.org/index.php/Internal_Revenue_Code:Sec._645._Certain_revocable_trusts_treated_as_part_
of_estate is Bad; describes a different section of the code (Sec. 645), and has no useful content specific to the intent.
en-us, {large losses in 401(k)}
o http://articles.sfgate.com/2004-09-19/business/17445487_1_metrocall-holdings-messaging-wireless-e-mail-stock-
market-investments is Bad; information is very specific to a particular case (too narrow) and dated (2004).
en-us, {eaton powerware 5110 ups 220-240 vac price}
o http://computers.pricegrabber.com/upss/Powerware-5110-Model-500-VA/m24409229.html is Bad; correct model,
but voltage is probably 120VAC since the LP targets US users. (Note that 500 VA specified power rating, not voltage).
en-us, {heavy equipments for sale in poland}
o http://www.owenmachinery.com/dealer_inventery.php?ID=768&INVID=ELBEX%20P.H.U. is Bad; LP is a search result
page saying no relevant items were found (it doesn't have what the user was looking for); also no reference to Poland.
en-us, {nba playoffs}
o http://www.nba.com/magic/08playoffs_rd1gm5splash.html is Bad; this is an old (2008) entry page. Clicking on the
link for tickets leads to the message: Sorry, but this event no longer exists in our database.
en-us, {sofitel beds}
o http://www.sofitel.com/gb/hotel-6308-sofitel-philippine-plaza-manila/room.shtml is Bad; the LP describes bed
configuration for the Sofitel Hotel in Manila, which is very unlikely to be the intent of users entering this query.
en-gb, {national parks}
o http://www.costarica-nationalparks.com is Bad; its very unlikely that UK users have Costa Rica in mind for this query.
en-gb, {connex timetable}
o http://www.hartley-kent.org.uk/news/news_2003.htm is Bad; LP is old and doesnt actually contain a timetable.
fr-fr, {Jeux d'ordinateur} (computer games) Formatted: French (France)
o http://www.publispain.com/juegos.htm is Bad; Spanish LP not useful to most users.

15
5.6 Detrimental
Some LPs have explicit or graphic content whether text, images, or video, in the main content or in ads that is
inappropriate for a general audience. Search engines try to hide such results by default, and many schools, libraries, and
workplaces rely on this filtering (known as SafeSearch) to block inappropriate content. By marking an LP as Detrimental
you are indicating that it should be not be visible to users unless they deliberately turn this filtering off.
Detrimental content typically falls into one of the following categories:
o is intended to cause sexual excitement (adult material)
o depicts or promotes serious criminal activity (e.g. sex crimes or hate crimes, violence, or illegal narcotics)
o is otherwise shocking or offensive to most users (e.g. extreme foul language).
One way to distinguish Detrimental content is to consider whether the content on the LP might ever appear in the
imagery, dialog, or music of a prime-time television, satellite television, or radio show with a large audience of users in your
market. Note that in many markets restricted content is allowed on television outside prime time (e.g. 10pm onwards),
while some satellite and cable channels do not restrict their programming at all.
Content that has reasonable artistic, literary, comedic, or scientific value is not Detrimental. For example, LPs depicting
nude Renaissance paintings, describing prostitution or sex in a factual and non-graphic manner, scientific articles about
breast implants, historical documents related to hate speech, or a comedy show about drug users, are not detrimental.
Spam or Foreign-language LPs with detrimental content should always be marked as Detrimental.
If the adult content is not a focus of the page such that most users would not notice or be offended, then rate for relevance.

en-us {videos}
o http://www.dailymotion.com/us is Excellent; Dailymotion.com is a very popular website where users can find a wide
variety of online videos. This website contains a link to Adult content, labeled as sexy at the bottom of the page.
Though a user can access adult content through this link, we would not rate this LP Detrimental because Dailymotion is
a popular site where the number of users looking for general content far outweighs the number of users looking for
adult content; clicking on the sexy link at the bottom of the page also brings up a further warning indicating that the
family filter is on. Note that an explicit video posting on this website would be Detrimental.
en-us {jenna jameson}
o http://en.wikipedia.org/wiki/Jenna_jameson is Good; satisfies a likely intent to learn all about the porn star Jenna
Jameson. While the LP contains a link to Jamesons official site, which contains pornographic content, LP is
informational and on a well-known site and so would not be shocking to a large audience of users in the US. This LP
should therefore be rated normally for relevance.
de-de {teamwork beim sex}
o http://www.menshealth.de/love/sex-themen/das-orgasmus-dossier.29232.htm is Good; moderately satisfies the
most likely intent. While the sidebar contains links to explicit images on this site, LP contains factual information from
the mainstream magazine Mens Health. An LP with explicit images on this site would be Detrimental.

5.6.1 Adult Queries on non-Adult rosters Formatted: French (France)


Decide on the basis of the LP content, not the query; some queries with possible adult intent will turn up many adult results, Formatted: Font: Bold, French (France)
but only LPs with adult content as described above will be rated Detrimental:

en-us {capri styles}


o [!]http://www.keezmovies.com/kapri-styles is Detrimental; strongly satisfies the most likely intent, but contains adult-
only images.
o http://www1.macys.com/shop/womens-clothing/womens-capri-pants?id=44717 is Fair; strongly satisfies an unlikely
intent to shop for capri pants.
en-us {sex on the beach}
o http://en.wikipedia.org/wiki/Sex_on_the_Beach_(cocktail) is Good; moderately satisfies a very likely intent to find a
recipe for this cocktail drink.

5.6.2 Child pornography


If a page displays content that is possibly child pornography, please immediately clear your browser cache and cookies and
report the query and the URL using the proper escalation process recommended by your manager. Each vendor company
will compile these reports and escalate to Microsoft, who will contact law enforcement agency as required.
Break up the URL in your email and ensure it is not a clickable link, e.g. change:
http://blah.com (intact URL, clickable link) to http://blah. com (URL not intact, not a clickable link).

16
en-us, {penis anatomy}
o http://catalog.nucleusinc.com/generateexhibit.php?ID=6619 is
Fair; although the landing page features an illustration of a
sexual organ it is not intending to cause sexual excitement, it is
not depicting sexual conduct and it has scientific/medical value.

en-us, {breastfeeding monkeys}


o http://www.amazon-tribes.com/Indigenous-Woman-Nursing-
Monkey.html is Fair; although the LP features nudity, it has
scientific value, and is not trying to cause sexual excitement or
to depict sexual conduct.

en-us, {famous Goya paintings}


o http://en.wikipedia.org/wiki/La_maja_desnuda is Good;
although the landing page features an image of a nude woman
in a provocative position it has significant artistic value.

17
en-us, {body painting}
o http://www.visual-arts-cork.com/painting/body.htm is Good;
although the LP features nudity, its primary intent is not to
cause sexual excitement, and there is also some artistic value.

en-us, {what is a shress?}


o http://www.thefashionpolice.net/2008/05/shresses-by-
kos.html is Good; although the LP features nudity, its primary
intent is not to cause sexual excitement, and the content is not
offensive.

en-us, {holly valance}


o http://www.holytaco.com/holly-valance is Fair; although the
LP features partial nudity, its sole purpose is not to cause sexual
excitement, and it is not graphic or offensive enough to be rated
detrimental.

en-us, {anime babes images}


o http://photobucket.com/images/anime%20babes/?page=1is
Excellent; although the LP features partial nudity and some
sexual conduct, it is not sufficiently graphic to warrant a
detrimental rating.

18
en-us, {heidi staley}
o http://www.imdb.com/name/nm0821648/ is Good; lists films in which this adult actress has participated. The names
of the movies are sexually suggestive, but not Detrimental.
en-us, {how to get pot}
o http://chucklesthesane.tripod.com/psguide/procure.htm is Good; legally questionable, but relevant to the intent.
en-us, {child prostitution}
o http://www.childrenofthenight.org is Excellent; not adult-only content or child pornography, so not Detrimental.
en-us, {yanomami indians}
o http://www.amazon-indians.org/yanomami.html is Good; the nudity is not sexual in nature, so not Detrimental.
en-us, {anarchist cookbook}
o http://www.scribd.com/doc/387846/The-Anarchist-Cook-Book is Good; complete copy of the book online; even
though the much of the content is about breaking the law, its not so shocking that it warrants a detrimental rating.
en-us, {obama}
o [!]http://www.stormfront.org/forum/showthread.php?t=669235 is Detrimental; LP contains racial hate speech.
en-us, {car crash}
o [!]http://www.bestgore.com/wp-content/uploads/2008/12/nikki-catsouras-porsche-girl-car-accident-07.jpg is
Detrimental; has a very graphic image of a person killed in a car crash.
en-us, {make meth}
o [!]http://www.textfiles.com/uploads/methmethod.txt is Detrimental; LP provides instructions for making illegal
narcotics.
en-us, {hot reality TV}
o [!]http://www.realitypassplus.com is Detrimental; LP features adult-only content.
en-us, {adults}
o [!]http://freexxxadultmovies.blogspot.com is Detrimental; has adult-only content.
en-us, {girls riding motorbikes}
o [!]http://www.xcylite.org/gallery2/v/Easyriders2010/ is Detrimental; has adult-only content.
en-us, {Saudi yotobe}
o [!]http://members.multimania.co.uk/ynebiqyco is Detrimental; the page is full of links with adult themes.
en-us, {christina hendricks}
o [!]http://idly.craveonline.com/2012/03/christina-hendricks-leaked-phone-pics-gallery is Detrimental; some of the
thumbnails link to detrimental images.
en-us, {tattoos ideas in sex lines}
o [!]http://www.bullz-eye.com/gallery/tattoos.htm is Detrimental; has adult-only content (photographs, links, and ads).
en-us, {blink 182 family reunion}
o [!]http://www.azlyrics.com/lyrics/blink182/familyreunion.html is Detrimental; the lyrics of this song contain
extremely foul language.
en-us, {anime babes images}
o [!]http://onlybestsex.com/tgp/anime.html is Detrimental; contains adult-only images.
o [!]http://www.anime.100topsites.net/ is Detrimental; also contains adult-only images.

19
5.7 No Judgment: Login
You cannot determine the relevance of the LP because the LP appears relevant to the query, but the content is hidden
behind a login screen (see Section 7.3). If it is clear that the LP is irrelevant to the query, even if the LP is a login page, then
rate Bad.

en-us, {Freshcope rewards}


o http://www.freshcope.com is Login; cannot determine relevance to query without registering and logging in.
en-us, {volta download}
o https://www.motu.com/login_form?came_from=http://www.motu.com/download is Login; without registering and
logging in, we cannot determine whether the specified software is available for download from this page.
en-us, {wells fargo David Kowach}
o http://online.wsj.com/article/SB10001424127887323551004578117432081993580.html is Login; cannot determine
relevance to query without registering and logging in.
en-us, {att prepaid account}
o https://www.paygonline.com/websc/index.jspis Perfect; we can determine the relevance of this LP without needing
to log in.
en-gb, {Isaac hayes}
o http://pro.imdb.com/name/nm0005002 is Fair; the LP appears in the background behind a registration message. We
can determine that this will be relevant to the query. However, since t is unlikely that the average user will have an
IMDB Pro account or be interested in paying to register for one, this LP will be useful to only some users.
en-us, {facebook tony lo galbo}
o http://www.facebook.com/login.php is Bad; this is the official login page for Facebook but does not satisfy the intent.
zh-cn, {}
o http://www.tudou.com/programs/view/rNwWqHShFGk/ is Bad; it is not possible to view this video without a private
password provided by the uploader. This would therefore be useful to almost no users and should be rated Bad.

5.8 No Judgment: Foreign


Use this rating when there appears to be a connection between the query and the LP, but you cannot determine the
relevance of the LP because the content is in a language you dont understand (see Section 7.8). If you can tell that the LP
bears no relevance to the query, even if it is in a foreign language, then rate Bad.

en-us {al-eghtesadia}
o http://www.aleqt.com/ is Perfect; definitive Arabic LP (Foreign if you cant rate this).
en-gb, {Justin bieber Brazil fan page}
o http://www.biebermania.com.br/ is Good; Brazilian fan site in Portuguese (Foreign if you cant rate this).
en-au, {Goin Down}
o http://m.weblio.jp/c/Goin'_Down is Bad; it is very unlikely that an Australian user would be seeking this Japanese
language content.
en-in, {nokia x7}
o http://www.nokia.com/vn-vi/ho-tro/ho-tro-san-pham/x7-00/ is Bad; even though this LP is on an official Nokia site,
this content in Vietnamese would be useful to almost no users in India.

20
5.9 No Judgment: Page Did Not Load (PDNL)
You cannot determine the relevance of the LP because the usual content of the LP is unavailable for a likely temporary
reason; you will find some examples below.
The LP may not load at all, or you may be automatically redirected to a search page (this is controlled by the configuration
settings on your browser and the policies of your Internet Service provider).
The LP may display a message indicating the content is unavailable. Web servers and webmasters use a very wide range of
messages to notify users of such problems.
Some LPs may have an under construction message while still being usable; this usually indicates that the site owner is
hoping to add more content. Since you can view the existing content, you should rate it (do not mark as Page Did Not Load).
Before assigning this rating, try to reload the LP by clicking on the refresh icon in the Bing Toolbar (Shortcut key: Alt-W).
If the expected content is unavailable and you are certain you will see the same message if you revisit after one month (for
example, the company has gone out of business, or a video has been removed), then you should award a Bad rating.
Common error codes include 401, 403, 404, 500, 502, 503, and 504 errors; click to read more about http error codes.
The examples in this section would all get a Page Did Not Load rating.

5.9.1 Domain lookup failure


http://sitethatdoesnotexist.com

5.9.2 Search page on domain lookup failure, returned by the users ISP.
http://search3.comcast.com/?cat=dnsr&con=ds&url=www.zgwholesale.com

5.9.3 No error message, just a blank page

21
5.9.4 LP hacked
http://thomas.loc.gov; usual content (left), and hacked page (right):

5.9.5 Access denied

22
5.9.6 Requested page cannot be found
Note: clicking on the URLs below from within MS Word may fail and simply show an error dialog box; you should paste this URL into
the browser location bar to view the LP.

http://cpanel.net/blah.html; standard not found error message


from the web server:

http://abc.go.com/blah.html; friendly message

http://slashdot.org/blah.html;
another friendly message

http://kids.pbskids.com/go/games.html; yet
another friendly message

www.primalwear.com/p-2392-custom-kit-leg-warmer-size-md-only.aspx;
product not found error message

http://www.daum.net/7437.php; the content


of this URL is not available and redirects to the
URL
http://status.daum.net/error/error404.html;
even though the content is in Korean, its
obvious from the URL that this is a 404 error
page.

23
5.9.7 Temporary server error or application crash
Web server error message

Application crash error message:

Application crash error message:

Friendly server error message:

5.9.8 LP down for maintenance or is under construction

24
6 Query Considerations
6.1 Misspellings, Punctuation, Capitalization, and Advanced Search Operators
Many queries contain misspellings, or colloquial or non-standard abbreviations; you need to identify the likely intent.
Users often ignore punctuation and capitalization when typing in queries, so lack of punctuation or consistent letter casing
(all lowercase or all uppercase) should not play a significant role when determining intent; however, a user who has taken
care to include punctuation or capitalize parts of the query is likely to be indicating their intent.
Search engines allow users to enclose terms in quotes to ensure that these terms appear together in the LP; LPs that do not
contain these words exactly as specified should be rated by relevance to the users intent and penalized by one rating level.
Pay attention to queries that include advanced search operators such as OR, AND, site:, define:, logical grouping
using parentheses, phrase searches using quotations. You can find a list of common operators in the glossary.
en-us, {all state}
o Most likely intent: allstate insurance (misspelling)
en-us, {gyco}
o Very likely intent: {geico} (misspelling)
o Likely intent: gyco
en-us, {amtrack}
o Most likely intent: Amtrak (misspelling)
en-us, {medical forms}
o Most likely intent: forms used in the medical profession
o Likely intent: administrative forms used by Medi-Cal (missing punctuation).
en-us, {MediCal forms}
o Most likely intent: administrative forms used by Medi-Cal (missing punctuation).
o Likely intent: forms used in the medical profession
en-us, {yspace}
o Most likely intent: MySpace (misspelling)
o Likely intent: ySpace
en-us, {jean hackman}
o Most likely intent: Gene Hackman (misspelling)
o Likely intent: Jean Hackman
en-us, {decomposing corps}
o Most likely intent: decomposing corpse (misspelling)
en-us, {Startek CO}
o Most likely intent: Startek Colorado (abbreviation)
o Likely intent: Startek Corporation (incorrect capitalization of O)
en-us, {Startek co.}
o Most likely intent: Startek Corporation
o Likely intent: Startek Colorado (abbreviation, extra punctuation)
en-au, {dangerous crocs}
o Most likely intent: safety problems with Crocs shoes.
o Likely intent: safety issues with crocodiles (extra punctuation).
en-us, {DOR}
o Most likely intent: Department of Revenue (abbreviation)
en-us, {dor}
o Very likely intent: dor
o Unlikely intent: Department of Revenue (capitalization, punctuation)
en-us, {poem my name is}
o The user wants a poem with the exact phrase my name is; anything else is very unlikely to satisfy their intent.
en-us, {candid camera and americas funniest home videos }
o The user is indicating they want information on both these shows.
en-us, {(install kubuntu) or (download kubuntu)}
o The user is indicating they would be equally happy with an LP on either of these topics.
en-us, {system()}
o Most likely intent: the system() programming language function (the parentheses make other intents very unlikely).
en-us, {#include}
o Most likely intent: the #include compiler directive used in programming languages (the # makes other intents very
unlikely).
25
6.2 URL Queries
Some queries are in the form of a complete URL (like http://www.msn.com) or a URL fragment ( like msn.com); such queries
contain a URL component such as http, https, mailto, ftp, site, www, com, edu, org, .tv, or .html (this list is not exhaustive).
As with any query, you should determine the users intent:
o Consider first if the query, as written, leads to an actual LP; if it does, evaluate whether this is the most likely intent.
o If not, try to correct for obvious mistakes by eliminating spaces, changing commas to periods, fixing misspellings
etc. to see if the corrected version resolves to an actual LP.
o Allow for the fact that users often append a .com to their query terms to mean the web site.

en-us, {www.maacys.com}
o http://www.macys.com is Perfect; strongly satisfies the navigational aspect of the likely intent, assuming that the
query was misspelled.
en-us, {www.badongo.com}
o http://www.badongo.com/ is Perfect; strongly satisfies the navigational aspect of the most likely intent.
o http://download.cnet.com/Badongo-Buddy/3000-2648_4-10544768.html is Fair; popular application for this site.
o http://support.badongo.com/index.php?_m=knowledgebase&_a=view is Fair; this is part of the support section of the
official web site; it has some relevant content, but weakly satisfies the most likely intent to navigate to the homepage.
en-us, {irs.com}
o http://www.irs.gov is Perfect; this LP strongly satisfies the most likely intent to navigate to the official homepage of
the Internal Revenue Service.
o http://irs.com is Excellent; strongly satisfies the very likely intent to navigate to this URL, an exact match to the query.
o http://www.banks.com/taxes/ is Excellent; also strongly satisfies the very likely intent to navigate to the query URL,
since http://irs.com redirects here.
en-us, {utube.com}
o http://youtube.com is Perfect; strongly satisfies the most likely intent to navigate to YouTube, assuming that the
query was misspelled.
o http://www.utube.com is Bad; even though its an exact match, this site has no useful content and is essentially a
parked domain and so can be considered a very unlikely intent (See Section 7.4).
en-us, {goldenretrievers.com}
o http://goldenretrievers.com/ is Page Did Not Load; even though its an exact match, this page doesnt load.
o http://goldenretrievers.org is Excellent; strongly satisfies the very likely intent to navigate to this particular URL.
However this isnt an exact match to the query URL and the LP is focused on a narrow geographic area (North Texas);
consequently we cant be confident that this URL represents the most likely intent.
en-us, {latimes/jobs}
o http://latimes.com/jobs is Perfect; strongly satisfies the most likely intent to navigate to the Jobs subpage of the LA
Times website, assuming that the query is a URL fragment; we could also interpret the query as being a misspelling of
{la times jobs}, but the intent remains the same.
en-us, {worl com}
o http://world.com is Bad; interpreting this as a URL fragment query leads us to this site, which has no useful content
and is essentially a parked domain (See Section 7.4). The most likely intent is {worldcom}.
en-us, {sararose.com}
o http://sararose.com is Bad; exact match, but this appears to be a parked domain and therefore a very unlikely intent.
o http://sarahrose.com is Excellent; strongly satisfies the very likely intent to navigate to this particular site, but we
cant be sure its the most likely intent since its not an exact match to the query URL.
o http://sararose.tumblr.com is Bad; this is a personal blog, very unlikely to be the users intent for this particular query.
en-us, {bestbuy.com}
o http://bestbuy.com is Perfect; strongly satisfies the most likely intent to navigate to the homepage of Best Buy..
o http://www.bestbuy.com/site/olspage.jsp?id=cat12090&type=page is Good; strongly satisfies a likely intent to use
the Best Buy website to find the nearest store.
en-us, {bethel4success.net}
o http://www.bethel4success.net is Perfect; this direct match to the query strongly satisfies the most likely intent to
navigate to this exact URL.
o https://www.bethel4success.com/DesktopDefault.aspx?showlogin=1 is Perfect; also strongly satisfies the most likely
intent to navigate to the query URL, since the query URL redirects here.
o http://www.bethelsuccess.net is Bad; poorly satisfies the most likely intent to navigate to the homepage of the Bethel
Success Program; this is the homepage of Bethel University, but there is no link to the success program or any
information about it on this LP.

26
6.3 Location-specific queries
A user who specifies a location in their query is specifically looking for results related to that location.
If the location in the LP does not appear to match the query, look it up on a map to see whether they are close.
Even users who do not specify a location are likely to prefer results geographically close to them. The Bing team will
sometimes note the users location in brackets along with the query; when you see this information, please consider the
intent from the perspective of users in that location.

en-us, {jobs in tampa fl}


o http://www.indeed.com/l-Tampa,-FL-jobs.html is Excellent; well-known site with many job listings for Tampa FL.
Strongly satisfies the most likely intent to find jobs in Tampa FL.
o http://jobsearch.monster.com/PowerSearch.aspx?where=orlando is Bad; lists jobs in Orlando FL.
en-us, {rental houses,o'day,rock hill,mo}
o http://www.realtor.com/realestateandhomes-search/Rock-Hill_MO/type-rentals is Bad; from research we find that
there is an ODay Road in the Rock Hill area of Missouri. The most likely user intent is to find rental property on this
road; since the LP lists rental property in the area, but none (at the time of writing) on ODay Road.
en-us, {hispanic chamber of comerce glendale az}
o http://www.azhcc.com is Perfect; although this is the Phoenix office, we see from the map (http://www.bing-vs-
google.com/?q=glendale+az) that Glendale is a suburb of Phoenix. There is no Hispanic Chamber of Commerce in
Glendale itself, and so the Phoenix office is the most likely intent.
o http://www.ethicalaz.com/arizonahispanicchamberofcommerce is Good; this LP strongly satisfies the likely intent to
find information on this chamber by providing some facts about the organization, in addition to noting contact
information (the LP is useful to many users).
en-us, {east bank imaging new orleans}
o http://proscan.com/fw/main/Louisiana-New-Orleans-1297.html is Perfect; from the map (http://www.bing-vs-
google.com/?q=metaire+la) we see that Metaire is a district of New Orleans, and therefore this is the most likely intent.
en-us, {[Seattle WA] library} (the Bing team has noted this query as coming from a user in Seattle WA)
o http://www.kcls.org is Good; although the King County Library System does not contain branches in Seattle itself,
this LP strongly satisfies a likely intent to find out about libraries in the greater Seattle area.
o http://en.wikipedia.org/wiki/Library is Good; this detailed LP strongly satisfies a likely informational intent.
o http://www.lapl.org is Bad; local library system for Los Angeles, very unlikely intent of Seattle users for this query.
en-us, {[New York NY] traffic} (the Bing team has noted this query as coming from a user in New York NY)
o http://www.traffic.com/New-York-Traffic/New-York-Traffic-Reports.html is Excellent; strongly satisfies the most likely
intent for New York traffic updates.
o http://www.imdb.com/title/tt0181865/ is Fair; LP about the movie Traffic released in 2000 (an unlikely intent).
o http://www.dallasnews.com/traffic/ is Bad; very unlikely that a user in New York is interested in Dallas traffic.
en-us, {[New York NY] outback steakhouse} (the Bing team has noted this query as coming from a user in New York NY)
o http://www.outback.com is Perfect; definitive LP, most likely intent.
o http://hosted.where2getit.com/outback/index.html?form=locator_search&addressline=ny%20ny is Good; LP strongly
satisfies the likely intent to find information on Outback Steakhouse locations in New York.
o http://hosted.where2getit.com/outback/state2.html?state=WA is Bad; Washington State locations, very unlikely to
be useful for a user in New York.
en-us, {pizza}
o http://pizza.com is Good; this site estimates your ZIP code and presents a simple list of pizza vendors in your area. Of
course, most users would not have the same ZIP code as you, but the behavior would be generally correct for each user.
Strongly satisfies the most likely intent to find nearby pizza locations. While the LP may not satisfy the user in terms of
scope (the results do not represent an exhaustive list of pizza restaurants in locations), it does return an adequate
number of listings.

27
6.4 Personal Names
Intent likelihood:
o Many queries specify a persons name; when the name matches a famous person, e.g. {Michael Jackson} or {jimeoin}, it
is most likely that users are searching for the celebrity, rather than a non-famous person with this name.
o When the name matches people who are not famous, you may be unable to determine the most likely intent; there
may be many different people called Fred McMaster.
o The likelihood that the user is looking for a specific non-famous person will depend on how common their name is; for
example, the more people you find with this name, the less likely the query intent for any one of them.
Name variants:
o LPs that contain the name (including variations in name order, such as McMaster, Fred) should be rated according to
the quality and detail of the content and the likelihood that the data will be updated; home pages, blogs, or profiles on
social networking sites, are more specific and likely to contain fresh data about the person. Queries and LPs often use
nicknames or misspellings:
LPs on William Henry Gates III are relevant to the query {Bill Gates}.
For the query {furtatdo}, the most likely intent is {Nelly Furtado}, although {furtatdo} is still a likely intent.
o Users often do not know or remember middle names, initials, or name suffixes, and many people dont include these on
their home pages, so the absence of these on the query or the LP should not heavily influence your rating.
o If both the query and the LP include details, and these are mismatched, the LP probably does not belong to the person
the user wants. For example, an LP belonging to John D. Smith is very unlikely to satisfy a user looking for {John R.
Smith}.
o LPs with only part of the name, e.g. "McMaster" without "Fred" next to it, and clearly not related to the person with the
correct name, should be rated as Bad.
Social networks:
o Profile LPs on social networking sites should not be judged solely by the number of people they cover, or even by
whether they contain detailed profiles; many social network users hide their full profiles from the public this does not
make the result less satisfying, as it would be expected by a user seeking this type of result.

en-us, {Fred McMaster}


o https://www.facebook.com/fhmcmaster is Good; LP owned by a Fred McMaster.

o http://www.linkedin.com/pub/fred-l-
mcmaster/19/31a/6a1 is Good; note that
the user didnt specify a middle initial.

o http://www.mylife.com/people-search/fred-mcmaster is Fair; LP includes


references to several people called Fred McMaster, but clicking on the links brings
up pages with very little specific or useful information.
Formatted: Indent: Left: 0", First line: 0"
en-us, {Jeff Rice}
o http://www.linkedin.com/pub/dir/Jeff/Rice is Good; LP lists LinkedIn profiles with this name, allowing the user to
identify the particular Jeff Rice theyre interested in.
o http://uk.linkedin.com/pub/jeff-rice/13/6b9/2b2 is Fair; this is a relatively common name, and there is no famous
person with this name. The LinkedIn profile for each individual person satisfies an unlikely intent. Note that although
this particular Jeff Rice is in the UK, the query is broad enough to include persons with this name outside the US.

28
6.5 Commerce Queries
You will encounter queries where the user is looking to buy and/or find information on a certain product. Dont assume that
shopping aggregators will be the intended stopping point for most users. In general, users with commerce queries are
looking for the search engine to order the most authoritative pages that feature the item (i.e, specific pages) at the top.
Some queries will be specific, where the user is looking for one particular item. Example queries include {fi-6140z-fujitsu
scanner}, {nike air max 2013 premium}, {nokia lumia 928}. The page on the official website that covers the particular
product will likely be the best result, but this is not always the case.
Depending on the query, some aggregators will be more trusted or popular than others.
Even if the query is broad, carefully consider which kinds of pages will be most satisfying.

en-us, {baby einstein}


o http://www.babyeinstein.com/home/ is Perfect; official homepage for Baby Einstein.
o http://www.amazon.com/s?ie=UTF8&page=1&rh=i%3Aaps%2Ck%3Ababy%20einstein is Good; good selection of
products which could be a useful starting point for some users.
o http://www.ebay.com/sch/i.html?_nkw=baby+einstein is Good; selection focuses on Baby Einstein DVDs which could
be a useful starting point for some users.
en-us, {Barbie photo fashion doll}
o http://shop.mattel.com/product/index.jsp?productId=13334037 is Perfect; official shopping page for this product
that also provides detailed information and customer reviews.
o http://www.amazon.com/Mattel-X7738-Barbie-Photo-Fashion/dp/B006O6EIEI is Excellent; specific page on Amazons
shopping portal that offers the doll for sale, with detailed information and customor reviews.

o http://www.barbie.com/photo-fashion-doll/ is Good; official page on this Barbie, but doesnt offer an option to buy.
o http://www.ebay.com/sch/i.html?_nkw=barbie+fashion+photo+doll is Good; offers this specific Barbie for sale.
o http://www.amazon.com/s?ie=UTF8&keywords=barbie%20fashion%20photo%20doll&page=1&rh=i%3Aaps%2Ck%3Ab
arbie%20fashion%20photo%20doll is Fair; one one of the products listed here is the one the user is looking for.
en-us, {nike shoes}
o http://store.nike.com/us/en_us/pw/shoes/brk is Perfect; devoted to shoes on Nikes official shopping portal.
o http://www.footlocker.com/nike-shoes/ is Excellent; well-known retailer for sports shoes, and this is the subpage
dedicated to Nike shoes.
o http://ww.nike.com is Good; official HP of Nike which could be a useful starting point for some users.
o http://www.amazon.com/s?ie=UTF8&page=1&rh=n%3A672123011%2Cp_4%3ANike is Good; most users wouldnt be
completely satisfied and would keep looking.
o http://www.ebay.com/sch/i.html?_nkw=nike+shoes is Good; most users would would keep looking.
en-us, {shower stalls}
o http://www.us.kohler.com/us/catalog/categoryList.jsp?id=431250&parentCategoryId=430957 is Excellent; well-
known retailer for bathroom supplies, and this is the subpage focused on shower stalls.
o http://www.nextag.com/Shower-Stall/products-html is Good; provides useful content, but more users would prefer
shopping from a site known for its focus on quality bathroom merchandise.
o http://www.sears.com/search=shower+stalls?vName=Tools&autoRedirect=true&viewItems=50&redirectType=CAT_RE
C_PRED is Good; provides some information on available bathroom stalls, but also includes other items.
o http://www.amazon.com/s?ie=UTF8&keywords=shower%20stalls&page=1&rh=i%3Aaps%2Ck%3Ashower%20stalls is
Good; provides some information on available bathroom stalls, but also includes other items.
o http://shower-stalls.com/ is Fair; only weakly satisfies the user intent; this is a site of unknown authority with little
content of any substance.
29
7 Landing Page Considerations
7.1 General tips
If the original URL that youve been asked to rate automatically redirects to a different URL, even after a delay, rate the final
LP that you see. The toolbar will alert you when the LP URL is different from the original URL.
If the LP simply informs users that the content has moved and that they should click to visit the new location, rate the LP as
Bad (do not click to go to the new location). If the LP does have additional useful content, rate by relevance.
If the LP URL points to a file such as a Word document, Excel spreadsheet, or PDF file, you should download, open, and rate
the file in the same way as a web page.
Always click to check content thats an important component of the query or LP; this is often the only way to see whether a
video is still available, or a download link is not broken.
Take care not to run programs that may be downloaded during rating (these may contain malware).

If your browser warns you that the web site certificate is


invalid, continue past the warning to view and rate the LP. If
you see this warning for a query where authority is important
(for example on a banking or email site), take additional care
when determining the authority of the LP.

en-us, {vegaburm}
o http://vegaburm.tripod.com is Bad; LP adds no value
beyond telling users that the content has moved.

en-us, {wamu.com}
o https://www.chase.com/wamuwelcome3/ is Perfect;
although this LP asks users to click to visit the content at
a different URL, its actually adding value by telling users
looking for wamu.com about the takeover. As such, this
is the best result for a user looking for wamu.com.

30
7.2 Splash Pages and Entry Pages
Many web sites show an introduction page that the user must pass before viewing the main content.
o Splash pages are designed to get visitors excited about viewing the content of the page, often by showing an
interesting animation or video sequence; users can often bypass this intro by clicking on it or on a skip intro
link. You should continue past the splash page and rate the main LP.
o Entry pages ask users to provide some information on their background or interests, for example their language,
market, ZIP code, age, or preferred topic, before proceeding to the end LP content; you should provide this
information and rate the page to which you are redirected. Many sites use a cookie to store the users preferences
and so do not show the entry page on subsequent visits.
The rating for the splash or entry page is the same as the rating for the main LP. In rare cases you may find that the splash
or entry page is confusing or out of date, and therefore merits a lower rating.
An LP that requires users to register to view the desired content is not an entry or splash page; see Section 7.37.3 for details.

en-us, {Lufthansa}
o http://lufthansa.com is Perfect; you may see the German-language or English-language entry page, where the user
can select their language and continue on to the main LP; the site stores a cookie to record the users selection, so
future visits bypass the entry page.

31
en-us, {rubber association of canada }
o http://www.rubberassociation.ca/rubberroom/Welcome.html is Perfect; splash page, allows user to click to the
required content.

it-it, { Air France}


o http://airfrance.it is Perfect; Italian site.
o http://airfrance.com is Perfect; site entry page, allows user to select country and redirects to the Italian site.
zh-hk, {} (Hong Kong Observatory)
o http://www.weather.gov.hk/contentc.htm is Perfect; official homepage for this observatory.
o http://www.weather.gov.hk/ is Perfect; this is an entry page.
o http://www.hko.gov.hk/contentc.htm is Perfect; this has identical content (its a mirror site).
o http://www.hko.gov.hk/ is Perfect; this is an entry page on the mirror.
en-us, {absolut vodka}
o http://www.absolut.com/us is Perfect; enter your age to proceed and rate the LP to which you are redirected.

en-us, {adult urls}


o [!]http://adultlinks101.com/ is Detrimental; you can make the rating on the basis of clear warning messages about
sexually explicit content.

Formatted: Right

32
7.3 Login pages
Many web sites restrict content to only registered users. You should mark such pages as Login unless:
o you can determine how well the LP satisfies the users intent without logging in, or
o you already have an account on the site and can quickly log in to determine relevance.

en-us, {Outlook Compose email}


o http://www.hotmail.com/secure/start?action=inbox is
Login; were not sure whats behind the login screen.

en-us, {Facebook}
o http://facebook.com is Perfect; we are certain this URL
satisfies the users intent.

en-us, {online.firsttechcu.com}
o https://online.firsttechcu.com/, as shown in the picture,
is Perfect; we are certain this URL satisfies the users
intent.

en-us, {Liquent policy handbook}


o http://lounge.liquent.com/default.asp?p=/secure/news-
view.asp?n=849 is Login; we cant see if the handbook
is there.

33
7.4 Spam, Phishing sites, and Parked Domains
Spam LPs include apparently relevant content to trick search engines into thinking the LP is relevant to a query; however,
they contain little valuable information to users. Such pages often link to adult content or malware; some are parked
domains registered in the hope of resale at a higher price. Look out for pages that have the query terms appearing with
unusually high frequency, content that mimics a well-known site, no individual logo, or an invitation to buy this domain.
Phishing sites aim to trick users into entering sensitive information such as phone numbers or login details; be suspicious of
LPs where the URL does not correspond to the content, or does not directly map to the organization name. Such examples
could include URLs like http://paypa1.com purporting to be http://paypal.com, or http://www.myhsbcbankonline.com
purporting to be http://hsbc.com (these two URL examples arent actual phishing sites). Phishing sites often use URLs that
are common misspellings of the definitive URL or URLs that are easily confused with the definitive URL.
Phishing and Spam pages should always be rated as Bad.

en-us, {PayPal}
o http://gizmogeekstv.com/ggtv/contentmedia/124/home.php is
Bad; this site has identical content to http://paypal.com (this
particular page is designed as an example, and isnt actually a
malware page). There are many similar pages that mimic
popular sites such as Facebook, MySpace, Amazon, UPS, and
Yahoo! Mail.

en-us, {furniture}
o http://www.cnpsaa.org/ is Bad; this page is now a parked
domain, but was previouslypacked with the query keyword to
mislead the search engine.

en-us, {www.dz-clan.net}
o http://www.dz-clan.net/ is Bad; even though this is an exact
match, there is no useful content on this page.

34
en-us, {usability index}
http://www.usabilityindex.com/ is Bad; this domain is for sale and contains no useful content.

en-us, {Rosa Parks}


o http://www.answers.com/topic/rosa-parks is
Excellent; although the content is aggregated
from other sources, these are clearly cited and the
LP does not appear to be using spam techniques to
mislead users or search engines.

en-us, {secluded beaches}


o http://secludedbeaches.com is Bad; this is a parked domain, as indicated by the generic messaging, query terms
appearing in Related Searches, and the offer to sell the domain name.

en-us, {Persepolis}
o http://www.sellingtimeshare.co.uk/wiki.asp?k=persepolis is Page Did Not Load; as shown, however, it would have
been rated Bad since the LP served Wikipedia pages after inserting links to timeshare sites below the heading; this
tricks search engines into giving them importance.

35
7.5 Proxies and Frames
Some sites openly or silently display content from other sites. While this is sometimes hard to identify, users may often notice that
the domain and content seem to be mismatched.

7.5.1 Proxies
Proxies are web sites that allow users to load the content of other sites; these are often used to circumvent access restrictions.
Unless the user has specifically asked for such pages, rate them as Bad (they correspond to very unlikely intent).

en-us, {facebook}
o http://www.proxyfacebook.co.uk/index.php?q=aHR0cDovL3d3dy5mYWNlYm9vay5jb20%3D&hl=3ed is Bad; while
this is a declared proxy site, there is no guarantee that this particular URL will always point to Facebook.
en-us, {bing}
o http://anonymouse.org/cgi-bin/anon-www.cgi/http://www.bing.com/ is Bad; this is the Bing page loaded through the
anonymouse.org proxy.

7.5.2 Frames
A web page may use frames to divide the browser display area into multiple sections, with each frame displaying content
from a different URL. This is usually done to help with page layout, but sometimes frames are used to show content from
another web site.
If you see multiple scroll bars on the page, or notice that the LP content doesnt seem to match the URL, check for frames by
hovering over links to see whether they go to a different domain; confirm by right-clicking on the web content, selecting
Properties, and comparing the URL shown with the URL of the LP.
If an LP uses frames in a way that is unhelpful or confusing, award a lower rating.
If an LP uses frames to mislead users, rate it as Bad; be very suspicious of LPs that collect sensitive information such as
login details while using framed content from another site (such as a bank) to gain the users trust.
You may find that the LP you are rating is designed to be shown only in context of a frame, rather than to stand on its own;
take this into account when assigning a rating.

en-us, {roy nelson}


o http://roynelson.com is Perfect; definitive LP.
o http://roynelson.tv is Perfect; this is a legitimate mirror site serving content from the main site (notice how the links
on this page all point to the .com site) in a full-page frame (no difference in authority or usability).
en-us, {Poptropica}
o http://www.poptropica.com/ is Perfect; definitive LP.
o http://demo.dropmark.com/175/2153 is Fair, shows the Poptropica home page within a clearly visible frame, which is
less usable; note also that the URL in the browser bar does not match the Poptropica content, and would be confusing
to users.
Formatted: Centered

36
en-us, {Temple Daily Telegram}
o http://www.theworldpress.com/press/state
spress/texaspress/temple.htm is Fair; this
web site simply presents content from
different press web sites, in this case
http://tdtnews.com, in a frame with an ad
at the top (confusing and less usable).

en-us, {kwik kar pflugerville}


o http://www.kwikkarpflugerville.com/Lube%20and%20Tun
e/Main%20Frameset/Lube%20and%20Tune%20Frameset.
htm is Perfect. This LP has three frames: one at the top
for the header, one at the left for the navigation menu,
and one for the main content.

o http://www.kwikkarpflugerville.com/Lube%20and%20Tune/
Main%20Frameset/Lube%20&%20Tune%20Main.htm is
Fair. This LP is designed to be shown within the main LP
frame, not to stand on its own. Users will be confused by the
missing context and navigation links.

o http://www.kwikkarpflugerville.com/Lube%20and%20Tune/Main%20Frameset/Menu_top-Frameset.htm is Bad. The


content is designed to be shown within the main LP frame, and has almost no useful content on its own.

Formatted: Centered

37
7.6 Freshness

7.6.1 General Tips


For many queries, content freshness is important, and old content is of little value. For example, freshness is important for
sales or pricing information, product reviews, event schedules, train timetables, school admission procedures, software
product versions, technical information, and laws and regulations.
For some queries, such as news, weather, or traffic, only very fresh content is useful; LPs that do not mention recent key
developments or current facts are near-useless. You may need to include news sites to determine possible fresh intent.
Remember that breaking stories may not be lengthy; consider whether the page you are reviewing has the extent of
information available at the time of judging. If it does, then it can be a strongly satisfying result.
If in doubt about the freshness of the content, check the LP and other parts of the site for dates; there is often a last-
modified date or a copyright year at the bottom of the LP. The date the LP was last modified is not always the same as the
date of the content, which may be dynamically loaded from a database. Similarly, a copyright year may simply indicate the
year the web site was created; however, a stale copyright year (e.g. 1998-2004) may be a hint that the site is closed or no
longer being updated. There will often be other clues that a site may be abandoned, such as broken links or missing images.
You must also determine whether the associated query intents would vary with time, in the case of a major event (like the
death of Michael Jackson or Whitney Houston). If so, then the LP must be evaluated for relatively current dates, and
demoted if it poses little value to the user due to its age/date of release. The following chart illustrates this using the query
{oceans}; a movie by this name debuted in 2010.

300

250

200

150 Oceans Movie


Water Oceans
100

50

0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

There are two prominent intents for this query; the intent for the movie is indicated in red, and the intent to learn about the
oceans is shown in blue. As you can see, the number of people looking for the actual oceans of the world stays fairly
constant. However, the number of people looking for the movie spikes for a brief period. During this period, the movie
represents a very likely intent. Before and after this period, interest in the actual oceans represents the most likely intent.

en-us, {oceans}
o http://en.wikipedia.org/wiki/Ocean is Excellent when the movie is not in theatres; the LP strongly satisfies the most
likely intent to find general information about the worlds oceans.
o http://en.wikipedia.org/wiki/Ocean is Good when the movie is in theatres; LP strongly satisfies the users likely
intent, since the intent to learn about the movie is more likely.
en-us, {oceans}
o http://disney.go.com/disneynature/oceans/ is Excellent when the movie is in theatres; the movies official homepage
strongly satisfies the very likely fresh intent to find current information about the movie.
o http://disney.go.com/disneynature/oceans/ is Fair after the movie has been in theatres, because the movie is no
longer current or very popular.

38
7.6.2 Identifying Fresh Intent for Queries
There are essentially three types of queries:
o Fresh not Important: Many of the queries you will see will fall into this category and include queries with navigational
intents or general informational intents.
o Very Likely Fresh: Queries covering on-going events, active public celebrities or with a transactional intent usually
indicate a very likely intent to find the freshest content or the latest product.
o Most Likely Fresh: Queries that include words like current, latest results, new, upcoming or gossip indicate a most likely
intent for the freshest available content.

Fresh Not Important Very Likely Fresh Most Likely Fresh


Barack Obamas childhood Barack Obama Obamas upcoming speech
Ashton Kutcher Kelso Ashton and Demi Ashton Kutcher gossip
Eagles Superbowl wins Philadelphia Eagles Current Eagles standings
1980s Madonna Madonna Madonna concert schedule
First Transformers movie Transformers movie New Transformers movie
Top selling bands of 2010 Black Eyed Peas Current #1 album

Queries where Fresh is not Important


en-us, {Fayette county change fact transit}
o http://www.factbus.com/ is Perfect; definitive page for the most likely intent.
en-us, {ghd}
o http://www.ghdhair.com/ is Perfect; definitive page for the most likely intent.
en-us, {1906 san francisco earthquake}
o http://earthquake.usgs.gov/regional/nca/1906/18april/index.php is Excellent; the event in question happened a long
time ago, so freshness of LP content isnt important.
en-us, {friends episodes list}
o http://en.wikipedia.org/wiki/List_of_Friends_episodes is Excellent; even though the page was last updated on
September 27th 2008, the shows last season was in 2004, so there is no expectation of fresh content.
o http://www.friends-tv.org/epguide.html is Good; complete information but less credible source.
en-us, {ashton Kutcher kelso}
o http://en.wikipedia.org/wiki/That_'70s_Show is Good; the most likely intent is to find out about Kutchers character
named Kelso on That Seventies Show. This Wikipedia page moderately satisfies that intent as the scope of the article
is too broad compared to the query specifics.

Queries with a Very Likely Fresh Intent


en-us, {owen wilson} (judging on 7/23/11)
o http://www.imdb.com/name/nm0005562/ is Excellent; LP offers clips, images, filmography, and has a related news
section. This LP strongly satisfies a very likely intent to find current movie and other related information on this actor.
This LP would be rated Good if there were a more likely fresh intent (Owen Wilson had been shot).
en-us, {SIGIR} (judging in December 2012)
o http://sigir2013.ie/ is Perfect; most users entering this query are interested in the conference, and the closest event
(upcoming or just past) is the most likely intent. SIGIR 2013 will be held in July 2013, with paper submissions due in
mid-January.
o http://www.sigir.org is Excellent; this is the home page of the group.
o http://www.sigir.org/sigir2012/is Excellent; this contains valuable information on the immediate past conference,
which is a very likely intent; people may still be interested in details on that conference and the papers presented there.
o http://sigir2011.org/is Fair; contains information on the 2011 conference, but this corresponds to an unlikely intent.
o http://www.sigir2010.org is Fair; contains information on the 2010 conference, again an unlikely intent.
o http://sigir.org/sigir2009 is Fair; contains information on the 2009 conference, again an unlikely intent.
o http://www.sigir.mil is Good; this is a likely intent.

39
en-us, {American cancer society}
o http://www.fathermag.com/health/circ/acs/ is Bad; content is outdated (from 1996) and most likely not very useful
to a user seeking information.
en-us, {bear cookie jar}
o http://www.amazon.com/Bear-Cookie-Jar-BACKORDERED-Until/dp/B002VDM5UW is Bad; the item is currently out of
stock (back ordered), so the LP does not satisfy the users transactional intent (to buy such a cookie jar). It also has very
little related content, so it doesnt satisfy the informational intent either.

Queries with a Most Likely Fresh Intent (at Time of Judging)


en-us, {9 year old suspended after michael Jackson dance} ( judging on 2/7/12)
o http://lacrossetribune.com/news/local/year-old-suspended-after-imitating-michael-jackson/article_470d061a-4eeb-
11e1-bde0-001871e3ce6c.html is Excellent; news article from a nearby paper that contains the most recent and
comprehensive information.
en-us, {earthquake Philippines} (judging on 2/7/12)
o http://www.foxnews.com/world/2012/02/05/magnitude-68-earthquake-shakes-philippines-no-immediate-reports-
injuries-or/ is Excellent; news article with the most recent and comprehensive information.
en-us, {whitney Houston} (judging on 2/14/12, one day after Houstons death)
o http://www.telegraph.co.uk/culture/music/music-news/9082132/Whitney-Houstons-funeral-to-take-place-on- Formatted: Indent: Left: 0.5"
Saturday.html is Excellent; while the query is broad, most users will want the latest news on the singers death, which
is a major event. Article contains the most updated and comprehensive information on the upcoming funeral.
o http://www.whitneyhouston.com/us/home is Excellent; news on Houstond death is most likely at time of judging,
and the official home page does does not strongly satisfy this intent.
en-us, {miss world 2011} (judging on 11/9/11)
o http://today.msnbc.msn.com/id/45183468 is Good; article is very narrowly focused on the winner and lacks details
about the contest and only moderately satisfies the most likely intent for the latest comprehensive information.
en-us, {Haiti earthquake} (judging 1/13/10)
o http://www.cnn.com/2010/WORLD/americas/01/12/haiti.earthquake/index.html is Excellent; many details were not
known at this point, and this article provides a summary and pictures on an authoritative news source.
o judging on 1/20/10, the rating is Fair; many details have come in, and this LP is no longer very relevant and only
weakly satisfies the most likely fresh intent to find coverage of this story.
o Judging today, the rating is Bad; the article is incredibly out-dated and does not contain much of the information on
the aftermath. The LP is a poorly satisfying result.
en-us, {nba all star} (judging in July 2010)
o http://www.nba.com/allstar2010 is Perfect; this is the definitive page at the time of judging.
o http://en.wikipedia.org/wiki/2010_NBA_All-Star_Game is Excellent; LP offers a detailed report and analysis of this
2010 game.
o http://www.nba.com/allstar2009 is Fair; interest in the 2009 game is very low.
en-us, {black friday} (judging in July 2012)
o http://www.2012blackfridayads.com/ is Good; focused on upcoming Black Friday deals.
en-us, {Brittany Spears gossip} (judging on 7/25/11)
o http://www.thehollywoodgossip.com/categories/britney-spears/is Excellent; LP provides high quality and recently
updated gossip on Brittany Spears, so the LP strongly satisfies the most likely fresh intent to find gossip about this
singer. Notice that there are also older articles on the lower part of the page. However, the main focus of the landing
page is on the recent content, so the older content should not count against the page.
o http://www.imdb.com/name/nm0005453/is Bad; this query has clearly fresh intent, but this page contains no gossip
on Spears and is not designed to contain the most recent information.
en-us, {netflix} (judging on 7/21/11)
o http://thestartupfoundry.com/2011/07/21/the-netflix-price-increase-the-%E2%80%9Cstartup-playbook%E2%80%9D-
and-my-own-hypocrisy/ is Fair; at the time, Netflix had announced its price increase. LP is from that same day and on
that subject, but does not contain significant information on the subject, but rather a persons opinion on it.
en-us, {one life to live} (judging on 1/13/12)
o http://onelifetolive.about.com/library/couples2001/blsam_nora.htm is Bad; content is too narrow and outdated (no
longer relevant). The soap opera last aired on 1/13/12, so users will be disappointed with this 2001 information.
en-us, {Plymouth antiquarian society events}
o http://www.plymouthantiquariansociety.org/events_2008.htm is Bad; content is outdated (from 2008), whereas the
user is most likely looking for information on upcoming events.

40
7.7 Market
LPs aimed at users in a different country may be less useful than those tailored for the users country, particularly for
transactional queries.
Consider the users intent; if an LP is clearly less useful because of issues such as international shipping fees , location,
currency or measurement unit mismatches, award a lower rating.

7.7.1 Queries where market does not matter


en-us, {ryan air}
o http://www.ryanair.com is Perfect; this is the definitive home page of the Irish airline Ryan Air and strongly satisfies
the most likely intent of the query. Users expect this site to be aimed at European users (prices shown are in GBP).
en-gb, {diabetes}
o http://diabetes.ca/diabetes-and-you/what/facts/ is Excellent; this LP strongly satisfies the most likely intent to find
out about diabetes. Content is very useful, even though the site is based in Canada, and is on an authoritative site.
en-us, {Tropical Vacation}
o http://www.bermudatourism.com is Good; users expect out-of-market content. The LP is authoritative but only
moderately satisfies the most likely intent to find out about tropical vacations, as it is narrowly focused on Bermuda.
en-us, {quadratic equation}
o http://www.mathsteacher.com.au/year10/ch12_quadratic_equations/01_solving/quad.htm is Good; the content is
useful, even though the site is based in Australia, and moderately satisfies the most likely intent to find out about this
equation; the credibility of the site is unknown, so most users would keep looking.
en-us, {Martin Luther King}
o http://www.spartacus.schoolnet.co.uk/USAkingML.htm is Good; content is useful, even though the LP is from the UK.

7.7.2 Queries where market does matter


en-us, {buy HDTV}
o http://www.amazon.co.uk/s/ref=nb_ss?url=search-alias%3Daps&field-keywords=hdtv is Bad; LP poorly satisfies the
most likely intent to buy a TV. US users would face problems such as currency conversion, shipping costs, delayed
delivery, and warranty restrictions if they were to purchase from this UK site.
en-gb, {vw dealer}
o http://www.volkswagen.co.uk/find-a-retailer is Perfect; LP is the definitive UK dealers page and strongly satisfies the
most likely intent of the query.
o http://www.vw.com/dealerlocator/en/us is Bad; LP is the definitive US dealers page and poorly satisfies the most
likely intent to find VW dealers in Great Britain. This LP would be useful to almost no users.
en-us, {youtube}
o http://www.youtube.com/?gl=GB&hl=en-GB is Fair; aimed at British users, and in British English; the featured TV
shows and Popular sections are different from those on the en-us page.
en-gb, {yahoo answers}
o http://uk.answers.yahoo.com is Perfect; in-market homepage.
o http://answers.yahoo.com is Excellent; very usable but not as appropriate as the UK site, so demote by only one level.
en-sg, {Harley Davidson}
o http://www.harleysingapore.com/ is Perfect; official home page for the Singapore market.
o http://www.harley-davidson.com/wcm/Content/Pages/home.jsp?locale=en_XX is Excellent; global home page.
o http://www.harley-davidson.com/en_US/Content/Pages/home.html is Good; US home page. While the site content
is useful and its unlikely users want to actually buy online, the dealer information isnt useful to a Singapore user.
en-us, {nokia.com}
o http://nokia.com is Perfect; redirects to the US homepage.
o http://nokiausa.com is Perfect; US page.
o http://www.nokia.co.uk is Bad; UK models with prices in GBP, not useful to US users.
en-us, {buy Honda car}
o http://www.honda.co.za is Bad; South African site; unlikely that US users want to purchase and ship car from there.
en-us, {download MP3 music}
o http://www.mfiles.co.uk/mp3-files.htm is Fair; the content is useful to an en-us user, but the music selection and LP
content wont be as appropriate as an en-us site.
en-au, {car insurance}
o http://www.theaa.com/aboutaa/contact.html is Bad; it is very unlikely a user would be looking for this information on
AA for such a general query.

41
7.8 Language
The language and script of the LP plays a big role in determining how well users can read and navigate the content. Users
generally prefer results in the target language and script, but they sometimes expect or actually want foreign content.
When evaluating LPs with foreign content, consider whether this content is expected or usable by most users for the query.
For convenience, users often type queries, especially personal or brand names, in Latin characters or in English, but this
doesnt necessarily mean that they want English content. Similarly, a foreign name in a query does not automatically
indicate foreign intent.
Choose Foreign only if the LP appears relevant to the query, but you dont understand enough to confidently determine
relevance for most users for this query.

en-us, {le monde}


o http://www.lemonde.fr is Perfect; definitive page for this French newspaper, most likely intent of users for this query.
es-es, {Patrick Bruel}
o http://www.patrickbruel.com is Perfect; home page for the singer, in French (the user would expect this).
zh-hk { } (Shandong Airlines)
o http://www.shandongair.com.cn/ is Perfect; definitive Simplified Chinese LP.
es-us, {Universidad de Washington}
o http://washington.edu is Perfect; in English; there is no Spanish equivalent, and users would expect the content in
English.
en-us, {yahoo Taiwan}
o http://tw.yahoo.com is Perfect; definitive Taiwanese page, clearly what the user wanted.
en-in, {photo of Obama speaking}
o http://www.siyasetdergisi.com.tr/Haber/OBAMA'NIN-SURIYE'YE-YONELIK-KONUSMASI/51 is Fair; has a single photo
that moderately satisfies a likely intent. Its not important that most users wouldnt understand the Turkish text.
ja-jp, {google maps}
o http://maps.google.co.jp is Perfect; definitive Japanese LP; there is probably no foreign intent.
o http://maps.google.com is Excellent; definitive English LP, possible intent of this query, and also widely usable.
es-es, {Universidad de Barcelona}
o http://www.ub.edu/homeub/es is Perfect; definitive Spanish LP.
o http://www.ub.edu is Excellent; definitive Catalan LP, (main university LP; expected, but not in target language).
en-us, {dalay}
o http://www.dalay.de is Bad; its very unlikely that many US users have this particular German tobacconist in mind
(both language and market are mismatched).
fr-fr, {lastminute}
o http://www.lastminute.de/ is Bad; there is a French version of this site, and there is no foreign language intent for
this fairly well-known travel site.
fr-fr, {Mir Hossein Mousavi}
o http://fr.wikipedia.org/wiki/Mir_Hossein_Moussavi is Good; in French.
o http://en.wikipedia.org/wiki/Mir-Hossein_Mousavi is Good; this is an Excellent result but in English.
o http://fa.wikipedia.org/wiki/%D9%85%D9%88%D8%B3%D9%88%DB%8C is Bad; in Persian, not widely usable.
en-ca, {Harley Davidson}
o http://www.harley-davidson.com/wcm/Content/Pages/home.jsp?locale=en_CA is Perfect; definitive English LP.
o http://www.harley-davidson.com/wcm/Content/Pages/home.jsp?locale=fr_CA is Excellent; in French, widely usable.
o http://www.harley-davidson.com/wcm/Content/Pages/home.jsp?locale=es_ES is Bad; in Spanish, not widely usable.
ja-jp, {htc}
o http://www.htc.com/jp/r is Perfect; definitive JapaneseLP.
o http://www.htc.com/asia is Excellent; definitive English LP for Asia.
o http://www.htc.com/europe is Fair; definitive English LP for Europe.
o http://www.htc.com/hk-tc is Bad; definitive Traditional Chinese LP for Hong Kong.
vi-vn, {Douglas Adams}
o http://www.douglasadams.com is Perfect; definitive English LP, strongly satisfies the most likely intent.
fr-fr, {ferrari}
o http://www.ferrari.com/Pages/Gateway.aspx?CountryId=114&CountryTitle=France is Perfect; definitive French LP,
most appropriate for navigational intent.
o http://www.ferrari.com/Italian is Fair; definitive Italian LP, not widely usable, and language selector is hard to find.

42
8 Mixed Rating Examples
en-us, {avg free edition}
o Most likely intent is transactional (download the anti-virus program), with a navigational aim (the official product page)
o http://free.avg.com/download-avg-anti-virus-free-edition is Perfect; this is the official page for the product, and
allows users to download the application with a single click.
o http://free.avg.com is Excellent; this is the site home LP, slightly less specific and deep than the above.
o http://free.avg.com/download?prd=afe is Good; LP provides a list of the various free AVG installation files, rather
than a direct download link.
o http://www.avg.com.au/index.cfm?section=avg&action=avg_anti_virus_free_edition is Fair; Australian-specific page;
after a registration step, users can download the same version as on the US site (see the Installation files tab).
o http://www.avgfree.com.au is Fair; similar to the above.
o http://www.oldapps.com/AVG_antivirus.php is Fair; has the latest version for download, but credibility is uncertain.
o http://download.cnet.com/AVG-Anti-Virus-Free-Edition/3000-2239_4-10726661.html is Bad; this old version (7.5.484
from August 2007) is useless to most users.
o http://www.avg.com/product-avg-internet-security is Bad; LP features a different product.
o http://www.avg-free.us is Bad; malware site tricking users to signing up and paying for this free software.
en-us, {bioshock}
o http://www.bioshockgame.com is Perfect; official entry page for BioShock.
o http://www.2kgames.com/bioshock is Perfect; mirror of the above.
o http://www.bioshock2game.com is Good; official LP for BioShock 2. Users interested in this game would likely have
written the query as {Bioshock 2}.
o http://www.2kgames.com/bioshock2 is Good; mirror of the above.
o http://xbox360.ign.com/objects/793/793105.html is Good; LP provides cheats and walkthrough information for
BioShock, a likely intent.
o http://xbox360.ign.com/objects/142/14240341.html is Bad; about BioShock 2, moderate LP for very unlikely intent.
en-us, {coheed and cambria tees}
o http://coheedandcambria.cinderblock.com/coheed-and-cambria-concert-shirts is Perfect; T-shirt section of official
online store; this satisfies the navigational aspect of the most likely intent.
o http://www.backstreetmerch.com/artist.asp?id=coca is Good; T-shirts are prominent, but not the focus of this LP.
o http://shop.ebay.com/i.html?_nkw=coheed+and+cambria is Fair; LP lists all items of this band offered for sale on
eBay (not limited to T-shirts, though the user can get to those by choosing a filter on the left of the LP).
o http://www.backstreetmerch.com/artist.asp?artist=coca&category=1 is Fair; only offers one T-shirt.
o http://www.concertshoppe.com/storefrontprofiles/DeluxeSFItemDetail.aspx?sfid=52527&c=513990&i=199723916 is
Bad; LP does not show any Coheed and Cambria tees.
en-us, {lufthansa.com}
o http://lufthansa.com is Perfect; entry page albeit mostly in German; has option to switch languages.
o http://www.lufthansa.com/online/portal/lh/us is Perfect; US LP.
o http://www.lufthansa.com/online/portal/lh/uk is Fair; UK LP, allows user to switch country.
o http://www.lufthansa.com/online/portal/lh/de is Fair; German LP, allows user to switch country and language.
en-us, {khalid aicha lyrics}
o http://www.metrolyrics.com/aicha-lyrics-dj-khaled.html is Good; English LP with French lyrics (the song is in French).
o http://www.lyricstime.com/khaled-a-cha-english-verison-lyrics.html is Good; LP has moderate authority, provides the
lyrics of the English adaptation, which is a very likely intent for the query.
o http://www.tv5.org/TV5Site/musique/paroles-182-aicha.htm is Bad; French LP with no English navigation or content.
en-us, {Pfaff Sewing Machine Dealer}
o http://webservices.us.vsmgroup.com/DealerLocator.aspx?Brand=Pfaff is Perfect; official web page catering
specifically for customers interested in locating a dealer (the most likely intent)
o http://www.pfaff.com/us/4547.html is Excellent; dealer sub-page for the official site, but contains only links to the
dealer locator for customers, and links for users who are dealers or trainers.
o http://www.pfaff.com/us is Good; US site, with link to dealer sub-page.
en-us, {remember the milk}
o http://www.rememberthemilk.com/?hl=en-US is Perfect; definitive page.
o http://www.rememberthemilk.com/?hl=en-GB is Excellent; in UK English, but language and market arent very
important for this online task management application.
o http://www.rememberthemilk.com/?hl=de is Bad; LP in German, which almost no users would find useful.

43
en-us, {Macys}
o http://macys.com is Perfect; this is the Macys home page.
o http://www1.macys.com/catalog/index.ognc?CategoryID=13247 is Fair; Macys shoes page (satisfies unlikely intent).
o http://www1.macys.com/catalog/product/index.ognc?ID=491442&CategoryID=13247&LinkType=PDPZ1 is Bad; a
specific model of shoe (very unlikely intent).
en-us, {Macys shoes}
o http://www1.macys.com/catalog/index.ognc?CategoryID=13247 is Perfect; this is the Macys shoes page.
o http://www.macys.com/shoes is Excellent; alternative page for shoes, but not equally appropriate.
o http://macys.com is Good; Macys home page, broader than the query intent.
o http://www1.macys.com/shop/product/isaac-mizrahi-new-york-shoes-leeza-pumps?ID=784295&CategoryID=50763 is
Fair; specific model of shoe.
o http://www.macys.com/catalog/index.ognc?CategoryID=544 is Bad; watches and jewelry page.
en-us, {Macys womens watches}
o http://www1.macys.com/shop/jewelry-watches/watches/all-watches/gender/womens-
watches?id=57385&edge=hybrid is Perfect; womens watches page.
o http://www1.macys.com/shop/jewelry-watches/all-watches?id=23930&edge=hybrid&cm_sp=us_catsplash_jewelry-
%26-watches-_-row3-_-watches is Excellent; main watches page. Slightly broader than query intent, but still very
useful.
o http://www.macys.com is Good; Macys home page, broader than the query intent.
o http://www1.macys.com/shop/jewelry-watches/watches/all-watches/gender/mens-watches?id=57386&edge=hybrid is
Fair; mens watches. User would need to select View all gender from the navigation menu on the left.
o http://www1.macys.com/catalog/product/index.ognc?ID=277928 is Fair; specific womens watch model.
o http://www1.macys.com/catalog/product/index.ognc?ID=451677 is Fair; specific unisex watch model.
o http://www1.macys.com/catalog/product/index.ognc?ID=451678 is Bad; specific mens watch model.
o http://www.macys.com/catalog/index.ognc?CategoryID=13247 is Bad; main shoes page.
o http://www.macys.com/catalog/product/index.ognc?ID=477565 is Bad; specific model of shoe.
en-us, {discovery 925}
o http://www.plantronics.com/us/product/discovery-925 is Perfect; official LP for this headset (black is the most likely
intent). Even though this product has been discontinued, this is still the definitive, official landing page for the result
and a must have result for the query.
o http://www.amazon.com/Plantronics-Discovery-925-Bluetooth-Packaging/dp/B001630QZE is Excellent; credible site,
with details and reviews. The query is informational (read product reviews) and transactional (buy the product).
o http://www.nextag.com/discovery-925-bluetooth-headset is Good; comparison shopping LP for this headset.
o http://www.nextag.com/discovery-925 is Fair; comparison shopping LP for related items (broader than query).
o http://www.msgdigital.com/productdetail.asp?productid=46105 is Page Did Not Load; standard error message.
o http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=Plantronics+Discovery+925 is
Good; list includes related items on Amazon.com.
o http://www.plantronics.com/north_america/en_US/search/index.jsp?term=Discovery+925 is Fair; search results for
these terms on the official web site; doesnt really satisfy the users intent.
o http://www.nextag.com/Plantronics-Discovery-925-Bluetooth-602609259/prices-html is Bad; there are no products
available on this page.
o http://store.apple.com/nz/product/TR199X/A is Page Did Not Load; LP returns a page cannot be found error message.
en-us, {clemson university admissions}
o http://www.clemson.edu/admissions is Perfect; admissions page.
o http://www.clemson.edu/prospectivestudents is Excellent; very useful, almost as good as the admissions page.
o http://www.clemson.edu/admission is Excellent; redirects to the prospective students page.
o http://collegesearch.collegeboard.com/search/CollegeDetail.jsp?collegeId=3086&profileId=1 is Good; information on
Clemson University admissions from a credible third party (the College Board).
o http://www.applyingtoschool.com/forms/clemson_university.aspx is Fair; uses Clemson University as an example.
o http://www.grad.clemson.edu/Deadlines.php is Bad; lists important dates for students planning to graduate this year.
en-us, {my bobs}
o http://www.mybobs.com is Perfect; this domain name is heavily featured in advertising and is the most likely intent.
o http://www.answers.com/topic/bob-s-discount-furniture is Fair; aggregated information on Bobs Discount Furniture
(unlikely intent); users interested in facts on the company would be more likely to use the official company name.
o http://www.flickr.com/groups/foamcorebob is Bad; features photos of a dog called FoamcoreBob, and includes the
phrase where is my Bobs at. Very unlikely intent.

44
o http://www.alexa.com/search?q=mybobs is Bad; search results page on Alexa.com, showing ratings for sites related
to the term mybobs.
o http://chat.mybobs.com/locations/furniture-store-totowa-nj.asp is Page Did Not Load; domain lookup failure.
en-us, {mira costa high school football}
o http://www.costafootball.com is Perfect; most users would want current and detailed information, and this is the
definitive web site directly satisfying the most likely intent.
o http://www.miracostahigh.org/apps/pages/index.jsp?uREC_ID=75186&type=d is Good; this is the official football
team page on the school web site, but has very limited information (moderately satisfies the most likely intent).
en-us {ms} (ambiguous query) Formatted: French (France)
o http://www.nationalmssociety.org/ is Excellent; satisfies the very likely intent, Multiple Sclerosis.
o http://microsoft.com is Good; satisfies the intent, Microsoft, that is likely for this specific query.
o http://en.wikipedia.org/wiki/Ms is Good; disambiguation page.
en-us, {comcast complaints}
o http://consumeraffairs.com/cable_tv/comcast_cable.html is Excellent; popular web site showcasing user complaints
about Comcast. The query indicates that reading complaints is a more likely intent than filing a new complaint.
o https://www.consumeraffairs.com/php/a_report.php is Good; form to report a complaint on this web site, not
specific to Comcast.
o http://www.comcast.com/Corporate/Customers/contactus/ContactUs.html is Good; this official LP explaining how to
contact Comcast customer support will be useful to many users.
en-us, {paris}
o http://en.parisinfo.com Perfect; the official website of the Paris Convention and Visitors Bureau, satisfies the most
likely intent of US users.
o http://www.paris.fr/portail/english/Portal.lut?page_id=8118 is Good; this is the official web site of the City
government; has some relevant information, but does not directly address the most likely intent.
o http://paristexas.com is Good; definitive site of Paris, Texas (likely intent for US users).
en-us, {babys dancing to michael Jackson}
o http://search.aol.com/aol/search?enabled_terms=&s_it=aolon&q=babies%20dancing%20michael%20jackson is
Good; search results page showing many copies of the same video clip on YouTube (there are more diverse results on
the next pages of results). Many users would find this LP useful.
o http://www.youtube.com/watch?v=IIt5lGhFyE0 is Good; a clip of babies dancing to Billie Jean by Michael Jackson.
en-us, {valley club}
o http://www.thevalleyclub.org/ is Perfect; most likely intent.
o http://valleyhuntclub.com is Fair; LP of The Valley Hunt Club, unlikely intent. We can rate without needing to log in.
o http://www.redfin.com/CA/Napa/104-Valley-Club-Cir-94558/home/2224786 is Bad; lists a single property on Valley
Club Circuit, Napa, CA.
en-us, {capital one online banking}
o https://onlinebanking.capitalone.com/CapitalOne/ is Perfect; main online banking page.
o http://www.capitalone.com/online-banking/ is Excellent; satisfies the informational aspect of the query, and also has
a link to the online banking site.
o https://onlinebanking.capitalone.com/CapitalOne/Enrollment.aspx is Good; LP to sign up for online banking.
o https://servicing.capitalone.com is Good; online banking for credit cards, a likely intent.
o https://onlinebanking.capitalone.com/CAPITALONE/Login.aspx?ori=COAFPARTNER is Fair; online banking page for
auto finance accounts, an unlikely intent.
en-us, {Movies thar Salena Gomez played in}
o http://www.imdb.com/name/nm1411125/ is Excellent; lists films in which this actress has played.
o http://selenagomez.com/ is Fair; this is Selena Gomez official homepage with a link to a subsection on her films, but
the film section is not very robust or satisfying.
o http://movies.yahoo.com/movie/contributor/1808823462 is Fair; lists only a few films with no dates.
o http://www.starpulse.com/Actresses/Gomez,_Selena/gallery/ALO-027034 is Bad; photo gallery LP with a link to a
poor filmography page.
en-us, {www.airtrans.com}
o http://www.airtran.com is Perfect; most likely intent (assumes query was misspelled).
o http://www.airtran.com/Home.aspx is Perfect; the above URL redirects here.
o http://airtrans.com.hk is Bad; LP for Airtrans Logistics International in Hong Kong, very unlikely intent for US users.
o http://www.airtrans.com is Page Did Not Load; domain lookup failure.
en-us, {slashdot}
o http://slashdot.org is Perfect; official homepage.

45
o http://twitter.com/slashdot is Good; tweets linking to the latest postings, useful for many users of this news site.
o http://tech.slashdot.org/story/10/02/15/1738249/Anti-Terror-Honor-System is Bad; a specific satirical posting (very
unlikely intent)
en-us, {rand()}
o http://www.php.net/manual/en/function.rand.php is Good; describes the PHP random number function.
o http://www.perlmeme.org/howtos/perlfunc/rand_function.html is Good; covers the Perl random number function.
o http://www.rand.org/ is Bad; definitive LP for the RAND corporation (very unlikely intent in light of the parentheses).
en-us, {port solent}
o http://www.portsolent.com is Good; web site of the Boardwalk retail area in Port Solent (too specific).
o http://www.welcometoportsmouth.co.uk/port%20solent%20shopping.html is Fair; describes the Boardwalk.
o http://www.sunsail.com/destinations/united-kingdom is Fair; LP offers chartered yachts out of Port Solent, an
unlikely intent.
o http://www.sail-the-net.com/marinas/europe/united-kingdom/hampshire/portsmouth/port-solent-marina.html is
Fair; LP provides information on the Port Solent Marina, satisfying an unlikely intent strongly.
en-us, {x men wolverine origins}
o http://en.wikipedia.org/wiki/X-Men_Origins:_Wolverine is Good; detailed page satisfying the informational intent.
o http://www.uncaged.com is Good; official site for the video game.
o http://www.gamespot.com/x-men-origins-wolverine/ is Fair; review of the video game for PlayStation 3.
o http://www.nydailynews.com/news/ny_crime/2009/12/16/2009-12-
16_bronx_archnemesis_of_wolverine_is_nabbed_by_fbi_for_allegedly_pirating_xmen_movi.html is Bad; old article
about an arrest involving video piracy, very unlikely intent.
en-us, {Playtex}
o http://www.playtexbras.com is Excellent; for Playtex Bras (very likely intent).
o http://www.playtexproductsinc.com is Excellent; for Playtex products (very likely intent).
o http://playtex.com is Perfect; entry page that caters for both intents.
en-us, {matt christopher books}
o http://www.mattchristopher.com/books/books.asp is Perfect; books sub-page of authors official web site.
o http://www.mattchristopher.com is Good; authors official LP, not the most likely intent for this specific query.
o http://www.amazon.com/Matt-Christopher/e/B000APFXQQ/ is Excellent; the most likely intent is to learn about and
buy the books.
o http://www.infoplease.com/ipea/A0762143.html is Fair; very sparse information on the author and his books.
en-us, {Andrea Bocelli}
o http://www.andreabocelli.com is Perfect; the artists official home page.
o http://en.wikipedia.org/wiki/Andrea_Bocelli is Good; most users will be looking for current news on Bocelli, or very
likely want to buy his music; this strongly satisfies a likely intent to learn all about Bocelli.
en-us, {stewart milne homes}
o http://www.stewartmilnehomes.com/ is Perfect; official homepage.
o http://www.stewartmilne.com/PageProducer.aspx?Page=2123 is Fair; companys mortgage products (unlikely intent).
en-us, {beretta color leather riding jacket}
o http://www.thefind.com/apparel/info-air-motorcycle-jacket is Bad; LP features motorcycle jackets by other
manufacturers; the only reference to Beretta is a link to waxwear field jackets.
o http://www.gizmodiva.com/fashion/washed_leather_jacket_by_beretta.php is Fair; showcases a single leather jacket
by Beretta. Its in black (not color), and theres no reference to riding.
en-us, {electro comm distributing}
o http://www.huttononline.com is Perfect; official site (Hutton acquired Electro Comm Distributing in 2006).
o http://www.electro-comm.com is Bad; despite the URL, LP has nothing to do with the query.
en-us, {www.nctreasurer.com}
o http://www.nctreasurer.com is Perfect; exact match and the most likely intent.
o http://www.nctreasurer.net is Perfect; this has identical content to the .com site (its a mirror).
en-us, {www.zen-cart.com}
o http://www.zen-cart.com/ is Perfect; exact match and the most likely intent.
o http://www.nobownogo.com is Bad; only connection is that the site runs on the zen-cart online store platform (see
copyright notice at the bottom of the LP).
en-us, {little america marshall wi}
o http://www.insiderpages.com/b/3723631780 is Good; contains a map and many reviews.
o http://searchandtravelguide.com/region/3078-Dane-WI.htm is Bad; lists this attraction but has no specific content.
en-us, {crosswalk.com}

46
o http://crosswalk.com is Perfect; definitive LP.
o http://www.biblestudytools.com/ is Good; despite URL, LP is one of the main subsections on crosswalk.com and
satisfies a likely intent strongly.
en-us, {fight cancer from within}
o http://www.amazon.com/Fighting-Cancer-Within-Power-Healing/dp/080506916X is Good; a likely intent is to
purchase this book (note that the book title isnt exactly the same as the query).
o http://www.time.com/time/magazine/article/0,9171,1003358,00.html is is Login;Article appears relevant, but we
cannot read more than a few lines without logging in. Therefore, we cannot fully assess the relevance.
en-us, {Jamaica gleaner}
o http://www.jamaica-gleaner.com is Perfect; definitive home page.
o http://gleaner-ja.com is Perfect; this has identical content (its a mirror).
en-us, {birds}
o http://www.audubon.org/ is Excellent; detailed LP from well-known specialist organization.
o http://www.audubon.org/bird/stateofthebirds/rarest.html is Fair; too specific, no context, and poor linked content.
en-us, {US Open}
o http://www.usopen.com is Excellent; satisfies the very likely intent, US Golf Open; when the Golf Open is on, this
would be Perfect.
o http://www.usopen.org is Excellent; satisfies the very likely intent, US Tennis Open; when the Tennis Open is on this
would be Perfect.
en-us, {airline tickets}
o http://www.expedia.com/daily/flights is Excellent; well-known and credible LP sells tickets from many airlines.
o http://www.expedia.com is Excellent; main LP not specific to flights, but not meaningfully worse than the above LP.
o http://www.united.com is Good; only United Airlines tickets are available from this LP (narrower than intent).
en-us, {vacoule under microscope}
o http://en.wikipedia.org/wiki/Vacuole is Fair; no specific content on microscopic examination of vacuoles.
o http://www.docstoc.com/docs/70704500/Observing-cells-under-a-microscope is Bad; contains teacher's notes on
observing cells under a microscope. Has no content addressing vacoule.
en-us, {business tax fort myers, fl}
o https://www.cityftmyers.com/businesstax is Perfect; this is the local government LP for business tax for the town of
Fort Myers.
o http://www.fortmyersbeachfl.gov/index.aspx?NID=214 is Bad; for the town of Fort Myers Beach (very unlikely intent).
en-us, {capital athletics conference}
o http://www.cacsports.com is Perfect; official homepage of this conference.
o http://groups.ycp.edu/cac/cacmbindex.html is Bad; although this site says its the official site, its not been updated
since the 2003-2004 season, and almost no users would find this page useful.
en-us, {Pacific Dessert}
o http://www.boorah.com/restaurants/2322/CA/Redondo-Beach/best-desserts.html is Bad; the query terms appear on
the page as Pacific Coast Hwy and Dessert Restaurant, which is very unlikely to be what users want.
en-us, {arctic cat parts}
o http://store.arcticcat.com/Parts is Perfect; strongly satisfies the most likely intent.
o http://www.arcticcatpartshouse.com/ is Excellent; a specialized site that exactly addresses the users intent, with a
good range of parts for many Arctic Cat models.
o http://www.alpha-sports.com/arctic_cat_atv_parts.htm is Good; ; while the page has ATVs selected, users can easily
choose other Arctic Cat products to look up and purchase related parts.
o http://www.arcticcat.eu/int/parts-amp-accessories.html is Page Did Not Load.
en-gb, {Radisson sas lillehammer}
o http://www.radissonblu.com/hotel-lillehammer is Perfect; in English.
o http://www.lillehammerhotel.no is Bad; in Norwegian by default, with no easy option to switch to English.
en-us, {map of v&a museum} (judging in June 2013)
o http://www.vam.ac.uk/__data/assets/pdf_file/0020/231770/museum_map_spring_summver_2013.pdf is Perfect;
spring-summer 2013 map from the official museum web site.
o http://www.vam.ac.uk/your_visit/va_map/ is Excellent; official web page of the museum map. Most users would be
satisfied with this result, although they would need to click on a link to access the actual map.
en-us, {asahi newspaper in Japanese}
o http://www.asahi.com is Perfect; in Japanese, as specified by the query.
o http://www.asahi.com/english is Fair; English LP; has option to change to Japanese, but isnt what the user asked for.
en-us, {aafes}

47
o http://www.aafes.com is Perfect; official homepage.
o http://www.shopmyexchange.com/ is Perfect; this has identical content (its a mirror).
en-us, {tiger woods sponsors}
o http://web.tigerwoods.com/sponsors/sponsors is Perfect; definitive LP on Tiger Woods web site, has current list.
o http://www.forbes.com/2009/09/29/tiger-woods-billion-business-sports-tiger.html is Fair; this LP is from September
2009, but the LP is broad and could be relevant to some users looking for the history of his sponsorships.
en-gb, {preston library}
o http://www.lancashire.gov.uk/libraries/librarydetails/libsearch1.asp?name=Preston+Harris is Perfect; library in
Preston, Lancashire, UK (most likely intent).
o http://www.librarytechnology.org/lwc-displaylibrary.pl?RC=35278 is Bad; library in Preston, Vic, Australia.
fr-fr, {anpe}
o http://www.pole-emploi.fr is Perfect; official LP of the French national employment agency (with the acronym: ANPE).
o http://anpeniger.blogspot.fr/ is Bad; official LP of the National Employment Agency in Niger, which has the same
acronym. The page is in French, but its very unlikely that the French user is looking for the agency in Niger.
ko-kr, {ds3053 statement}
o http://travel.state.gov/passport/forms/ds3053/ds3053_846.html is Perfect; definitive English LP.
o http://www.usimmigrationsupport.org/form_ds3053.html is Fair; in English (the query has implicit foreign intent).
fr-fr, {khalid aicha paroles anglais} (Khalid Aicha English words)
o http://lyricskeeper.fr/fr/khaled/achaenglishverison.html is Good; in French, with the English lyrics the user wants.
o http://www.lyricstime.com/khaled-a-cha-english-verison-lyrics.html is Fair; in English, with the English lyrics
ko-kr, {hyatt cambridge}
o http://cambridge.hyatt.com is Perfect; definitive English LP.
o http://www.hotels.co.kr/ho106028/hyatt-regency-cambridge-keimbeuliji-migug/ is Excellent; in Korean.
es-es, {ebay.com}
o http://ebay.es is Perfect; this the official site for this market.
o http://www.ebay.com is Excellent; in English and not the official site for this market.
zh-cn, {} (Hong Kong government)
o http://www.gov.hk/sc/residents/ is Perfect; definitive LP in Simplified Chinese, target encoding for zh-cn.
o http://www.gov.hk/tc/residents/ is Excellent; in Traditional Chinese (widely understood by users).
o http://www.gov.hk/en/residents/ is Excellent; in English (widely understood by users).
zh-cn, {mazda}
o http://www.mazda.com.cn is Perfect; in Chinese.
o http://www.mazda.com is Excellent; global English site, no text in Chinese to help the user change the language, but
since the user has typed the query in English, they can probably navigate this site.
o http://www.mazdausa.com is Fair; content isnt very useful to most users in China.
en-us, {qq}
o http://www.qq.com is Perfect; definitive Chinese LP, satisfies the most likely navigational intent (Foreign if you cant
rate this).
o http://www.imqq.com is Excellent; international English LP; this is not a direct equivalent of the Chinese LP above,
and most QQ users speak Chinese, so this is less likely to be what most users entering this query want.
ar-xa, {sun java download}
o http://java.com/en/download/index.jsp is Perfect; definitive English LP for the Java runtime, most likely intent.
o http://java.sun.com/ is Good; definitive English LP for Java development, likely intent.
ar-xa, {( } BBC)
o http://www.bbc.co.uk/arabic is Perfect; definitive Arabic LP.
o http://www.bbc.co.uk is Good; definitive English LP, usable for many users, but not as good as the Arabic LP.

48
9 Glossary
These sites gather together specific content from multiple websites. For instance, shopping aggregators like
Aggregators
Amazon, eBay, Nextag, and TheFind all provide extensive results for different products users may search for.
These operators help to further refine a user's search. Here are common operators you may encounter:
Operator Shows only results that: Examples (en-us)
site: are from this web site rich guy site:onion.com
url: are from this web site url:.ir persepolis
filetype: are of this file type Introductory statistics filetype:pdf
link: link to this URL link:vogon.com/guide
Advanced
define: have definitions for this term define:nihilism
Search
cache: are stored by the search cache:http://www.bbc.co.uk/cult/hitchhikers/guide/art
Operators
engine hur.shtml
Inurl: have this word in the URL inurl:bing
intitle: have this term in the title intitle:"personal finance"
Inbody: have this term in the body intitle:composting
+ definitely have this term guide anchor +flexwiki heading
- do not contain this term U2 Joshua tree
Definitive/ The LP published by the person or organization directly connected to the subject of the query, e.g. http://www-cs-
Official LP faculty.stanford.edu/~knuth/ for {Donald Knuth}, or http://microsoft.com/windows for {Microsoft Windows}.
A technique for dividing the browser window into sections, each showing content from a different LP. See Section
Frames
7.5.2 for more details.
HRS Human Relevance System, the tool judges use to rate how well a web page satisfies a user query.
Intent The outcome the user intended when they entered the query into a search engine.
Landing page The result page to be rated by the judge; with the exception of entry and splash pages, this page is directly
(LP) displayed by the HRS tool, and there is no need to manually navigate to the LP.
The combination of target language and market for which the judging is performed, e.g. en-us (English, US), fr-ca
Locale
(French, Canada), and zh-cn (Simplified Chinese, China). For more examples see this page.
Mirrors Legitimate web sites that contain identical content, usually for reliability or ease of access.
No Judgment The content of the landing page cannot be accurately assessed, typically due to problems with the server or
(NJ) because it is a language the judge does not understand.
Non-Target
A language or script other than the language of the locale, e.g. a Chinese LP when judging for en-us users.
Language
Parked A web site registered but not serving content useful to users; these are often registered in the hope of reselling to
Domain an interested buyer at a higher price, and are used to serve ads until an interested buyer comes along.
PDNL Page did not load; the usual content of the LP is not available for a temporary reason.
A web service that is used to load content from other sites; these are often used to circumvent access restrictions.
Proxy
See Section 7.5.1 for more details.
Query Terms typed in to the search engine by a user; results are judged by relevance to the intent behind these terms.
Query / URL The combination of the query entered by the user, and the LP URL returned by the search engine. Judges should
pair rate the relevance of the LP to the query intent.
A categorization of queries by intent (expected results). There are three query types: Navigational (to find a
Query type particular site), Informational (to find information on a subject), and Transactional (to complete a transaction on
the Web); a query may span multiple types.
Rating A label assigned by a judge to indicate how well an LP satisfies the intent of the user.
Relevance The degree to which a result is logically related to a query and satisfies the users intent.
Content designed to mislead users and/or search engines into thinking that a web page is relevant to the users
Spam
intent. Spam pages often aim to get users to click into sites serving ads, pornography, or malware.
A usually colorful and graphics-intensive page designed to get users excited about the main landing page they are
Splash page
about to view, and to call out particular features that users should look out for.
Target
The language and script of the specified locale, e.g. Simplified Chinese for zh-cn.
Language
URL Uniform Resource Locator, the address of a page or other content, e.g. http://yahoo.com.

49
10 Process Diagram

Manual redirect: user must click to visit


Start: Graphic or new location of content
Research
Cant rate
sexually explicit?
Cant rate Cant rate Spam or Parked Domain
possible user
No as regular content is No
Adult rosters: child
No as content behind a No as content in a No Proxy
intents
unavailable?
pornography?
login screen? foreign language? Malware: tries to install or run unknown
software, or otherwise harm PC.

Yes Yes Yes Yes Yes

NJ: Page Did Not Load Detrimental NJ: Needs Login NJ: Foreign Bad

No
No No No No No

Addresses the intent of >50% of users Addresses the intent of >25% of users Addresses the intent of >10% of users Addresses the intent of >1% of users Addresses the intent of <1% of users
Exactly what most users want and Has what most users want, and will be Very helpful, but not exactly what Related, and could be useful to some Almost nobody typing in the query will
expect to see at the top of the search happy to see in the top search results users want; users want to see such users; such LPs should be shown after find this page useful; no useful relation
engine results (navigational intent) results in the first few pages of results the better results to the query.
Often official company or celebrity LP Must provide some relevant content;
Not necessarily the most detailed do not assign this rating just because
page, but the most authoritative one the search terms appear on the LP
Usually only 1 perfect LP for a query

Criteria: Criteria: Criteria: Criteria: Criteria:


Scope: matched to query Scope: matched to query Scope: close to query Scope: not close to query Scope: completely mismatched
Freshness: fresh where important Freshness: fresh where important Freshness: dated where important Freshness: old where important Freshness: very old where important
Authority: credible where important Authority: credible where important Authority: doubtful where important Authority: low where important Authority: very low where important
Market: matched where important Market: matched where important Market: matched where important Market: mismatched where important Market: mismatched where important
Language: Expected and/or usable Language: Expected and/or usable Language: Expected and/or usable Language: Expected and/or usable Language: Unusable

Yes Yes Yes Yes Yes

Select appropriate checkboxes Select appropriate checkboxes Select appropriate checkboxes Select appropriate checkboxes Select appropriate checkboxes

Perfect Excellent Good Fair Bad

Checkboxes:
Not confident: very uncertain of rating, should get a second opinion
Time-sensitive: rating for this LP will likely be different in six months time.

50