General guidelines 2011

General Guidelines Version 3.18 March 30, 2011

Part 1: Rating Guidelines ........................................................................................ 5
1.0 Welcome to the Search Quality Rating Program! ........................................................................................... 5
1.1 URL Rating Overview ................................................................................................................................ 5
1.2 Important Rating Definitions and Ideas .................................................................................................. 5
1.3 The Purpose of Search Quality Rating .................................................................................................... 6
1.4 Raters Must Represent the User .............................................................................................................. 6
1.5 Internet Safety Information ....................................................................................................................... 7
2.0 Understanding the Query .................................................................................................................................. 8
2.1 Understanding User Intent ....................................................................................................................... 8
2.2 Task Language and Task Location.......................................................................................................... 8
2.3 Queries with Multiple Meanings............................................................................................................... 9
2.4 Classification of User Intent: Action, Information, and Navigation – “Do-Know-Go”....................... 9
2.4.1 Action Queries – “Do” ................................................................................................................. 9
2.4.2 Information Queries – “Know” ................................................................................................. 11
2.4.3 Navigation Queries – “Go”........................................................................................................ 11
2.4.4 Queries with Multiple User Intents (Do-Know-Go) ................................................................. 12
3.0 The Language of the Landing Page ............................................................................................................... 13
4.0 The Rating Scale .............................................................................................................................................. 14
4.1 Vital ........................................................................................................................................................... 14
4.1.1 Examples of English (US) Navigation Queries with Vital Pages for the Task Location ...... 14
4.1.2 Examples of Entity Queries with Vital Pages ........................................................................... 15
4.1.3 Vital Pages for People Queries .................................................................................................. 16
4.1.4 Other Important Vital Concepts ................................................................................................. 17
4.1.5 Vital Pages and Geographic Location ...................................................................................... 19
4.2 Useful ......................................................................................................................................................... 20
4.2.1 Examples of Useful Pages ......................................................................................................... 20
4.3 Relevant .................................................................................................................................................... 21
4.3.1 Examples of Relevant Pages ..................................................................................................... 22
4.4 Slightly Relevant....................................................................................................................................... 22
4.4.1 Examples of Slightly Relevant Pages ....................................................................................... 23
4.5 Off-Topic or Useless – On 3/30/2011, this label was changed from Off-Topic to Off-Topic or Useless
............................................................................................................................................................................ 25
4.5.1 Examples of Off-Topic or Useless Pages ................................................................................. 25
4.6 Unratable ................................................................................................................................................... 28
4.6.1 Unratable: Didn’t Load .............................................................................................................. 28

Proprietary and Confidential – Copyright 2011 1

4.6.2 Unratable: Foreign Language .................................................................................................. 31
5.0 Rating: From User Intent to Assigning a Rating ......................................................................................... 32
5.1 User Intent and Page Utility..................................................................................................................... 32
5.2 Location is Important ............................................................................................................................... 33
5.3 Language is Important (This section is for Non-English Task Languages) ........................................ 34
5.4 Multiple Interpretations ............................................................................................................................ 36
5.5 Specificity of Queries and Landing Pages ............................................................................................ 38
5.6 Common Rating Problems ...................................................................................................................... 42
5.6.1 Dictionary or Encyclopedia Results ......................................................................................... 42
5.6.2 Action vs. Information Intent ..................................................................................................... 43
5.6.3 Queries that Ask for a List ......................................................................................................... 43
5.6.4 Misspelled and Mistyped Queries ............................................................................................. 47
5.6.5 URL Queries ................................................................................................................................ 49
5.6.6 New and Old Pages ..................................................................................................................... 53
5.6.7 Search Engine Result Pages ..................................................................................................... 54
5.6.8 Video Landing Pages .................................................................................................................. 59
6.0 Flags ................................................................................................................................................................... 60
6.1 Spam Flag ................................................................................................................................................. 60
6.2 Pornography Flag ..................................................................................................................................... 60
6.2.1 Clear Non-Porn Intent ................................................................................................................. 60
6.2.2 Possible Porn Intent ................................................................................................................... 61
6.2.3 Clear Porn Intent ......................................................................................................................... 61
6.2.4 Reporting Illegal Images ............................................................................................................ 62
6.3 Malicious Flag ........................................................................................................................................... 63
6.4 Compatibility between Ratings and Flags ............................................................................................. 63

Part 2: URL Rating Tasks with Query Locations ................................................ 64
1.0 Query Locations ............................................................................................................................................... 64
2.0 Location-Specific Rating Task Screenshot ................................................................................................... 66
3.0 Assigning a Rating When There is a Query Location .................................................................................. 67
3.1 When Does the Query Location Matter? ..................................................................................... 67
4.0 Query Location Rating Examples................................................................................................................... 69

Part 3: Rating Examples ....................................................................................... 76
1.0 Named Entity Queries ....................................................................................................................................... 76
2.0 Action Queries................................................................................................................................................... 83
3.0 Information Queries .......................................................................................................................................... 86
4.0 Queries that Ask for a List ............................................................................................................................... 89

5.0 Rating Examples for Task Locations other than English (US) ..................................................................... 93

Part 4: Webspam Guidelines ................................................................................ 95
1.0 What is Webspam ? .......................................................................................................................................... 95
1.1 The Relationship between Ratings and Spam ...................................................................................... 95
1.2 Why do Spammers Create Spam Pages? .............................................................................................. 95
1.3 When to Check for Spam ......................................................................................................................... 95
2.0 Browser Requirement ....................................................................................................................................... 96
3.0 Looking for Technical Signals ......................................................................................................................... 96
3.1 Hidden Text and Hidden Links ................................................................................................................ 97
3.1.1 Apply Ctrl-A to the Landing Page.............................................................................................. 97
3.1.2 Disable CSS ................................................................................................................................. 97
3.1.3 Disable JavaScript ...................................................................................................................... 98
3.1.4 View the Source Code ................................................................................................................ 98
3.1.5 Look Outside the Normal Viewing Area ................................................................................... 99
3.2 Keyword Stuffing ...................................................................................................................................... 99
3.2.1 Keyword Stuffing in the URL ..................................................................................................... 99
3.3 Sneaky Redirects.................................................................................................................................... 100
3.3.1 Using “Whois” ........................................................................................................................... 100
3.4 Cloaking .................................................................................................................................................. 101
3.4.1 JavaScript Redirects ................................................................................................................ 101
3.4.2 100% Frame ............................................................................................................................... 101
4.0 Helpful Webpages vs. Spam Webpages ....................................................................................................... 101
4.1 Pages with Copied Content and PPC Ads ........................................................................................... 102
4.1.2 Copied Text and PPC Ads ........................................................................................................ 102
4.1.3 Feeds and PPC Ads .................................................................................................................. 102
4.1.4 Doorway Pages ......................................................................................................................... 102
4.1.5 Templates and Other Computer-Generated Pages ............................................................... 103
4.1.6 Copied Message Boards .......................................................................................................... 103
4.1.7 Recognizing Copied Content ................................................................................................... 103
4.2 Fake Search Pages with PPC Ads ........................................................................................................ 104
4.3 Fake Blogs with PPC Ads ...................................................................................................................... 104
4.4 Fake Message Boards with PPC Ads ................................................................................................... 104
4.5 Copied Content that is NOT Spam........................................................................................................ 105
5.0 Commercial Intent ........................................................................................................................................... 105
5.1 Thin Affiliates .......................................................................................................................................... 105
5.1.1 Recognizing Thin Affiliates ...................................................................................................... 105
5.1.2 Not all Affiliates are Thin .......................................................................................................... 106


5.1.3 Recognizing True Merchants ................................................................................................... 106
5.2 Pure PPC Pages...................................................................................................................................... 106
5.3 Parked (Expired) Domains ..................................................................................................................... 107
5.4 Pages with Unhelpful Content and PPC Ads ....................................................................................... 107
6.0 Phishing Websites.................................................................................................................................. 108
7.0 Spam and the Resolving Stage ..................................................................................................................... 108
8.0 Conclusion ....................................................................................................................................................... 109

Part 5: Using EWOQ ............................................................................................ 110
1.0 Introduction ..................................................................................................................................................... 110
2.0 Accessing the EWOQ Rating Interface ......................................................................................................... 110
3.0 Rating ............................................................................................................................................................... 110
4.0 Rating Home Screenshots ............................................................................................................................. 111
5.0 Resolving Tasks (Re-rating Unresolved Tasks) / Moderators .................................................................... 116
6.0 Commenting Etiquette .................................................................................................................................... 118

Part 6: Quick Guide to URL Rating .................................................................... 120

Part 7: Quick Guide to Webspam Recognition ................................................. 123


Part 1: Rating Guidelines

1.0 Welcome to the Search Quality Rating Program!

As a Search Quality Rater, you will work on many different types of rating projects. These guidelines cover just one
type of search quality rating – URL rating.

Please take the time to carefully read through these guidelines. The ideas presented here are important for other types
of rating. When you can do URL rating, you will be well on your way to becoming a successful Search Quality Rater!

1.1 URL Rating Overview

For each URL rating task you acquire, you will see a query and a URL. You will:

• Research the query
• Click on the URL to visit the landing page
• Assign a rating based on these guidelines

1.2 Important Rating Definitions and Ideas

Search Engine: A search engine is a website that allows users to search the Web by entering words or symbols into a
search box.

Query: A query is the set of word(s), number(s), and/or symbol(s) that a user types in the search box of a search
engine. We will sometimes refer to this set of words, numbers, or symbols as the “query terms”. Some people also
call these “key words”. In these guidelines, queries will have square brackets around them. If a user types the words
digital cameras in the search box, we will display: [digital cameras].

User Intent: When a user types a query, he is trying to accomplish something, such as finding information or
purchasing an item online. We refer to this goal as the user intent.

Task Language and Task Location: Queries have a task language and task location associated with them and will
look like this in these guidelines: [digital cameras], Spanish (ES). This format indicates that the query digital
cameras was typed into a search box by a Spanish reading user in Spain. Task locations are represented by a two-
letter country code. The country code for Spain is ES. If the query had been typed by a Spanish reading user in
Mexico, it would look like this: [digital cameras], Spanish (MX).

For a current list of country codes, go to
http://www.iso.org/iso/country_codes/iso_3166_code_lists/english_country_names_and_code_elements.htm.

Homepage (of a website): When we use the term “homepage”, we are referring to the main page of a website. It is
the first page that users see when the website loads. The URL for the homepage of a website usually ends
with .com, .edu, .org, .gov, etc., or the two-letter code for a country outside the US, such as .jp, .mx, .ru, etc. For
example, http://www.apple.com/ is the homepage of the Apple computer company website, and
http://www.mcdonalds.com/ is the homepage of the McDonald’s hamburger corporation website. We are aware that
some countries use the term “homepage” to refer to the entire website of a company, organization, individual, etc.
However, we use “homepage” to refer to the main page only.


Subpage: A page on a website that is not the homepage. For example, http://www.apple.com/iphone/ is a subpage on
the Apple website. An example of a subpage on the McDonald’s website is
http://www.mcdonalds.com/usa/rest_locator.html.

Webpage or Web Page: Any page on a website. It may be the homepage or a subpage of the website.

URL: The URL is the Web address of the webpage you will evaluate, such as http://www.microsoft.com. It is important
to look at the URL, but remember that you will evaluate the landing page.

Landing Page or Page: This refers to the webpage that you will evaluate. It is the page you see after you click on the
URL. These guidelines will explain how to evaluate the content of the landing page. You may see ads and sponsored
links on many landing pages. You will evaluate only the content posted by the webmaster. Your rating will not be
based on ads or sponsored links on the page (even if they are related to the query).

Topic: The topic of the query is the focus or subject of the query; it is what the query is about. Users typing the query
want to find pages on the Web that are related to the topic of the query.

Utility: The utility of the landing page is a measure of how helpful the page is for the user intent. Pages with good
utility are helpful for users. Pages with no utility are useless. Utility is the most important aspect of search engine
quality, and is therefore the most important thing for you to think about when evaluating webpages.

The Rating Scale will be described in detail in Section 4, but here is a brief overview. For each task, you will assign
exactly one of the following ratings:

Rating Scale Description
Vital A special rating category (see Section 4.1)
Useful A page that is very helpful for most users.
Relevant A page that is helpful for many or some users.
A page that is not very helpful for most users, but is somewhat related to the query. Some or few
Slightly Relevant
users would find this page helpful.
Off-Topic or Useless A page that is helpful for very few or no users.
Unratable A page that cannot be evaluated. A complete description can be found in Section 4.6.

You will also assign any of the following flags that apply: Not Spam, Maybe Spam, Spam, Porn, and Malicious.
They will be discussed in Section 6.

1.3 The Purpose of Search Quality Rating

Your ratings will be used to evaluate search engine quality around the world. Good search engines give results that
are helpful for users in their specific language and location.

1.4 Raters Must Represent the User

It is very important for you to represent the user. The user is someone who lives in your task location and reads the
task language, and who has typed the query in the search box.

You must be very familiar with the task language and task location in order to represent the experience of users in your
task location. If you do not have the knowledge to do this, please inform your employer.


1.5 Internet Safety Information

In the course of your work, you will visit many different webpages. Some of them may harm your computer unless you
are careful. Please do not download any executables, applications, or other potentially dangerous files, or click on any
links that you are uncomfortable with. We strongly recommend that you have antivirus and anti-spyware
protection on your computer. This software must be updated frequently or your computer will not be
protected. There are many free and for-purchase antivirus and anti-spyware products available on the Web.

Here are links to Wikipedia articles with information about antivirus software and spyware:

http://en.wikipedia.org/wiki/Antivirus_software
http://en.wikipedia.org/wiki/Spyware

We suggest that you only open files you are comfortable with. Please feel free to release rating tasks if they contain
unknown or suspicious file formats.

The file formats listed below are generally considered safe if antivirus software is in place.

 .txt (text file)
 .ppt or .pptx (Microsoft PowerPoint)
 .doc or .docx (Microsoft Word)
 .xls or .xlsx (Microsoft Excel)
 .pdf (PDF) files

If you encounter a page with a warning message, such as “Warning-visiting this web site may harm your computer,” or
if your antivirus software warns you about a page, you should not try to visit the page to assign a rating. You should
instead assign a rating of Unratable: Didn’t Load. A description of this rating can be found in Section 4.6.1.

You may also come across pages that require RealPlayer or the Adobe Flash Player plug-in. These are safe to
download at:

http://www.real.com/
http://www.adobe.com/shockwave/download/download.cgi?P1_Prod_Version=ShockwaveFlash

Examples of pages that require Flash Player are: http://www.ferrariworld.com and http://www.atraircraft.com.


2.0 Understanding the Query

Before you can evaluate the task, you must understand the query. Please use an online dictionary or encyclopedia
that is available for your task location, or do web research to help you understand all of the words in the query. All web
research must be done using the Firefox browser.

Important: If you use a search engine to research the query, please do not rely only on the ranking of results that you
see displayed on the search results page. A query may have other meanings besides those represented in the top
results. Do not assign a high rating to a webpage just because it appears at the top of a list of search results.

Here are some examples of the kinds of reliable resources available on the Web that may be helpful:

Online encyclopedias:
http://en.wikipedia.org/wiki/Main_Page: the English language version of Wikipedia
http://www.wikipedia.org/: portal to other language/locale versions of Wikipedia

Translation tools:
http://babelfish.altavista.com/
http://www.wordreference.com/
http://translate.google.com/

2.1 Understanding User Intent

In addition to understanding the meaning of the query, you must also consider user intent. What was the user trying to
accomplish when he typed the query? You will need to understand user intent to evaluate the landing page.

Consider the query [tetris], English (US). Most English speaking users in the United States who type this query know
that Tetris is a popular computer game. The most likely user intent is to play the game online.

Here are some other examples of queries and user intents:

Query Likely User Intent
[Fedex], English (US) Track a package or find a Federal Express location

Find, customize, and print a calendar for the current month or year

[calendar], English (US) Find a calendar that displays holidays

Find an online calendar to use to organize one’s time

[ebay], English (US) Buy or sell merchandise on eBay, or navigate to the eBay homepage

2.2 Task Language and Task Location

All queries have a task language and task location. Keeping these in mind will help you to understand the query and
user intent. Users in different parts of the world may have different expectations for the same query.

Query Query Meaning in the Task Location Likely User Intent in the Task Location
American football played with a brown Find recent game scores, game schedules, pictures, team
[football], English (US)
oval ball information, etc. for American football in the US.
Find recent game scores, game schedules, pictures, team
The game Americans call soccer,
[football], English (UK) information, etc. for soccer in the UK or perhaps around
played with a round ball
the world.


2.3 Queries with Multiple Meanings

Many queries have more than one meaning. For example, the query [apple], English (US) might refer to the computer
brand or the fruit. We will call these possible meanings query interpretations.

Dominant Interpretation: The dominant interpretation of a query is the interpretation that most users have in mind
when they issue the query. For example, most users typing [windows], English (US) want results on the Microsoft
operating system, rather than the glass windows on a wall. The dominant interpretation should be clear to you,
especially after doing a little web research.

Common Interpretations: In some cases, there is no dominant interpretation. The query [mercury], English (US)
might refer to the car brand, the planet, or the chemical element (Hg). While none of these is clearly dominant, all are
common interpretations. Many or some people might want results related to these interpretations.

Minor Interpretations: Sometimes you will find less common interpretations. These are interpretations that few users
have in mind. We will call these minor interpretations. Consider again the query [mercury], English (US). Possible
meanings exist that even most English (US) users probably don’t know about, such as Mercury Marine Insurance and
the San Jose Mercury News. These are minor interpretations.

When you evaluate pages associated with a minor interpretation of the query, you will use lower ratings on the Rating
Scale. In Section 5.4, we will discuss in detail how to rate pages when the query has multiple interpretations.

2.4 Classification of User Intent: Action, Information, and Navigation – “Do-Know-Go”

Sometimes it is helpful to classify user intent for a query in one or more of these three categories:

 Action intent – Users want to accomplish a goal or engage in an activity, such as download software, play a
game online, send flowers, find entertaining videos, etc. These are “do” queries: users want to do something.
 Information intent – Users want to find information. These are “know” queries: users want to know
something.
 Navigation intent – Users want to navigate to a website or webpage. These are “go” queries: users want to
go to a specific page.

An easy way to remember this is “Do-Know-Go”. Classifying queries this way can help you figure out how to rate a
webpage. Please note that many queries fit into more than one type of user intent.

2.4.1 Action Queries – “Do”

The intent of an action query is to accomplish a goal or engage in an activity on the Web. The goal or activity may be
to download, to buy, to obtain, to be entertained by, or to interact with a resource that is available on the Web.

Users want to do something. Here are some examples of goals and activities:

• Purchase a product
• Download software for free or for money
• Pay a bill online
• Play a game online
• Print a calendar
• Send flowers
• Organize photos or order prints online
• Watch a video clip
• Copy an image or piece of clipart
• Take an online survey
• View entertaining webpages, such as pictures, gossip, videos, etc.


Helpful pages for an action query are pages that allow users to do the activity or accomplish the goal.

Description of
Query Likely User Intent URL of a Helpful Page
The Landing Page
[geography quiz], Take an online geography http://www.lufthansa- Page with an online geography
English (US) quiz usa.com/useugame2007/html/play.html quiz that users can take

Find an image of a
[Beatles poster], http://www.allposters.com/-sp/- Page on which to view or
Beatles poster or perhaps
English (US) Posters_i317216_.htm purchase a Beatles poster
purchase a Beatles poster

[download adobe http://www.adobe.com/products/acrobat Official free download page on
Download software
reader], English (US) /readstep2.html the Adobe website

[fairy tale coloring http://www.dltk-teach.com/rhymes/color- Page with printable coloring
Print coloring pages
pages], English (US) index.htm pages

Page on which to take the
[online personality Take an online personality http://www.humanmetrics.com/cgi-
Humanmetrics Jung Typology
test], English (US) test win/JTypes1.htm
Test

[what is my bmi?], Calculate the BMI (body http://nhlbisupport.com/bmi/ Reputable pages with BMI
English (US) mass index) http://www.cdc.gov/nccdphp/dnpa/bmi/ calculators

[good cop baby cop], View the “Good Cop, http://www.funnyordie.com/videos/33f26 Page on which to view this
English (US) Baby Cop” video 87080 video

[cute kitten pics], View photos of cute Page of cute kitten photos to
http://thecuteproject.com/tags/kitten/
English (US) kittens look at

http://www.amazon.com/Citizen-Kane-
Georgia-Backus/dp/B00003CX9E
[Citizen Kane DVD], Pages on which to purchase
Purchase this DVD
English (US) this DVD
http://www.cduniverse.com/productinfo.
asp?pid=1980921

http://www.ftd.com/
[flowers], Pages on which to order
Order flowers online http://www.1800flowers.com/
English US flowers online
http://www.proflowers.com/

[play sudoku], http://www.websudoku.com/ Pages on which to play
Play Sudoku online
English (US) http://sudoku.com.au/ Sudoku

[calculate running Calculate running pace http://www.coolrunning.com/engine/4/4_ Page with running pace
pace], English (US) online 1/96.shtml calculator

Play Bubble Spinner 2
[bubble spinner 2], http://www.addictinggames.com/bubble Pages on which to play and/or
online or download the
English (US) spinner2.html download this game
game

[Spanish English Translate Spanish words http://www.spanishdict.com/ Pages on which to translate
dictionary], into English or English http://www.wordreference.com/English_ words between Spanish and
English (US) words into Spanish Spanish_Dictionary.asp/ English


2.4.2 Information Queries – “Know”

An information query seeks information on a topic. Users want to know something; the goal is to find information.

Helpful pages have high quality, authoritative, and comprehensive information about the query.

Description of
Query Likely User Intent URL of a Helpful Page
The Landing Page

Find travel and tourism
http://www.lonelyplanet.com/switzerla Travel guide on Switzerland
information for planning a
[Switzerland], nd
vacation or holiday, or find
English (US)
information about the Swiss
https://www.cia.gov/cia/publications/f Informative CIA World
geography, languages,
actbook/geos/sz.html Factbook webpage on
economy, etc.
Switzerland
[cryptology use in Find information about how United States Air Force
http://www.nationalmuseum.af.mil/fac
WWII], cryptology was used in Museum article about
tsheets/factsheet.asp?id=9722
English (US) World War II cryptology use during WWII

[how to remove Find information on how to http://www.goodhousekeeping.com/h Page on a well-known
candle wax from remove candle wax from ome/heloise/floors-carpets/remove- magazine website with this
carpet], English (US) carpet candle-wax-mar03 information

2.4.3 Navigation Queries – “Go”

The intent of a navigation query is to locate a specific webpage. Users have a single webpage or website in mind.
This single webpage is called the target of the query. Users want to go to the target page.

The most helpful page for a navigation query is the navigational target page.

Query Likely User Intent URL of the Target Page Description of the Target Page

[ibm], Official homepage of the IBM
Go to the IBM homepage http://www.ibm.com/
English (US) Corporation

[youtube],
Go to the YouTube homepage http://www.youtube.com/ Office homepage of YouTube
English (US)

[ebay],
Go to the Italian eBay homepage http://www.ebay.it/ Official homepage of eBay Italy
Italian (IT)

[harvard
Go to the admissions page on the http://admissions.college.h Office of Admissions page on the
admissions],
Harvard website arvard.edu/index.html official Harvard website
French (FR)

[best buy store http://www.bestbuy.com/sit
Go to the store locator page on the Store Locator page on the official
locator], English e/olspage.jsp?id=cat12090
Best Buy website Best Buy website
(US) &type=page

[sony customer
Go to the customer support page on eSupport page on the official Sony
support], English http://esupport.sony.com/
the Sony website website
(US)

[outback
Go to the menu page on the Outback http://www.outback.com/me Menu page on the official Outback
steakhouse menu],
website nu/ Steakhouse website
English (US)


Query Likely User Intent URL of the Target Page Description of the Target Page
Go to the digital cameras page on the
Canon website. Although Canon is http://www.usa.canon.com/
[canon.com digital
primarily known for its digital cameras, consumer/controller?act=Pr Digital Cameras page on the official
cameras], English
the target of the query is the digital oductCatIndexAct&fcategor Canon website.
(US)
cameras page, not the Canon yid=113
homepage.

Go to the login page on the Facebook
website. Although users can log in
[facebook login], http://www.facebook.com/lo Login page on the official Facebook
from the Facebook homepage, the
English (US) gin.php website.
target of the query is the login page,
not the homepage.

2.4.4 Queries with Multiple User Intents (Do-Know-Go)

Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is
more likely than another intent. Here are some examples.

Query Likely User Intent URL of a Helpful Page Description of The Landing Page

Do and Go. This could be a The landing page is the Firefox browser download page
“do” and a “go” query. on the cnet.com website, which is a well-known,
http://download.cnet.co
Users want to download the respected website. Many users would feel comfortable
m/mozilla-firefox/
[download web browser Firefox (“do” downloading from this site. This page is helpful for the
firefox], user intent). Many users “do” user intent.
English (US) may want to download the
browser from the official http://www.mozilla.com/ The landing page is the official Firefox browser
Firefox website (“go” user en- download webpage. This page may be the target of the
intent). US/firefox/firefox.html query and is helpful for the “do” and “go” user intents.

Do, Know, and Go. This The landing page is the “Nikon” page on the target.com
http://www.target.com/N
could be a “do” and a “know” website. There are over 30 models of Nikon digital
ikon-
and a “go” query. Users are cameras for sale and the page has prices,
Electronics/b?ie=UTF8
probably interested in a specifications, and reviews. This page is helpful for
&node=1084298
Nikon digital camera. Some both the “do” and “know” user intents.
[Nikon digital
users may have decided to
cameras],
buy a Nikon (“do”), but some The landing page is the “Nikon Digital cameras” review
English (US) http://reviews.cnet.com/
may be researching the page on the cnet.com website, with helpful information
Nikon brand (“know”), and digital-camera-
about many different Nikon digital cameras organized
some may want to go to reviews/?filter=1000036
by price, resolution, digital camera type, and features.
digital camera pages on the _108496_&tag=centerC
The page allows users to compare prices, features, etc.
Nikon website (“go”). olumnArea1.0
This page is helpful for the “know” user intent.

http://www.engadget.co The landing page on the engadget.com website has a
m/2010/04/03/apple- comprehensive review of the iPad. This page is helpful
Do, Know, and Go. This ipad-review/ for the “know” intent.
could be a “do” and a “know”
and a “go” query. Users are The landing page is the iPad product page on the
probably interested in buying http://www.apple.com/ip official Apple website. This page may be the target of
[ipad], ad/ the query and is helpful for the “know” and “go” user
an iPad (“do”), but some
English (US) intents.
may be doing research
(“know), and some may The landing page is the iPad page on the Store part of
want to go to iPad pages on http://store.apple.com/u
the official Apple website. Users can make a purchase
the Apple website (“go”). s/browse/home/shop_ip
and find information. This page may be the target of
ad/family/ipad?mco=OT
the query and is helpful for the “do”, “know”, and “go”
Y2ODA0NQ
user intents.


3.0 The Language of the Landing Page

You are expected to read and understand your task language and English. You are also expected to have some
understanding of commonly used languages for your task location.

All landing pages will be flagged as one of the following:

 The task language
 An acceptable language
 English
 Foreign Language
 None of the above

Task Language: Use the flag that corresponds to your task language when the page content is entirely or mostly in
the task language.

Acceptable Language: Use the flag that corresponds to the appropriate acceptable language when the page content
is entirely or mostly in an acceptable language. Acceptable languages are other languages that are commonly used
by a significant percentage of the population in the task location. The rating task will display the acceptable languages
for the task location.

English: Use this flag when the page content is entirely or mostly English.

Foreign Language: Use this flag when you believe users in the task location would NOT be able to read/understand
the content of the page.

None of the above: Use this flag when there is no language on the page to identify. Examples are pages that are
completely blank, pages with images only, or pages with so much garbled text or so many encoding errors that you
cannot identify the language.

For mixed language pages: Use your best judgment. Don’t struggle with your selection of a language flag.

Here are some examples of landing page language flags:

Query Likely User Intent URL of the Landing Page Description Landing Page Language

Find information The landing page has Task Language – the page
[symptoms about http://www.mayoclinic.com/hea
about the information about content is in the task
diabetes], English lth/diabetes-
symptoms of diabetes. The text is language. English (US)
(US) symptoms/da00125
diabetes in English. users can read this page.

The landing page Foreign Language – the
appears to have page content is in a foreign
[diabetes], Find information http://www.dmedicina.com/enf
information about language. Most English
English (US) about diabetes ermedades/digestivas/diabetes
diabetes, but the text (US) users would not be
is in Spanish. able to read this page.

http://books.google.com/books
The landing page is a
Find information ?id=WVgRAAAAYAAJ&printse Foreign Language – the
book result for the
about the c=frontcover&dq=bollandists&s text is in a foreign language.
[bollandists], book “Analecta
association of ource=bl&hl=en&ots=yyEfxOJ Most English (US) users
English (US) Bollandiana, Volume
scholars known as abU&sig=22I2XRTHzNBBUOq would not be able to read
26”. The text of the
the bollandists. sK66tVqqUWbg#v=onepage& this page.
book is in French.
q&f=false


4.0 The Rating Scale

The rating scale offers five rating options that are based on user intent and the utility of the landing page: “Vital”,
“Useful”, “Relevant”, “Slightly Relevant”, and “Off-Topic or Useless”. In addition, there is a rating category that
will be used in special circumstances: Unratable.

4.1 Vital

The Vital rating is used for these very special situations:

1) The dominant interpretation of the query is navigation, and the landing page is the target of the navigation
query.
2) The dominant interpretation of the query is an entity (such as a person, place, business, restaurant, product,
company, organization, etc.), and the landing page is the official webpage associated with that entity.

In both cases, the query must have a dominant interpretation. If there is no dominant interpretation, it is not possible to
assign a Vital rating.

Most Vital pages are very helpful. Please note that this is not a requirement for a rating of Vital, however. Some Vital
pages are “official”, but not very helpful.

We will classify Vital pages further in section 4.1.5. First, here are examples of Vital pages for the English (US) task
location.

4.1.1 Examples of English (US) Navigation Queries with Vital Pages for the Task Location

Here are some examples of navigation or “go” queries and the target webpage.

Query Likely User Intent English (US) Vital Page Example Description of Vital Page

[nytimes], Go to the New York Times The homepage and target of the
http://www.nytimes.com/
English US online newspaper query

Go to the sports section of the
[nytimes sports], http://www.nytimes.com/pages/spor The sports section page and target
New York Times online
English US ts/ of the query
newspaper

[yahoo], Go to the official Yahoo The homepage and target of the
http://www.yahoo.com
English (US) homepage query

[yahoo mail], Go to the official Yahoo! Mail The Yahoo! Mail page and target of
http://www.mail.yahoo.com
English (US) login page the query

[walmart.com], Go to the official homepage of The homepage and target of the
http://www.walmart.com/
English (US) the Wal-Mart online retail site query

[walmart
Go to the storefinder page on http://www.walmart.com/cservice/c The storefinder page and target of
storefinder],
the Walmart website a_storefinder.gsp the query
English (US)

For “go” queries, the Vital page is the page requested by the user. If the query is for the homepage of a website, only
the homepage gets the Vital rating. If the query is for a subpage, only that particular subpage gets the Vital rating.

Please note that the URL you rate may not be the “standard” URL for the entity. The “standard” URL is the URL that
most users would expect to see. If the landing page for a “non-standard” URL is the same as the landing page for the
“standard” URL, the rating should be the same. Here are some examples:


Query Likely User Intent English (US) Vital Page Example Description of Vital Page
Standard URL:
The homepage and target of the
http://www.bedbathandbeyond.com/
Go to the official query.
[bed bath and
homepage of the Bed
beyond], Non-Standard URLs:
Bath and Beyond Even though the URLs look
English (US) http://www.bedbathandbeyond.com/default.asp
website different, the landing pages are the
http://www.bedbathandbeyond.com/default.asp
same and are all Vital for the query.
?order_num=-1&
The homepage and target of the
Standard URL:
query.
Go to the official http://www.officedepot.com/
[office depot],
homepage of the
English (US) Even though the URLs look
Office Depot website Non-Standard URL:
different, the landing pages are the
http://www.officedepot.com/index.do
same and are all Vital for the query.

Please note that some companies have corporate homepages, as well as “consumer” pages for regular users. Please
use your judgment and assign the Vital rating to the page you think most users want. Here is an example.

Query Likely User Intent URL of the Landing Page Rating
[toys r us], English (US) Go to the shopping
http://www.toysrus.com/ - This is the shopping page. Vital
page of Toys R Us.
Toys R Us is a well-known toy Most users issuing
store. It has two homepages: this query want to http://www1.toysrus.com/ - Relevant or
shopping and corporate. shop. This is the corporate homepage. Useful

4.1.2 Examples of Entity Queries with Vital Pages

Some entity queries have navigation intent, while others have information intent. For entity queries, the official
homepage of the entity is Vital, even if you think the user intent is information. Here are some examples:

Type of
Entity Query Example English (US) Vital Page Example Description of Vital Page
Entity Query
Celebrities [Madonna], English (US) http://www.madonna.com/ Madonna’s official homepage
Restaurants [Gary Danko], English (US) http://www.garydanko.com/ Official homepage of the restaurant
Official movie webpage on the movie
Movies [Bourne Ultimatum], English (US) http://www.thebourneultimatum.com/
studio website
Companies [Maytag], English (US) http://www.maytag.com/ Official homepage of the company
[The Da Vinci Code book], http://www.danbrown.com/#/davinci Official book page on the author’s
Books
English (US) Code website
Specific Official product page on the
[ipod nano], English (US) http://www.apple.com/ipodnano/
Products manufacturer’s site
[Statue of Liberty], English (US) Official page on the government
http://www.nps.gov/stli/
Famous website
locations [Baseball hall of fame],
http://baseballhall.org/
English (US) Official homepage of the museum
Special [Masters Golf Tournament], Official event homepage or official
http://www.masters.org/
Events English (US) webpage on the owner’s website
Government http://www.whitehouse.gov/administr Official page on the government
[President Obama], English (US)
officials ation/president-obama/ website
[Freakonomics blog], English http://freakonomics.blogs.nytimes.co Official blog page on the New York
Blogs
(US) m/ Times website
Universities [Harvard], English (US) http://www.harvard.edu/ Official homepage of the university


4.1.3 Vital Pages for People Queries

Queries for famous people, such as [george bush], [Madonna], and [david beckham], have obvious dominant
interpretations. Queries for common names, such as [bob smith] and [mary jones], which do not have a dominant
interpretation, can have no Vital result. If you are not sure about a name you don’t recognize, try doing query research.

A query for a non-famous person can have a Vital page if the person is uniquely specified or has a very unusual or
unique name so that there is a clear dominant interpretation. For example, Dave Jones is a common English name
and the query [dave jones], English (UK) can have no Vital result because we don’t know which Dave Jones the
user wants. However, the very specific query [dave jones codemonkey], English (UK) does have a clear dominant
interpretation.

Homepages, blogs, and social networking pages have become very popular, and many famous and non-famous
people now have multiple “official” personal pages on the Web. People may have multiple homepages, multiple blogs,
and multiple pages on various social networking sites, such as MySpace, Facebook, Friendster, Mixi, LinkedIn, Twitter,
YouTube, etc. Official homepages of all types are Vital for famous people (and for non-famous people who have
unusual, uniquely identifiable names).

Social networking pages for small groups of people (such as social clubs or musical bands) are also considered Vital.

Social networking pages for companies are NOT considered Vital.

It can sometimes be difficult to determine if a homepage, blog, or social networking page is official. Usually, official
webpages for famous people are “professional” in appearance and are often linked to from the individual’s other official
pages or from a Wikipedia article about the person. Please use your judgment and have high standards.

Here are some examples:

URL of the Landing
Query Description English (US) Vital Page?
Page

Hillary Clinton’s official campaign webpage. Even
[Hillary Clinton], http://www.hillaryclinto though the campaign is over, the page still exists for
Yes
English (US) n.com/ the purpose of accepting contributions to clear up her
campaign debt.

http://newyork.yankees Derek Jeter is a famous baseball player who plays for
[Derek Jeter],
.mlb.com/team/player.j the New York Yankees. This is his webpage on the Yes
English (US)
sp?player_id=116539 official Yankee website.

Arianna Huffington is a famous blogger. This is the
[Arianna blog], http://www.huffingtonp
homepage of The Huffington Post, a blog and Yes
English (US) ost.com/
commentary website founded by her.

Oprah Winfrey is a famous talk show host. This is
[oprah],
http://www.oprah.com/ the homepage of Oprah’s magazine, radio station, Yes
English (US)
book club, etc.
http://www.linkedin.co
[Lynn Bozof], Lynn Bozof is an uncommon name. This is her
m/pub/dir/?last=bozof& Yes
English (US) LinkedIn page.
first=lynn

Dave Smith is a common name without a dominant No – non-famous people
[Dave Smith], http://www.davewsmith
interpretation. A personal webpage for someone with common names can’t
English (US) .com/
named Dave Smith is not Vital. have Vital pages

[Britney
http://www.youtube.co Britney Spears is a famous singer and celebrity. This
Spears], Yes
m/user/britneytv is her YouTube Channel page.
English (US)


URL of the Landing
Query Description English (US) Vital Page?
Page

[green day], http://www.greenday.c Green Day is an American rock band. This is the
Yes
English (US) om/ band’s official homepage.

[green day], http://www.myspace.co
This is Green Day’s MySpace webpage. Yes
English (US) m/greenday

[green day], http://www.youtube.co
This is Green Day’s YouTube Channel page. Yes
English (US) m/user/greenday

No – social networking
pages can only be Vital
[photobucket], http://www.myspace.co Photobucket is an online photo sharing company.
for people, bands, and
English (US) m/photobucket This is the company’s MySpace page.
small groups. They are
not Vital for companies.

pages are only Vital for
[Ford], English http://www.facebook.c Ford is an automobile manufacturer. This is the
people, bands, and small
(US) om/ford company’s Facebook page.
groups. They are not
Vital for companies.

[Sheboygan pages are only Vital for
http://twitter.com/sheb Sheboygan Press is a newspaper. This is the
Press], English people, bands, and small
oyganpress newspaper’s Twitter page.
(US) groups. They are not
Vital for companies.

pages are only Vital for
[sesame street], http://www.youtube.co Sesame Street is a well-known children’s TV show.
people, bands, and small
English (US) m/user/SesameStreet This is the Sesame Street YouTube Channel page.
groups. They are not
Vital for TV shows.

No – company blogs are
[toyota], English Toyota maintains a company blog to communicate
http://blog.toyota.com/ not Vital, unless the blog
(US) with the public.
is specified in the query

4.1.4 Other Important Vital Concepts

Most queries do not have Vital webpages. Here are situations for which there is no Vital page.

 The query does not have a dominant interpretation.
 The query is not an entity or is not a navigation query.
 No official website or webpage exists for the entity.
 No person or entity can “own” the topic of the query.

Here are some examples of queries that do not have Vital pages:


Query Vital Page Description
There is no dominant interpretation. The following entities are all common
interpretations. Each interpretation has an official homepage, but none is Vital since
there is no dominant interpretation.
[ADA], No Vital page
English (US) is possible
Americans with Disabilities Act
American Dental Association
American Diabetes Association
This is an information query. Knitting is an activity anyone can do and that anyone
[knitting], No Vital page
can create a website for. There is no one official source for knitting information. No
English (US) is possible
one can own this topic.
[diabetes], English No Vital page This is an information query. No person or entity can claim ownership of the query
(US) is possible [diabetes].
[ipod reviews], No Vital page [ipod] is an entity query, but [ipod reviews] is not. [ipod reviews] is an information
English (US) is possible query. Users are looking for information that many sites can provide.
[how old is britney No Vital page [Britney Spears] is an entity query, but [how old is britney spears] is not. This is an
spears?], English (US) is possible information query. Users are looking for information that many sites can provide.

Some entities maintain official homepages on multiple domains. All such pages are Vital. Here are some examples.

Likely User
Query English (US) Vital Pages Description
Intent
[barnes and Navigate to http://www.barnesandnoble.com/ Multiple Vital URLs for the official homepage of this
noble], English the official http://www.bn.com company. These are different domains with the same
(US) homepage http://www.books.com owner; the landing pages are the same.

http://www.jcpenney.com/jcp/defaul
Navigate to Multiple Vital URLs for the official homepage of this
[penneys], t.aspx
the official company. These are different domains with the same
English (US) http://www.jcpenny.com/jcp/default.
homepage owner; the landing pages are the same.
aspx
Navigate to Multiple Vital URLs for the official homepage of this
[cheaptickets], http://www.cheaptickets.com/
the official company. These are different domains with the same
English (US) http://www.cheapticket.com/
homepage owner; the landing pages are the same.

Important: Often, the URL of the official homepage of an entity will contain the query terms. For example, the Vital
page for [ibm], English (US) is http://www.ibm.com. However, exact domain matches are not automatically Vital.

Sites claiming to be official may not actually be official sites. The Vital rating should NOT be assigned on the basis of
the URL alone. Just because the URL looks like the query does not mean that the page is Vital. Here are some
examples of URLs that look Vital, but are not:

Query Not Vital Description
No Vital page is possible for this query because it is an information query
[Diabetes],
http://www.diabetes.com and no one can claim ownership of it. Even though the URL “looks” Vital,
English (US)
it’s not.
[Ashley Tisdale], The landing page is not an official homepage for Ashley Tisdale; it is a fan
http://www.ashleytisdale.org/
English (US) site. This is her “real” official Vital page: http://www.ashleytisdale.com/

[simpsons], This is the “real” official Vital page for the query:
http://www.simpsons.com/
English (US) http://www.thesimpsons.com/index.html
The landing page has the words “Branson.com Official Website”. However,
it is the homepage of the Branson.com website. It is not the homepage of
[Branson,
the official city of Branson, Missouri website. The “real” official Vital page
Missouri], http://www.branson.com
for the city of Branson, Missouri is http://www.cityofbranson.org. Notice that
English (US)
the “real” city homepage has government-related links, while branson.com
has information about attractions, vacations, shows, etc.


4.1.5 Vital Pages and Geographic Location

When a page is Vital for the query, you will choose one of the following ratings:

 Appropriate Vital
 International Vital
 Other Vital

We have these three different Vital ratings because some official websites or pages have multiple versions for different
languages or countries.

When there is only one version of an official page for the query, it will always get the Appropriate Vital rating, no
matter what the task language or location is. Also, when the query is a URL or is clearly asking for a particular page,
that page is always Appropriate Vital, even if it doesn’t match the task language and location.

When there are multiple versions of an official page for different languages or countries, we want you to use your
judgment to assign one of the three Vital ratings:

• Use Appropriate Vital if the version of the official page seems right for the task location, or if the page is the
one “asked for” in the query.

• Use International Vital if the page is a “choose your language” or “choose your location” page. You can also
use International Vital for an English version that is designed to be an international page, helpful to many
users. For example, http://www.ebay.com/ would be the International Vital page for the query [ebay] for task
locations other than English (US). It would be Appropriate Vital for the English (US) task location.

• Use Other Vital if the language or location of the official page doesn’t match the task location, and a better
version exists. (If a better version for the task location doesn’t exist, then use Appropriate Vital). Please note
(as is shown in the examples below) that the Other Vital rating applies to homepages, not subpages.

Examples of different types of Vital ratings:

Query URL Rating Description

[Stanford], English (US) Stanford University has only one version of its
http://www.sta Appropriate
[Stanford], Chinese (CN) homepage. This page is Appropriate Vital for all
nford.edu/ Vital
[Stanford], Italian (IT) task locations and task languages.
Universidad de Sevilla (in Spain) has only one
[University of Seville], Spanish (ES)
http://www.us. Appropriate version (in Spanish) of its homepage. This page is
[University of Seville], Chinese (CN)
es/ Vital Appropriate Vital for all task locations and task
[University of Seville], Italian (IT)
languages.
[Microsoft.com], English (US) This is the page the user requested. This page is
http://www.mic Appropriate
[Microsoft.com], China (CN) Appropriate Vital for the query for all task locations
rosoft.com/ Vital
[Microsoft.com], Italian (IT) and task languages.
http://www.rola The French Open has three versions of its website:
[french open website], Spanish (ES)
ndgarros.com/ Appropriate French, Spanish, and English. The landing page is
[french open website], Spanish (MX)
es_FR/index.ht Vital the Spanish version. This page is Appropriate Vital
[french open website], Spanish (AR)
ml for all Spanish-speaking task locations.
The BBC has many versions of its website. The
[bbc], Arabic (EG)
http://www.bbc Appropriate landing page is the Arabic version. This page is
[bbc], Arabic (SA)
.co.uk/arabic/ Vital Appropriate Vital for all Arabic speaking task
[bbc], Arabic (MA)
locations.
Ikea has many country-specific versions of its
http://www.ikea Appropriate website. The landing page is the version for
[ikea], German (DE)
.com/de/de/ Vital Germany. This page is Appropriate Vital for the
German (DE) task language.


Query URL Rating Description
The United Nations website has six versions of its
[United Nations], English (US) website: Arabic, Japanese, English, French, Russian,
http://www.un. International
[United Nations], Chinese (CN) and Spanish. The landing page is a “choose your
org/ Vital
[United Nations], Italian (IT) language” page. It is International Vital for all task
locations and task languages.
[Ikea], English (US)
http://www.ikea International website. The landing page is a “choose your
[Ikea], Chinese (CN)
.com/ Vital location” page. It is International Vital for all task
[Ikea], Italian (IT)
locations and task languages.
[bbc], English (US) The BBC has many versions of its website. The
http://www.bbc
[bbc], Chinese (CN) Other Vital landing page is the Persian version, which is Other
.co.uk/persian/
[bbc], Italian (IT) Vital for non-Persian task locations.
[ikea], English (US) http://www.ikea Ikea has many country-specific versions of its
[ikea], Chinese (CN) .com/it/it/ Other Vital website. The landing page is the Italian version,
[ikea], Spanish (MX) which is Other Vital for other task locations.
[ikea], Spanish (MX)
http://www.ikea website. The landing page is the Australian version.
[ikea], English (UK) Other Vital
.com/au/en/ It is Other Vital for other task locations, even other
[ikea], English (US)
English-speaking task locations.

4.2 Useful

A rating of Useful is assigned to pages that are very helpful for most users. Useful pages should be high quality and
a good “fit” for the query. In addition, they often have some or all of the following characteristics: highly satisfying,
authoritative, entertaining, and/or recent (such as breaking news on a topic).

Useful pages are usually well organized and pages you trust. They are from information sources that seem reliable.
Useful information pages are not “spammy”.

Please note that more than one page can be rated Useful for a query. Please see the [csco], English (US) and
[meningitis symptoms], English (US) examples in Section 4.2.1.

4.2.1 Examples of Useful Pages

Query Likely User Intent Useful Pages Explanation
Find the answer to this http://www.cincinnatichildren Page on an authoritative website that
[is poison oak contagious?],
question. This is an s.org/health/info/allergy- answers this question very well and
English (US)
information query. asthma/diagnose/ivy.htm would be helpful for most users.
Read a review for this Webpage with over 300 reviews for
[sea salt Berkeley review], http://www.yelp.com/biz/_v4
restaurant. This is an this seafood restaurant. This page
English (US) Sq44bRYpj32unclB0EA
information query. would be helpful for most users.
Purchase tickets to a Reputable site on which to complete
[broadway tickets], http://www.ticketmaster.com
Broadway show. This is this transaction. This page would be
English (US) /broadway
an action query. helpful for most users.

http://finance.yahoo.com/q?
CSCO is the stock symbol for the
d=t&s=CSCO
Cisco Corporation. These pages are
Find stock quote
from well-known websites and are all
[csco], information for Cisco. http://money.cnn.com/quote/
basically the same, providing the
English (US) This is an information quote.html?symb=CSCO
same stock charts, trading
query.
information, etc. These pages would
http://finance.google.com/fin
be helpful for most users.
ance?client=ob&q=CSCO


General guidelines 2011

More Related Content

General guidelines 2011