03 Laws

9/29/2020
Text Technologies for Data Science

INFR11145
Laws of Text
Instructor:
Walid Magdy
30-Sep-2020
Pre-Lecture
• Lab 0: How did it go?
• Lab 1: next week, important to everyone
• Join Piazza (search for TTDS)
• Live lecture
2
Walid Magdy, TTDS 2020/2021
1
9/29/2020
Reminder: Skills to be gained

• Working with large text collections
• Few shell commands
• Python/Perl regex
• IR tools: Solr
• TEAM WORK
3
Lecture Objectives
• Learn about some text laws
• Zipf’s law
• Benford’s law
• Heap’s law
• Clumping/contagion
• This lecture is practical
4
2
9/29/2020
You can try with me …

• Shell commands: cat, sort, uniq, grep
• Perl (or alternative)
• Excel (or alternative)
• Download the following:
• Bible: http://www.gutenberg.org/cache/epub/10/pg10.txt
5
Words’ nature
• Word → basic unit to represent text
• Certain characteristics are observed for the
words we use!
• These characteristics are very consistent, that
we can apply laws for them
• These laws apply for:
• Different languages
• Different domains of text
6
3
9/29/2020
Frequency of words
• Some words are very frequent
e.g. “the”, “of”, “to”
• Many words are less frequent
e.g. “schizophrenia”, “bazinga”
• ~50% terms appears once
• Frequency of words has
Log(frequency)
Frequency
hard exponential decay
rank
Log(rank)
7
Zipf’s Law:
• For a given collection of text, ranking unique
terms according to their frequency, then:
𝑟 × 𝑃𝑟 ≅ 𝑐𝑜𝑛𝑠𝑡
• 𝑟, rank of term according to frequency
• 𝑃𝑟 , probability of appearance of term
• 𝑃𝑟 ≅ 𝑐𝑜𝑛𝑠𝑡
𝑟
→𝑓 𝑥 ≅
1
𝑥
8
4
9/29/2020
Zipf’s Law:
Wikipedia abstracts Term Rank Frequency r x freq
the 1 5,134,790 5,134,790
→ 3.5M En abstracts of
in
2
3
3,102,474
2,607,875
6,204,948
7,823,625
a 4 2,492,328 9,969,312
is 5 2,181,502 10,907,510
𝑟 × 𝑃𝑟 ≅ 𝑐𝑜𝑛𝑠𝑡 → and
was
6
7
1,962,326
1,159,088
11,773,956
8,113,616
to 8 1,088,396 8,707,168
𝑟 × 𝑓𝑟𝑒𝑞𝑟 ≅ 𝑐𝑜𝑛𝑠𝑡 by 9 766,656 6,899,904
an 10 566,970 5,669,700
it 11 557,492 6,132,412
for 13 493,374 5,970,456
as 14 480,277 6,413,862
on 15 471,544 6,723,878
from 16 412,785 7,073,160
9
Practical
Collection # words File size
Bible 824,054 4.24 MB
Wiki abstracts 80,460,749 472 MB
10
10
5
9/29/2020
Distribution of first digit in frequencies?

1) Uniform → Term Rank Frequency
the 1 5,134,790
of 2 3,102,474
in 3 2,607,875
a 4 2,492,328
is 5 2,181,502
2) Exp decay → and 6 1,962,326
was 7 1,159,088
to 8 1,088,396
by 9 766,656
an 10 566,970
3) Normal → it 11 557,492
for 13 493,374
as 14 480,277
on 15 471,544
from 16 412,785
11
11
Benford’s Law:
• First digit of a number follows a Zipf’s like law!
• Terms frequencies
• Physical constants
• Energy bills
• Population numbers
• Benford’s law:
1
𝑃 𝑑 = log(1 + )
𝑑
12
12
6
9/29/2020
Practical
13
13
Heap’s Law:
• While going through documents, the number of new
terms noticed will reduce over time
• For a book/collection, while reading through, record:
• 𝑛: number of words read
• 𝑣: number of news words (unique words)
• Vocabulary growth:
v (vocabulary)
𝑣 𝑛 = 𝑘 × 𝑛𝑏
where, 𝑏 < 1
typically, 0.4 < 𝑏 < 0.7
n (words)
14
14
7
9/29/2020
Heap’s Law: shouldn’t it saturate?
• 𝑛 = 80+ million, but still growing

• Think about:
- spelling errors
- names
- emails
- codes
• Accurate for most collections,
but different 𝑘, 𝑏
• Not very accurate when 𝑛 is
small
15
15
Practical
16
16
8
9/29/2020
Clumping/Contagion in text
• From Zipf’s law, we notice:
• Most words do not appear that much!
• Once you see a word once → expect to see again!
• Words are like:
• Rare contagious disease
• Not, rare independent lightening
• Words are rare events, but they are contagious
17
17
Clumping/Contagion in text
• Wiki abstract collection
• Identify terms appeared only twice
• Measure distance between the two
occurrences of the terms:
𝑑 = 𝑛𝑜𝑐𝑐𝑢𝑟𝑒𝑛𝑐𝑒2 − 𝑛𝑜𝑐𝑐𝑢𝑟𝑒𝑛𝑐𝑒1
• Plot density function of 𝑑
density
• Majority of terms appearing

only twice appear close to
each other.
distance (d)
18
18
9
9/29/2020
Applying the laws

• Given a collection of 20 billion terms,
• What is the number of unique terms?
Heap’s law: 𝑣 𝑛 = 𝑘 × 𝑛𝑏 , assume 𝑘 = 0.25, 𝑏 = 0.5
➔ 𝑣 𝑛 = 0.25 × (20𝐵)0.5 ≅ 35𝑀
• What is the number of terms appearing once?

Zipf’s law ➔ ~17M appeared only once
19
19
Summary
• Text follows well-known phenomena
• Text Laws:
• Zipf
• Heap
• Contagion in text
• Try it on another language …
20
20
10
9/29/2020
Recourses
• Text book:
• Search engines: IR in practice → chapter 4
• Videos:
• Zipf’s law, Vsouce:
https://www.youtube.com/watch?v=fCn8zs912OE
• Benford’s law, Numberphile:
https://www.youtube.com/watch?v=XXjlR2OK1kM
• Tools:
• Unix commands for windows
https://sourceforge.net/projects/unxutils
21
21
Next Lecture
• Getting ready for indexing?
• Pre-processing steps before the indexing process
• Reminder: 5-10 mins break after L1

• Have a break, stretch, get food … etc.
• Ask questions on chat
• Questions on L1 are allowed before starting L2
• Mind teaser math problem (for fun)
22
22
11

03 Laws

Uploaded by

Copyright:

Available Formats

03 Laws

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

03 Laws

Uploaded by

Copyright:

Available Formats

9/29/2020

Text Technologies for Data Science

Reminder: Skills to be gained

• This lecture is practical

You can try with me …

Collection # words File size

Bible 824,054 4.24 MB

Wiki abstracts 80,460,749 472 MB

Distribution of first digit in frequencies?

Heap’s Law: shouldn’t it saturate?

• 𝑛 = 80+ million, but still growing

• Words are rare events, but they are contagious

• Majority of terms appearing

Applying the laws

• What is the number of terms appearing once?

• Try it on another language …

• Reminder: 5-10 mins break after L1

You might also like