Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd Logo
Upload

Welcome to Scribd!

  • 252 views

    Beautifulsoup: Web Scraping With Python: Andrew Peterson

    Uploaded by

    tansoei
    The document discusses web scraping using BeautifulSoup in Python. It provides an overview of BeautifulSoup and how it can be used to extract data from HTML and XML documents. It also discusses using regular expressions and examples of extracting table data and writing it to a CSV file. Additionally, it covers scraping PDF files using pdfminer and performing actions on multiple files.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd

    Beautifulsoup: Web Scraping With Python: Andrew Peterson

    Uploaded by

    tansoei
    252 views43 pages
    The document discusses web scraping using BeautifulSoup in Python. It provides an overview of BeautifulSoup and how it can be used to extract data from HTML and XML documents. It also discusses using regular expressions and examples of extracting table data and writing it to a CSV file. Additionally, it covers scraping PDF files using pdfminer and performing actions on multiple files.

    Original Description:

    Beautiful Soup is a Python library for web scraping and analysis

    Original Title

    Beautiful Soup

    Copyright

    © Attribution Non-Commercial (BY-NC)

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document discusses web scraping using BeautifulSoup in Python. It provides an overview of BeautifulSoup and how it can be used to extract data from HTML and XML documents. It also discusses using regular expressions and examples of extracting table data and writing it to a CSV file. Additionally, it covers scraping PDF files using pdfminer and performing actions on multiple files.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    252 views43 pages

    Beautifulsoup: Web Scraping With Python: Andrew Peterson

    Uploaded by

    tansoei
    The document discusses web scraping using BeautifulSoup in Python. It provides an overview of BeautifulSoup and how it can be used to extract data from HTML and XML documents. It also discusses using regular expressions and examples of extracting table data and writing it to a CSV file. Additionally, it covers scraping PDF files using pdfminer and performing actions on multiple files.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 43

    Introduction Example Regex Other Methods PDFs

    BeautifulSoup: Web Scraping with Python


    Andrew Peterson

    Apr 9, 2013

    les available at: https://github.com/aristotle-tek/BeautifulSoup_pres

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Roadmap

    Uses: data types, examples... Getting Started downloading les with wget BeautifulSoup: in depth example - election results table Additional commands, approaches PDFminer (time permitting) additional examples

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Etiquette/ Ethics

    Similar rules of etiquette apply as Pablo mentioned: Limit requests, protect privacy, play nice...

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Data/Page formats on the web

    HTML, HTML5 (<!DOCTYPE html>)

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Data/Page formats on the web

    HTML, HTML5 (<!DOCTYPE html>) data formats: XML, JSON PDF APIs other languages of the web: css, java, php, asp.net... (dont forget existing datasets)

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    BeautifulSoup

    General purpose, robust, works with broken tags Parses html and xml, including xing asymmetric tags, etc. Returns unicode text strings Alternatives: lxml (also parses html), Scrapey Faster alternatives: ElementTree, SGMLParser (custom)

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Installation

    pip install beautifulsoup4 or easy_install beautifulsoup4 See: http://www.crummy.com/software/BeautifulSoup/ On installing libraries: http://docs.python.org/2/install/

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    HTML Table basics

    <table> Defines a table <th> Defines a header cell in a table <tr> Defines a row in a table <td> Defines a cell in a table

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    HTML Tables

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    HTML Tables

    <h4>Simple <table> <tr> <td>[r1, <td>[r1, </tr> <tr> <td>[r2, <td>[r2, </tr> </table>

    table:</h4>

    c1] </td> c2] </td>

    c1]</td> c2]</td>

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Example: Election data from html table

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Example: Election data from html table

    election results spread across hundreds of pages want to quickly put in useable format (e.g. csv)

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Download relevant pages

    website might change at any moment ability to replicate research limits page requests

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Download relevant pages

    I use wget (GNU), which can be called from within python alternatively cURL may be better for macs, or scrapy

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Download relevant pages

    wget: note the --no-parent option!

    os.system("wget --convert-links --no-clobber \ --wait=4 \ --limit-rate=10K \ -r --no-parent http://www.necliberia.org/results2011/results

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Step one: view page source

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Outline of Our Approach

    1 2

    identify the county and precinct number get the table:


    identify the correct table put the rows into a list for each row, identify cells use regular expressions to identify the party & lastname

    write a row to the csv le

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Open a page

    soup = BeautifulSoup(html_doc) Print all: print(soup.prettify()) Print text: print(soup.get_text())

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Navigating the Page Structure

    some sites use div, others put everything in tables.

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    nd all

    nds all the Tag and NavigableString objects that match the criteria you give. nd table rows: find_all("tr") e.g.: for link in soup.find_all(a): print(link.get(href))

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Lets try it out

    Well run through the code step-by-step

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions

    Allow precise and exible matching of strings precise: i.e. character-by-character (including spaces, etc) exible: specify a set of allowable characters, unknown quantities import re

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    from xkcd

    (Licensed under Creative Commons Attribution-NonCommercial 2.5 License.)


    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions: metacharacters

    Metacharacters: |. ^ $ * + ? { } [ ] \ | ( ) excape metacharacters with backslash \

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions: Character class

    brackets [ ] allow matching of any element they contain [A-Z] matches a capital letter, [0-9] matches a number [a-z][0-9] matches a lowercase letter followed by a number

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions: Repeat

    star * matches the previous item 0 or more times plus + matches the previous item 1 or more times [A-Za-z]* would match only the rst 3 chars of Xpr8r

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions: match anything

    dot . will match anything but line break characters \r \n combined with * or + is very hungry!

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions: or, optional

    pipe is for or abc|123 matches abc or 123 but not ab3 question makes the preceeding item optional: c3?[a-z]+ would match c3po and also cpu

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions: in reverse

    parser starts from beginning of string can tell it to start from the end with $

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions

    \d, \w and \s \D, \W and \S NOT digit (use outside char class)

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Regular Expressions

    Now lets see some examples and put this to use to get the party.

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Basic functions: Getting headers, titles, body

    soup.head soup.title soup.body

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Basic functions

    soup.b id: soup.find_all(id="link2") eliminate from the tree: decompose()

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Other Methods: Navigating the Parse Tree

    With parent you move up the parse tree. With contents you move down the tree. contents is an ordered list of the Tag and NavigableString objects contained within a page element. nextSibling and previousSibling: skip to the next or previous thing on the same level of the parse tree

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Data output

    Create simple csv les: import csv many other possible methods: e.g. use within a pandas DataFrame (cf Wes McKinney)

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Putting it all together

    Loop over les for vote total rows, make the party empty print each row with the county and precinct number as columns

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    PDFs

    Can extract text, looping over 100s or 1,000s of pdfs. not based on character recognition (OCR)

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    pdfminer

    There are other packages, but pdfminer is focused more directly on scraping (rather than creating) pdfs. Can be executed in a single command, or step-by-step

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    pdfminer

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    PDFs

    Well look at just using it within python in a single command, outputting to a .txt le. Sample pdfs from the National Security Archive Iraq War: http://www.gwu.edu/~nsarchiv/NSAEBB/NSAEBB418/

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Performing an action over all les

    Often useful to do something over all les in a folder. One way to do this is with glob: import glob for filename in glob.glob(/filepath/*.pdf): print filename see also an example le with pdfminer

    BeautifulSoup

    Introduction Example Regex Other Methods PDFs

    Additional Examples

    (time permitting): newspapers, output to pandas...

    BeautifulSoup

    You might also like