Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
28 views

Hands-On Python Programming With HTTPX

The document provides an overview of using the HTTPX Python package to interact with various HTTP services and REST APIs. It covers basic HTTP concepts and methods, making requests with HTTPX, and handling redirects.

Uploaded by

secure.test987
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
28 views

Hands-On Python Programming With HTTPX

The document provides an overview of using the HTTPX Python package to interact with various HTTP services and REST APIs. It covers basic HTTP concepts and methods, making requests with HTTPX, and handling redirects.

Uploaded by

secure.test987
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 46

Click to edit Master title style

Hands-On Python Programming with HTTPX


Interact with various HTTP services and REST APIs
using this modern package

Nick Russo
Network and Automation Engineer
http://njrusmc.net
About This Class
Click to edit Master title style
• Lots of
• Pragmatic examples
• Hands-on demonstrations
• Absolutely no
• Academic "foo" and "bar" examples
• Gimmicks and trivia
• Have questions? Please:
• Use Q&A widget, not attendee/group chat
• Keep them timely and relevant
• Don't ask about your specific job
Section 1: HTTP Refresher
Click to edit Master title style
Basic request/response structure

Request methods

Return codes

Use cases
HTTP for Kindergarteners
Click to edit Master title style
• Stuff Transport Protocol (Russ White)
• Data exchange on the web

• Variety of operations/methods
• Create, read, update, and delete (CRUD)

• Variety of payload types and real-life applications


• Browsing, API exchanges, file transfer, etc.
HTTP Components
Click to edit Master title style
• A client sends requests, a server sends responses

• Both messages have:


• A request line (request) or status line (response)
• Headers
• An optional body, separated by blank line
Common HTTP Methods/Operations
Click to edit Master title style
• GET: Read a resource
• POST: Create a resource
• PUT: Update a resource
• DELETE: Delete a resource (together, CRUD)

• Less common
• HEAD: Get the headers only; no body
• PATCH: Update part of a resource
HTTP Status Codes – "Successes"
Click to edit Master title style
• 100 – 199: Continue
• 200 – 299: Success
• 200: OK
• 201: Created (new resource)
• 202: Accepted (asynchronous OK)

• 300 – 399: Redirect


• 301: Moved Permanently
• 307: Temporary Redirect
HTTP Status Codes – "Errors"
Click to edit Master title style
• 400 – 499: Client errors
• 400: Bad Request
• 401: Unauthorized (authentication)
• 403: Forbidden (authorization)
• 404: Not Found
• 405: Method not allowed
• 500 – 599: Server errors
• 500: Internal Server Error
• 501: Not implemented
HTTP Transaction Example - Success
Click to edit Master title style
Client Server

HTTP/1.1 200 OK
GET / HTTP/1.1
Content-Type: text/html
Host: njrusmc.net
Content-Length: 1796
Accept: */*
Server: AmazonS3

Request/status line <!DOCTYPE html>


Headers <html
xmlns="http://www.w3.org/
Body
1999/xhtml">
HTTP Transaction Example - Error
Click to edit Master title style
Client Server

GET /bad HTTP/1.1 HTTP/1.1 404 Not Found


Host: njrusmc.net Content-Type: text/html
Accept: */* Content-Length: 338
Server: AmazonS3

Request/status line <html>


Headers <head><title>404 Not
Body Found</title></head>...
Section 2: Exploring HTTPX Basics
Click to edit Master title style
HTTPX package overview and usage

Installation and documentation

Basic operations using REPL and scripts


What is Python HTTPX?
Click to edit Master title style
• Python HTTPX is an HTTP client
• Easy way to interact with web services
• Universally applicable across use-cases

>>> import httpx


>>> resp = httpx.get("http://njrusmc.net")
>>> resp.status_code
200
>>> len(resp.headers)
8
>>> resp.text
'<!DOCTYPE html>\n<html xmlns=...'
Issuing HTTP Requests with HTTPX
Click to edit Master title style
• Two main options

• Generic function:
• httpx.request(method="get")
• httpx.request(method="post")

• Specific functions:
• httpx.get(), httpx.post(), httpx.put(), etc.
• Simple wrappers for the generic function
Specifying Headers
Click to edit Master title style
• Headers are key-value pairs
• Great fit for a Python dictionary
• Often carries API keys/tokens, accepted encodings, etc.
• Very application-dependent

>>> import httpx


>>> my_headers = {"Accept": "text/html"}
>>> resp = httpx.get("http://njrusmc.net",
headers=my_headers)
>>> resp.headers["Content-Type"]
'text/html'
Specifying Query Parameters
Click to edit Master title style
• Query parameters are key-value pairs
• Great fit for a Python dictionary (again)
• Appended to URL in "?key=value&key=value" format
• Often not required; just a selection/filtering technique

>>> import httpx


>>> params = {"limit": 5, "type": "tech"}
>>> resp = httpx.get("http://njrusmc.net",
params=params)
>>> resp.request.url
URL('http://njrusmc.net/?limit=5&type=tech')
Specifying the Body
Click to edit Master title style
• Can use "data" to specify key-value pairs or plain-text
• Other techniques exist (discussed later)

>>> import httpx


>>> body = {"name": "nick", "kids": 2}
>>> resp = httpx.post("http://njrusmc.net", data=body)
>>> resp.request.content
b'name=nick&kids=2'

>>> resp = httpx.post("http://njrusmc.net", data="hi")


>>> resp.request.content
b'hi'
It's Not Over!
Click to edit Master title style

• Many more options to be revealed in code examples


• SSL verification
• Long-lived TCP sessions
• Content caching
• REST API interaction
• File upload/download
Installing HTTPX using pip
Click to edit Master title style
• Python package manager
• Works like yum/apt for Linux distributions
• "pip install httpx" (includes dependencies)

$ pip install httpx


Collecting httpx
Using cached httpx-0.23.0-py3-none-any.whl (84 kB)
(...)
Installing collected packages: httpx-0.23.0
A simple Python Script using HTTPX
Click to edit Master title style
• Typically, just "import httpx" and go

$ cat simple_script.py
import httpx
resp = httpx.get("http://njrusmc.net")
print(resp.status_code, resp.reason)
print(resp.text)

$ python simple_script.py
200 OK
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
(...)
Python Logging
Click to edit Master title style
• Set basic configuration
• Timestamp formatting
• Log level

• Create a "logger" object

• Call the logger's methods to write logs


• logger.info(msg), logger.debug(msg), etc.
Python Debugger (pdb)
Click to edit Master title style
• Four core actions
• list (l): reveal the code +/- 5 lines
• next (n): step over function
• step (s): step into function
• continue (c): run until next breakpoint

• Add breakpoint
• "breakpoint()" in py3.7+
• "import pdb; pdb.set_trace()" in py3.6-
Section 3: Handling HTTP Redirection
Click to edit Master title style
Real-life project from my private business

Follow or ignore HTTP redirection

Verify nuanced behavior with pdb


Handling Redirects
Click to edit Master title style
• Identified by HTTP redirects (status code 3XX)
• Target URL carried in "location" header

• HTTPX does not follow redirects by default!


• Set explicitly with follow_redirects=True
• Tracks redirected requests in "history" list

• Design choice
• Disable for security and step-by-step inspection
• Enable for ease of use, especially for trusted use-cases
What is tweeter-lite?
Click to edit Master title style
• Validates a Twitter message (called tweets)

• Start with library of known-good tweets

• Validates two important components


• Message is not too long
• URL in message is reachable

• Full version is more complex (but irrelevant today)


Section 4: Basic REST API Theory and Basic REST
Click to edit Master title style
API theory and interaction with Cisco SD-WAN

REST architectural style

SSL validation

HTTP body formats

Persistent sessions

Cisco SD-WAN (DevNet sandbox) demonstration


What is REST?
Click to edit Master title style
• Representational State Transfer

• An architectural style for web service interaction

• Not ...
• A protocol
• A framework
• A language
• A package
REST Constraints
Click to edit Master title style
• To be RESTful, a system must have/be:
• Uniform interface: resources directly accessible
• Client-server: separate user interface from data storage
• Stateless: requests are independent; no server state
• Cacheable: temporarily storable for improved performance
• Layered system: hierarchical servers, load balancers, etc.
• Code on demand (optional): server extends functionality of
client at runtime by sending code (JavaScript, etc.)
Why HTTP Is Usually a Good Fit
Click to edit Master title style
• Most constraints are met using HTTP
• Don't HAVE to use HTTP, but it's a natural fit
• One URI per resource à HTTP URL in a request
• CRUD operations easily map to HTTP methods
• Client-server and load balancing are inherent
• Caching can be public (CDN) or private (temp Internet files)
HTTP Body: Using "data"
Click to edit Master title style
• Relates to HTTP bodies

• If "data" is a dict, unpacks to HTML key/value pairs

• Automatically adds proper Content-Type header

• Commonly used for webform interaction

• Can also be plain-text (no transformation/header)


HTTP Body: Using "data" Example
Click to edit Master title style
>>> import httpx
>>> resp = httpx.post(
"http://njrusmc.net",
data={"name": "nick", "age": 36}
)

# request failed, but it doesn't matter


>>> resp.request.content
b'name=nick&age=36'
>>> resp.request.headers["Content-Type"]
'application/x-www-form-urlencoded'
HTTP Body: Using "json"
Click to edit Master title style
• Looks similar at a glance

• Must be a dictionary, but serializes into JSON as text

• Automatically adds proper Content-Type header

• Commonly used for interacting with REST APIs


HTTP Body: Using "json" Example
Click to edit Master title style
>>> import httpx
>>> resp = httpx.post(
"http://njrusmc.net",
json={"name": "nick", "age": 36}
)

# Very different headers and body!


>>> resp.request.content
b'{"name": "nick", "age": 36}'
>>> resp.request.headers["Content-Type"]
'application/json'
What is Cisco SD-WAN?
Click to edit Master title style
• Software-defined Wide Area Network

• If you're not a network engineer, don't worry!


• "data" and "json" together
• Persistent sessions
• Authentication/error handling
• TLS certificate checking

• Free and public!


• https://developer.cisco.com/sdwan/learn/
Section 5: Intermediate REST API Interaction
Click to edit Master title style
with pokeapi
Programmatic data collection and exploration

Navigating hierarchical data structures

HTTP pagination for large/unknown data sets


What is pokeapi?
Click to edit Master title style
• A free, read-only (GET) REST API

• Features 1,000+ Pokémon characters

• Tons of data, JSON-encoded, and well-documented

• Hosted at https://pokeapi.co/api/v2
Our Goals
Click to edit Master title style
• Project #1: Get details from some Pokémon
• Grab small number of Pokémon once
• Programmatically drill deeper into them

• Project #2: Get all Pokémon sequentially


• Use iterative HTTP pagination to "catch them all" J
• Grab large number of Pokémon with each request
Section 6: Advanced REST API Interaction with
Click to edit Master title style
pokeapi and asyncio
Concurrency for faster performance

Handling API throttling errors


Introducing asyncio
Click to edit Master title style
• One of many "concurrency" strategies in Python

• Single process and single thread

• Switches execution between tasks when busy


• Great for IO-bound operations
• e.g., establishing network connections

• asyncio is complicated; we are keeping it simple today!


Terms and keywords
Click to edit Master title style
• New terms
• coroutine: A function marked as async to perform a job
• task: awaitable instance of a coroutine
• future: represents the eventual result of a task or task list

• New syntax in Python 3.7+


• async: Mark a function as a coroutine (other uses also exist)
• await: Wait for a task or future to complete
Our Goals
Click to edit Master title style

• Project #3: Get all Pokémon concurrently


• Use asyncio to optimize the previous project

• Unanswered questions
• What about API rate limits?
• How do we know when to stop?
Section 7: REST API Interaction Involving File
Click to edit Master title style
Transfers and Asynchronous API Processing
Complete view of business application

OOP-based composition design

File upload and download

Error and status checking

Method customization
What is py-auphonic?
Click to edit Master title style
• Python client for www.auphonic.com REST API

• Used for post-production of audio files

• Great for VoD/AoD, podcasts, and music

• Well-documented, reliable REST API


• https://auphonic.com/help/api/
My Workflow – Building a Preset
Click to edit Master title style
• Preset is a template for producing audio files

• Many options
• Encoding type (mp3, wav, etc.)
• Noise cancellation
• Loudness adjustment
• Audio smoothing
• Metadata (title, artist, album, track, etc.)
My Workflow – Building a Production
Click to edit Master title style
• A production is a container for files

• Can do one file per production, or many ("batch")

• Basic steps
• Create production from preset
• Upload file(s) into production
• Produce the audio
• Download the resulting file
Recap and Q&A
Click to edit Master title style
HTTP/REST fundamentals

Using HTTPX to solve complex problems

Minor options (headers, params, data, json)

Troubleshooting (logging, debugging)

Optimization (persistent sessions, asyncio)

Composition design with OOP


Additional Resources
Click to edit Master title style
• Website as code (extensive "requests" usage at present):
• Videos: http://njrusmc.net/video/video.html
• Process: http://njrusmc.net/ci_scripts/webcicd.html

• More Python Training by Me


• O'Reilly: Real-World Python By Example
• Pluralsight: Automating Networks with Python

• Twitter: @nickrusso42518

You might also like