Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning Talk)
•
6 likes•6,228 views
Monitoring should be part of your solution, not a problem. This lightening talk takes a brief look at the ideas behind Inclusive Monitoring and how to use them with Python.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning Talk)
Similar to Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning Talk) (20)
More from Brian Brazil
More from Brian Brazil (20)
Recently uploaded
Recently uploaded (20)
Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning Talk)
- 1. Better Monitoring for Python Inclusive Monitoring with Prometheus
- 2. Monitoring is often a Problem, not a Solution Themes common among companies I talk to: ● Monitoring tools are limited, both technically and conceptually ● Tools don’t scale well and are unwieldy to manage ● Operational practices don’t align with the business Result: Engineers continuously woken up for non-issues, get fatigued
- 3. Fundamental Challenge is Limited Visibility
- 4. Prometheus Inspired by Google’s Borgmon monitoring system. Started in 2012 by ex-Googlers working in Soundcloud as an open source project. Mainly written in Go. Publically launched in early 2015. 100+ companies using it including Digital Ocean, GoPro, Apple, Red Hat and Google.
- 7. Monitor as a Service, not as Machines
- 8. Inclusive Monitoring Don’t monitor just at the edges: ● Instrument client libraries ● Instrument server libraries (e.g. HTTP/RPC) ● Instrument business logic Library authors get information about usage. Application developers get monitoring of common components for free. Dashboards and alerting can be provided out of the box, customised for your organisation!
- 9. Let’s Talk Code pip install prometheus_client from prometheus_client import Summary, start_http_server REQUEST_DURATION = Summary('request_duration_seconds', 'Request duration in seconds') @REQUEST_DURATION.time() def my_handler(request): pass // Your code here start_http_server(8000)
- 10. Multiple Dimensions (No Evil Twins Please) from prometheus_client import Counter REQUESTS = Summary('requests_total', 'Total requests', ['method']) def my_handler(request): REQUESTS.labels(request.method).inc() pass // Your code here
- 11. Exceptional Circumstances In Progress from prometheus_client import Counter, Gauge EXCEPTIONS = Counter('exceptions_total', 'Total exceptions') IN_PROGRESS = Gauge('inprogress_requests', 'In progress') @EXCEPTIONS.count_exceptions() @IN_PROGRESS.track_inprogress() def my_handler(request): pass // Your code here
- 12. Getting Data Out from prometheus_client import start_http_server if __name__ == '__main__': start_http_server(8080) Also possible with Django, Twisted etc.
- 13. Oh Noes, My Vendor Lock In! from prometheus_client.bridge.graphite import GraphiteBridge gb = GraphiteBridge(('graphite.your.org', 2003)) gb.start(10.0) # Push every 10 seconds Exposition to Prometheus doesn’t use a special API, can be hooked into whatever monitoring system you have. Instrument once, work with everything!
- 14. Running Prometheus Grab binary from http://www.robustperception.io/prometheus-nightly-binaries/ Put config in prometheus.yml: scrape_configs: - job_name: python target_groups: targets: [localhost:8000] Run it: ./prometheus
- 15. Dashboards
- 16. Prometheus is as Powerful as Python Can slide, dice, aggregate and do math to produce your alerts If it’s computable, Prometheus can compute it* Alert on what matters, not on what your monitoring limit you to! * Halting, tractability and sanity not guaranteed
- 17. Links! Blog Posts: www.robustperception.io/tag/prometheus Demo: demo.robustperception.io Python Client: https://github.com/prometheus/client_python Project Website: prometheus.io