A framework for real-life ML, AI, and data science
Open-source Metaflow makes it quick and easy to build and manage real-life ML, AI, and data science projects.
-
Modeling
Use any Python libraries for models and business logic. Metaflow helps manage library dependencies, locally and in the cloud. -
Deployment
Deploy workflows to production with a single command and integrate with other systems through events. -
Versioning
Metaflow tracks and stores variables inside the flow automatically for easy experiment tracking and debugging. -
Orchestration
Create robust workflows in plain Python. Develop and debug them locally, deploy to production without changes. -
Compute
Leverage the cloud to execute functions at scale. Use GPUs, multiple cores, and large amounts of memory as needed. -
Data
Access data from data warehouses. Metaflow flows data across steps, versioning everything on the way.
Metaflow is used by these companies and hundreds of others
Metaflow is built for ML/AI engineers and data scientists, not just for machines
-
Develop with Metaflow
Explore with notebooks, develop with Metaflow, and test and debug locally. Results are stored and tracked automatically for easy analysis.
-
-
Scale out to the cloud
Break out from the confines of a laptop or a single notebook. Scale out easily to the cloud, utilizing GPUs, multiple cores, and multiple instances in parallel. Metaflow organizes the work for easy collaboration on the way.
-
-
Deploy to production confidently
Deploy experiments to production with a single click without changing anything in the code. Make flows react to updating data and other events automatically.
Bring your own Cloud
Get started easily on a laptop. When you are ready to scale, deploy the Metaflow stack on your cloud account or on-premise Kubernetes cluster. Metaflow integrates seamlessly with your existing infrastructure, security, and data governance policies.
To get a taste of Metaflow in the cloud, try Metaflow Sandbox in the browser.
Battle-hardened at Netflix
Metaflow was originally developed at Netflix to address the needs of developers and data scientists who work on demanding real-life ML, AI, and data projects. Netflix open-sourced Metaflow in 2019.
Today, Metaflow is used by hundreds of companies across industries, powering diverse projects from state-of-the-art GenAI and compute vision to business-oriented data science, statistics, and operations research.
How leading ML, AI, and data science teams use Metaflow
-
Developing safe and reliable ML products at 23andMe
Our complex, multi-stage workflows are codified and orchestrated using Metaflow.
-
Accelerating ML within CNN
Our data science team believes they were able to test twice as many models in Q1 2021 as they did in all of 2020.
-
Accelerating experimentation with MLOps
Metaflow helped us avoid the anti-pattern of needing to push code to find out if something works.
-
Improving Data Science Processes to Speed Innovation at Realtor.com
The team has shaved months off the time it takes to build a productionized machine learning model.
Recent Updates
-
Run and deploy flows programmatically
New APIs allow you to run and deploy Metaflow in notebooks and scripts
-
New Documentation for Compute Patterns
Learn about various patterns of scalable compute with Metaflow.
-
Support for AWS Trainium
Train and fine-tune large language models and other generative AI models on AWS Trainium.
-
Real-Time, Dynamic Cards
Build observable ML/AI systems with cards that update in real-time.
-
Include any PyPI packages
Install dependencies from PyPI as well as Conda in your Metaflow steps.
-
Access Secrets Securely
Connect to external services securely using the new @secrets decorator.
-
Build reactive production systems with Metaflow
Metaflow 2.9 allows you to trigger workflows based on real-time events.
-
Load and process tabular data fast with Metaflow
Apache Arrow and Metaflow.S3 make it easy to process data fast.
-
Using Metaflow to train a large language model
Learn how to use Metaflow for demanding GPU tasks.
-
Metaflow supports Apache Airflow
Develop with Metaflow, deploy on your existing Apache Airflow servers.
-
Metaflow on Google Cloud
Deploy and operate Metaflow on GCP and all other major clouds.
-
Documentation updated
Enjoy a clearer learning path, more content.
-
Metaflow on Azure
Deploy and operate Metaflow on Microsoft Azure.
-
New Metaflow Sandboxes
Test Metaflow and the infrastructure behind it in the browser.
