etl

Star

Here are 4,068 public repositories matching this topic...

apache / airflow

Star

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Updated Nov 12, 2024
Python

airbytehq / airbyte

Star

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

Updated Nov 12, 2024
Python

apache / doris

Star

Apache Doris is an easy-to-use, high performance and unified analytics database.

bigquery real-time sql database spark hive hadoop etl snowflake olap query-engine redshift dbt elt iceberg hudi delta-lake lakehouse

Updated Nov 12, 2024
Java

dagster-io / dagster

Star

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

Updated Nov 12, 2024
Python

redpanda-data / connect

Star

Fancy stream processing made operationally mundane

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue data-engineering streaming-data stream-processor data-ops

Updated Nov 11, 2024
Go

mage-ai / mage-ai

Star

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

Updated Nov 12, 2024
Python

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.

Updated Nov 12, 2024
Rust

turbot / steampipe

Star

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

Updated Nov 8, 2024
Go

cloudquery / cloudquery

Star

The open source high performance ELT framework powered by Apache Arrow

Updated Nov 11, 2024
Go

apache / flink-cdc

Star

Flink CDC is a streaming data integration tool

mysql real-time kafka etl postgresql distributed batch data-integration schema-evolution elt flink cdc data-pipeline change-data-capture paimon

Updated Nov 12, 2024
Java

pathwaycom / pathway

Star

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

python rust streaming real-time kafka etl machine-learning-algorithms stream-processing data-analytics dataflow data-processing data-pipelines batch-processing pathway iot-analytics etl-framework time-series-analysis

Updated Nov 12, 2024
Python

rudderlabs / rudder-server

Star

Privacy and Security focused Segment-alternative, in Golang and React

Updated Nov 12, 2024
Go

orchest / orchest

Star

Build data pipelines, the easy way 🛠️

python docker kubernetes data-science machine-learning airflow cloud deployment jupyter etl ide pipelines self-hosted jupyterlab notebooks data-pipelines dag etl-pipeline orchest

Updated Jun 6, 2023
TypeScript

aws / aws-sdk-pandas

Star

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Updated Nov 11, 2024
Python

nucleuscloud / neosync

Star

Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.

mysql testing docker kubernetes golang open-source typescript etl reactjs nextjs postgresql faker self-hosted orchestration test-data-generator synthetic-data fine-tuning benthos synthetic-data-generation

Updated Nov 12, 2024
Go

quadratichq / quadratic

Star

Quadratic | Spreadsheet with Python, SQL, and AI

python webgl data-science data sql ai etl spreadsheet wasm data-engineering data-analysis quadratic

Updated Nov 12, 2024
Rust

blockchain-etl / ethereum-etl

Star

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

export bigquery aws csv sql etl ethereum transaction gcp google-cloud erc20 erc20-tokens blockchain-analytics erc721

Updated Aug 13, 2024
Python

apache / incubator-devlake

Star

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.