This solution provides an automated, serverless way to redact sensitive data from PDF files using Google Cloud Services like Data Loss Prevention (DLP), Cloud Workflows, and Cloud Run.
-
Updated
Jul 19, 2024 - HCL
Google BigQuery enables companies to handle large amounts of data without having to manage infrastructure. Google’s documentation describes it as a « serverless architecture (that) lets you use SQL queries to answer your organization's biggest questions with zero infrastructure management. BigQuery's scalable, distributed analysis engine lets you query terabytes in seconds and petabytes in minutes. » Its client libraries allow the use of widely known languages such as Python, Java, JavaScript, and Go. Federated queries are also supported, making it flexible to read data from external sources.
📖 A highly rated canonical book on it is « Google BigQuery: The Definitive Guide », a comprehensive reference.
Another enriching read on the subject is the inside story told in the article by the founding product manager of BigQuery celebrating its 10th anniversary.
This solution provides an automated, serverless way to redact sensitive data from PDF files using Google Cloud Services like Data Loss Prevention (DLP), Cloud Workflows, and Cloud Run.
This is a demo project to use Terraform to manage BigQuery scheduled queries with Cloud Build CI/CD
Yelp Data Processing Pipeline on GCP
This project uses Terraform to deploy a BigQuery Data Clean Room on Google Cloud
Final project for DataTalks.Club Data Engineering bootcamp
...an automated data pipeline that retrieves cryptocurrency data from the CoinCap API, processes and transforms it for analysis, and presents key metrics on a near-real-time dashboard
Use GCP Datastream to incrementally load PostgreSQL to BigQuery
A terraform module to copy BigQuery datasets across regions
Dataflow job subscriber to PubSub subscription. It takes message from subscription and push it into BigQuery table.
terraform-bigquery-googlesheet
Automatic Anomaly Decetor
A IaC script to ingest and process messages containing data of trips taken by vehicles.
Terraform module for BigQuery sink connector on Aiven KafkaConnect cluster
Simple HTTP endpoint for telemetry data type events in GCP.
Here we are have automated the entire process of Creation of Google Cloud Storage buckets to store that that we are transferring via various transfer services, as of now Transfer from other Google cloud Storage buckets and public http/https links are supported, finally all this data is being processed into BigQuery for observations and Analysis.
Released May 19, 2010