Created custom Kafka producers and consumers to seamlessly integrate external data sources with internal systems, enabling real-time analytics.Developed Python scripts for data extraction, transformation, and loading (ETL) from TXT and CSV files into Google BigQuery.Implemented efficient partitioning and clustering strategies in BigQuery for optimized query performance.Airflow to design and orchestrate complex ETL workflows, including data extraction from MySQL databases and loading into BigQuery. Developed Cloud Functions, to trigger the workflow in airflow upon files delivered to GCS storage.Collaborated with cross-functional teams to gather requirements, analyze data needs, and design effective data models.