Gcp Data Engineer
Current• Played a key role in a data migration initiative from AWS to Google Cloud Platform (GCP), contributing to the successful transition of data infrastructure.• Developed and implemented a detailed migration strategy to transfer data from AWS S3 and Redshift to Google Cloud Storage (GCS) and Big Query.• Contributed to the development and implementation of data transformation processes, converting AWS-based data into Big Query-compatible formats to ensure data compatibility and accessibility.• Transformed and optimized SQL queries from AWS Redshift to Big Query SQL, improving performance and reducing costs.• Participated in the extraction, transformation, and loading (ETL) process, ensuring smooth migration of source tables while maintaining data integrity throughout.• Collaborated in utilizing Info Works Replicator to replicate data from AWS source systems to Google Cloud Storage, facilitating reliable and scalable data transfer.• Identified opportunities for optimizing the data pipeline, proposing solutions to enhance the efficiency of data processing and reduce latency.• Developed scripts and automated processes for data synchronization between MongoDB and SQL databases during the migration, ensuring consistent and up-to-date data across systems.• Optimized query performance and resource allocation in Big Query by partitioning tables and applying best practices for data modeling.• Leveraged PySpark to process and transform large datasets, ensuring efficient handling of data throughout the migration pipeline and maintaining data quality.• Improved Big Query SQL queries using advanced techniques, resulting in significant performance gains and faster query execution times.