Sr Data Engineer
Current• Worked extensively with AWS services like EC2, S3, VPC, ELB, Auto Scaling Groups, Route 53, IAM, CloudTrail, CloudWatch, CloudFormation, CloudFront, SNS, and RDS. • Developed Python scripts to parse XML, JSON files and load the data in AWS Snowflake Data warehouse. • Design and Develop ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, and Parquet/Text Files into AWS Redshift. • Performed several Banking Centre level performance analyses aimed at streamlining and optimizing the sales process and monitoring sales of certain high-value checking products. • Building a Scala and spark-based configurable framework to connect common Data sources like MYSQL, Oracle, Postgres, SQL Server, Salesforce, and big query and load it in big query. • Worked on documentation of all worked Extract. Transform and Load, Designed, developed and validated, and deployed the Talend ETL processes for the Data Warehouse team using PIG, Hive. • Applied required transformation using AWS Glue and loaded data back to Redshift and S3. • Experience in analyzing and writing SQL queries to extract the data in JSON format through Rest API calls with API Keys, ADMIN Keys, and Query Keys and load the data into the Data warehouse. • Designed and implemented ETL pipelines between from various Relational Data Bases to the Data Warehouse using Apache Airflow. • Worked on Data Extraction, aggregations, and consolidation of Adobe data within AWS Glue using PySpark. • Developed SSIS packages to Extract, Transform and Load ETL data into the SQL Server database from the legacy mainframe data sources. • Worked on Building data pipelines in airflow in GCP for ETL-related jobs using different airflow operators. • Worked on Postman using HTTP requests to GET the data from RESTful API and validate the API calls. • Created custom T-SQL procedures to read data from flat files to dump to SQL Server database using SQL Server import and export data wizard.