• Maintained data pipelines, integration, ETL processes, and data management delivery within a global data lake and data warehouse environment as a Data Engineer.• Collaborated with data engineers, analysts, and stakeholders to understand data requirements, providing support for SQL queries and scripts.• Developed applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation.• Implemented CI/CD pipelines using Azure DevOps, Jenkins, or GitLab CI/CD.• Built and maintained data pipelines to ingest and integrate data from various sources into PySpark environments. • Designed and implemented database architecture and schema.• Collaborated with development teams to understand data requirements.

• Implemented a serverless architecture using API Gateway, Lambda, and Dynamo DB, deploying AWS Lambda code from Amazon S3 buckets.• Developed file cleaners using Python libraries and utilized Python Libraries like Boto3, NumPy for AWS.• Resolved complex build and deployment issues with on-call support and strong troubleshooting and debugging skills.• Worked with AWS services like Lambda function, Athena, DynamoDB, Step functions, SNS, SQS, S3, IAM.• Worked with ETL tools to migrate data from various OLTP and OLAP databases to the data mart. • Involved in all phases of data mining, data collection, data cleaning, developing models, validation, and visualization.• Experienced with event-driven and scheduled AWS Lambda functions to trigger various AWS resources.

Generated SQL scripts for data manipulation, validation, and materialized views for remote instances. Developed Python scripts for data cleaning, transfer data, REST APIs, and extract data from on-premises to AWS S3.• Troubleshot database-level issues, handled errors, and performed performance tuning of queries and procedures.• Wrote Python scripts to parse XML documents and load data into databases.• Developed multiple Spark SQL jobs for data cleaning and loaded tables into HDFS to run select queries.

• Performed data analysis in Hive by creating tables, loading data, and writing Hive queries that run internally using MapReduce.• Loaded data from multiple sources (SQL, DB2, Oracle) into HDFS using Sqoop and loaded into Hive tables.• Use PyCharm IDE for Python development and Git for version control and repository management. • Used HBase APIs to get and scan events data stored in HBase.• Collected and aggregated large amounts of log data and staged data in HDFS for further analysis.• Worked with NoSQL databases including HBase, MongoDB, and Cassandra, designing HBase tables for time series data and avoiding region server hot spotting.

• Extensively analyzed data by querying in SQL and generating various PL/SQL objects, including stored procedures and functions.• Created new stored procedures and optimized existing queries and stored procedures for improved performance.• Scripted complex stored procedures using T-SQL to create metrics for data analysis.• Analyzed and built proof of concepts to convert SAS reports into Tableau or use SAS datasets in Tableau.• Wrote multiple SQL queries to analyze data and presented results using Excel, Access, and Crystal reports.• Experience with building data pipelines in python/Pyspark.

• Worked with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas.• Experience in building python DAG in Apache Airflow.• Developed stored procedures in MS SQL to fetch the data from different servers using FTP and processed these files to update the tables.• Developed a python script to validate the files in one S3 with same daily files in another bucket.• Designed and Developed ETL jobs to extract data from Salesforce replica.