Senior Data Engineer
CurrentI have extensively utilized AWS components like EMR and EC2, integrating them with Big Data frameworks such as Hadoop YARN, MapReduce, Spark, and Hive. My expertise includes leveraging AWS Athena for data ingestion from S3 into RedShift, and developing Lambda functions using Python for deployment and public-facing websites. I’ve also implemented various serverless Lambda functions and API Gateways for efficient data submission and processing.In addition to my AWS skills, I have… Show more I have extensively utilized AWS components like EMR and EC2, integrating them with Big Data frameworks such as Hadoop YARN, MapReduce, Spark, and Hive. My expertise includes leveraging AWS Athena for data ingestion from S3 into RedShift, and developing Lambda functions using Python for deployment and public-facing websites. I’ve also implemented various serverless Lambda functions and API Gateways for efficient data submission and processing.In addition to my AWS skills, I have developed and maintained ETL processes using Informatica PowerCenter, including managing data masking with Informatica Persistent Data Masking. My experience extends to creating on-demand tables using AWS Glue and PySpark, and optimizing Spark applications for performance tuning. I’ve worked with Python, JSON, and Groovy scripts for deploying StreamSets pipelines and handling XML document parsing.Furthermore, I have expertise in designing and optimizing SQL queries, managing NoSQL databases like MongoDB and Cassandra, and handling MySQL and PostgreSQL databases. I am proficient in tools such as Apache Airflow for orchestrating data workflows, and have developed Tableau and Power BI reports for effective data visualization. My experience with version control tools like Git and JIRA has ensured effective project management and timely issue resolution. Show less