Data Engineer
CurrentWorked extensively with AWS S3 for scalable and secure data storage, managing data lakes, and optimizing data retrieval performance. Implemented AWS DynamoDB to develop high-performance NoSQL database solutions and crafted ETL workflows using AWS Glue to accurately extract, transform, and load data into AWS Redshift and PostgreSQL. Integrated streaming data sources with AWS Kinesis for high-velocity ingestion into EMR and enforced secure user access management with IAM.Automated ETL processes and data transformations with Python, enhancing efficiency and reducing manual effort. Developed complex data processing solutions using PySpark, and leveraged Databricks for Apache Spark-based processing, including data transformations, machine learning, and real-time analytics. Designed and optimized distributed data processing pipelines with Spark and PySpark, and fine-tuned Spark jobs and real-time streaming applications for optimal performance.Worked with Hadoop ecosystem technologies (HDFS, MapReduce, Hive, Sqoop), optimizing workflows through Databricks. Managed NoSQL databases like MongoDB and Cassandra, performed maintenance and performance tuning for MySQL and PostgreSQL databases, and developed data integration processes to ensure consistency across systems. Utilized Snowflake’s time travel features with Databricks for historical data analysis, and created secure visualizations and dashboards in Tableau. Employed Git for version control and JIRA for project management, applying Agile and Scrum methodologies to improve team collaboration.