Senior Data Engineer
Current• As a data engineer, I implemented Azure Synapse for unified analytics, integrating both structured and unstructured data seamlessly.• Executed T-SQL queries and deployed serverless SQL pools in Azure Synapse for cost-effective analytics as a data engineer.• Orchestrated scalable and flexible data warehousing solutions to handle large datasets effectively in my data engineering role.• Ensured data security and compliance, and collaborated within Synapse Studio to streamline data pipeline management and monitoring as a data engineer.• Implemented Snowpark UDFs for custom data transformations and contributed to workspace collaboration, fostering teamwork and insights sharing as a data engineer.• Leveraged extensive AWS cloud platform experience and applied Spark RDD, DataFrame, DataSet, and Spark Streaming knowledge in my data engineering projects.• Developed Spark applications using Python to handle data from various RDBMS and streaming sources, and improved existing algorithms in Hadoop as a data engineer.• Employed Spark Context, Spark-SQL, Spark MLib, and Spark YARN effectively, and used Spark Streaming APIs for real-time data transformations as a data engineer.• Developed a learner data model using Kafka for real-time data ingestion to Cassandra and crafted a Kafka consumer API in Python as a data engineer.• Processed XML messages using Kafka, developed preprocessing jobs with Spark DataFrames to flatten JSON• Experienced in live real-time processing and core job execution using Spark Streaming with Kafka, and migrated an existing on-premises application to AWS as a data engineer.• Maintained the Hadoop cluster on AWS EMR, loaded data into S3 buckets using AWS Glue and PySpark and configured Snowpipe to pull data into Snowflake tables as a data engineer.• Managed data storage in Snowflake's staging area, created ODI interfaces for Snowflake DB, and consolidated Data warehouses into Amazon Redshift, highlighting strategic data engineering initiatives.