Skilled Data Engineer / Big Data Engineer with 3 years of outstanding performance record in designing and implementing efficient ETL processes. Proficient in Azure ,Spark, Python, SQL, Hadoop, HDFS, Hive , Hbase ,Databricks, Delta lake , Azure Data Factory ( ADF ) | B. Tech in Information Technology from VIT Vellore | Actively seeking new opportunities | Serving Notice Period | Immediate Joiner |• Currently involved in a transformation project to migrate data files from on-premises to AzureCloud, including loading files into ADLS Gen2 and initiating the conversion of transformationlogic to Snowflake.• Improved data processing efficiency by developing and optimizing PySpark code, converting extract creation from Hive to PySpark, which increased speed by 63% and optimized Spark queriesto enhance extract creation performance by 20%.• Automated data loading processes with shell scripts, achieving a 60% reduction in manual effortand a 70% decrease in errors, while also applying custom PySpark transformations to ensure dataaccuracy and consistency.• Developed complex SQL queries for business logic and validated data, proactively resolving operational issues to ensure continuous data flow and minimal business interruptions.• Contributed to the enhancement• Managed OPCC application for AT&T client, specializing in Financial data, ensuring consistent data availability and integrity.• Enhanced data loading processes and Created new shell scripts to automate extract generation and Data loading resulting in a 60% reduction in manual effort and a 70% decrease in errors.• Collecting the data from from various systems and transforming it using custom Pyspark code as per the requirements.• Creating complex Business Logic into SQL Query.• Conducted performance tuning on Spark queries, optimized extract creation by 20%.• Validating the ingested and transformed data.• Successfully preempted and resolved daily operational setbacks, guaranteeing seamless data flow and minimizing business interruptions by analyzing problems and providing efficient client solutions.• Converted Extract creation from Hive to pyspark and increase the speed of extract creation by 63% .• Developed and optimized simple stored procedures in Oracle to execute business logic, enhancing data processing work flows.