 Hands on experience in Big Data projects using Hadoop, Hive, PySpark, AWS Glue, RedShift, AWS Athena, AWS EC2,EMR,AWS Lambda,AWS S3. Expertise in working on Agile (Scrum) methodologies, including Extreme Programming, Test-Driven Development and Agile Scrum. Good analytical skills with excellent knowledge of SQL. Experience using software version control tools (Git,Bitbucket) Good Experience in importing and exporting data between HDFS and Relational Database Management systems using Sqoop.  Good exposure on Data integration from external systems into existing Data Lake. Strong knowledge of Spark for handling large data processing with Python and Scala. Experienced in loading the huge data from local file system and HDFS to Hive.  Experience using different file formats like Text Files, AVRO Files and Parquet Files. Experienced in loading data to Hive partitions and bucketing.  Experience working with other engineers in defining data engineering best practices and leveraging software development life cycle best practices such as agile methodologies, coding standards, code reviews, source management, build processes, testing, and operations. Have a Strong customer focus and excellent interpersonal, verbal, and written communication skills.  Involved in Test Driven Development for unit testing (Data Validation Testing) by writing SQL queries and Pyspark code.

 Key role in the design of Spark scripts for Application  Developed Spark SQL scripts by using Python shell commands as per the requirement. Analyzed the SQL scripts and designed the solution to implement using Python. Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself. Implemented schema extraction for Parquet and Avro file Formats. Good experience with continuous Integration of application using Jenkins. Collaborated with the infrastructure, network, database, application and BI teams to ensure data quality and availability Experience optimizing ETL workflows, where data coming from different sources and it is processed.

 Analyze the Business requirements and User Stories. Preparing Technical Documents based on Requirement and transformation Logic and Action on Spark. Data is collected from RDBMS Sources and pushing into HDFS using Sqoop. Have used Spark Transformations and Actions logic to develop Spark Scripts. Have trained juniors on Sqoop,HIVE,Python and Spark concepts. Have automated process using Java and Python programming Developed scripts using Python Scripts for the analysis purpose. Developed Spark SQL Scripts in Python Language to migrate existing Hive Scripts Developed Spark SQL Scripts on Python language Review and guide to team juniors on technical logics. Maintaining all requirements and its approach login into Sharedrive. To Handle defect call and its update  To provide internal trainings on Spark and Hadoop parts in Wipro Company. Have handled team to deliver sprint tasks in timeframe.

 Developed jobs using Talend Studio and deploy in prod through Talend prod server. Scheduling through control M scheduling tool Provide support on daily basis. Data balancing between source and targets. Involved in testing in various phases like Unit Testing. Helping the Business users several times in clearing their queries. Involved in fixing the issues raised by either SIT or UAT or Production teams. Used Parameter files in the project.  Created jobs to check the record counts.