● Developed a ETL framework for data ingestion and data extraction to handle the different types of files into different source systems - Hive/PostgreSQL/Redshift tables and generates the extracts according to the business needs.● Experience in working with different types of file loads (TXT, CSV) to process the files from S3 bucket and load it into the raw tables in hive/PostgreSQL/Redshift.● Developed Spark data frames and Spark SQL scripts using Scala for faster data processing.● Involved in working on various columnar storage and file formats like CSV, Parquet, AVRO and ORC.● Involved in working on Historical and Incremental runs for monthly, daily and weekly load.● Experience in working in the Agile environment to deliver the development tasks/ story cards on time without any delay.● Good Experience in understanding the business requirements to load the data into the target tables.● Used Hue for UI based Hive and Impala query and used beeline for creating the external/managed tables in Hive.● Used Oozie for scheduling the jobs and developed oozie coordinator and workflow xml to automate the jobs for data ingestion and data extraction.● Involved in working on performance tuning, query optimization and Implemented Dynamic partitions and Bucketing in hive for efficient data access.● Managed and reviewed Hadoop log files to identify issues when a job fails.● Involved in performing Unit testing and System Integration testing to test jobs and the complete flow.● Involved in working on different compression techniques such as snappy and Gzip.● Used Git for version control. Prepared Deployment and backout plan documents to implement in the Production.Skills and Tools used – Amazon S3, Hive, Beeline, Spark, Hadoop, Oozie, Impala, Hue, PostgreSQL, RedShift, Eclipse, Cloudera, Scala, Linux, HDFS commands, Jira

● Worked for a project called Hanover Insurance Group, duties include to manage data coming from different sources and loading of structured and unstructured data into target tables.● Worked in an AGILE environment with continuously changing requirements and deadlines.● Worked on performance tuning and optimizing the Hive queries for faster accessing the data.● Worked with various file formats such as Avro, Sequence, ORC, JSON, Text (CSV).● Used Spark SQL to get data from Hive and process using Spark API’s.● Involved in converting Hive/SQL queries into Spark transformations for efficient data access.● Imported required tables from MySQL database to Hive tables using Sqoop.● Used Partitioning and Bucketing techniques, for controlling the data distribution in Hadoop hive tables.● Worked with Git for version control, Jenkins for Continuous integration and deployment.● Experienced in Monitoring the Hadoop Cluster.Skills and tools Used​ - Hadoop, Yarn, Hive, Sqoop, MySQL, Python, Cloudera