· Develop, test and maintain the data infrastructure, the data ingestion pipeline, data store and data processing on premise.· Working with business analysts in gathering business requirements, obtaining clarifications and approvals during functional analysis, data validations and integration testing.· Implement and execute data security and data quality measurements to ensure compliance to data security and data governance policies· Data modeling is done using Data Vault technique.· Worked on Scheduling the Jobs through NIFI scheduler· Design, development, testing, troubleshooting and debugging of the application.· Involved in creating Hive tables, data loading and writing queries.· Was responsible for Optimizing Hive queries that helped in saving Cost to the project.· The processed data is loaded into hive table using spark SQL

· Worked on Importing the data from the cloud environment like AWS S3 as well as from URL efficiently.· Processed the millions of complex data in Spark environment and exporting to the destination.· Create the Business Logic and frameworks in collaboration with the Clients.· Loaded and Processed the Serialized File like Parquet.· Loaded and transformed large sets of semi structured data.· Involved in working on the Data Analysis, Data Quality and data profiling for handling the business that helped the Business team.· Performed Import and Export of data into HDFS and Hive using Sqoop and managed data within the environment.

· Worked on Importing the data from the cloud environment like AWS S3 as well as from URL efficiently.· Processed the millions of complex data in Spark environment and exporting to the destination.· Create the Business Logic and frameworks in collaboration with the Clients.· Loaded and Processed the Serialized File like Parquet.· Loaded and transformed large sets of semi structured data.· Involved in working on the Data Analysis, Data Quality and data profiling for handling the business that helped the Business team.· Performed Import and Export of data into HDFS and Hive using Sqoop and managed data within the environment.