My task was to increase the ranking of ministry of culture through digitization and createa pipeline for gathering data of various schemes of ministry at root the level andcentralize different data sources and applying EDA over that data sources for visualizing,analyzing the data and building dashboards for the government schemes and alsocreating SQL queries to fetch the data.

Worked on My SQL database to analyse, query, sort and manipulate data according to defined business rules and procedures.•Assisting clients with optimizing data access performance, such as reviewing SQL Queries and analysing data transfer and suggesting improvements Developed and modified SQL queries to meet business requirements.•Worked on Data Cleaning, Manipulation, visualizations and creating insights from the data using python (pandas, NumPy, mat plot lib, sea born libraries).•Importing data using Sqoop to load data from my SQL to HDFS on regular basis.•Worked on various Hadoop formats like parquet, orc and Avro file.•Written Hive queries for data analysis to meet the business requirements.•Creating and loading data into hive tables with appropriate static and dynamic partitions, intended for efficiency.•Transformation data in spark using RDDs as well as High level APIs where Spark sql used for querying the data.•Perform ETL operations and deployment in aws emr using step execution•Data transferring from local to Hadoop using Sqoop commands, creating staging, externaltables in Hive

•Developed and managed data and/or databases that support performance improvement activities used various statistical techniques to analyze and interpret data from customers.•Performed a mix of Qualitative and Quantitative for a Telco Company study to understand the customers well and identified areas of improvement.•Performed data cleaning and analysis for sales reports•Collaborated with colleagues and executed other duties as assigned by managers•There were Daily 4 to 8 GB data for processing.•High level Data structure API used for analysis and used transformation in Spark with Scala•Scala Coding was used for ETL•Created staging tables in Hive and HBase Tables where Partitioned and bucketing used for query optimizing.•Full and Incremental data loading using Sqoop.•Worked on Stored Procedures, Views, Query Tuning Skills•Worked on Temporary Tables and Queries for Report Generation•Created ETL pipeline using airflow.

Used quantitative data gathered to develop an understanding of customer behavior, demographics, lifecycle Learned how to use databases, manage large files and run reports to analyze business trends Presented data that helped guide decisions of the company Performed data cleaning and analysis for sales reports Collaborated with colleagues and executed other duties as assigned by managers•Developed and managed data and/or databases that support performance improvement activities used various statistical techniques to analyse and interpret data from customers.•Performed a mix of Qualitative and Quantitative for a Telco Company study to understand the customers well and identified areas of improvement.•Performed data cleaning and analysis for sales reports•Collaborated with colleagues and executed other duties as assigned by managers•There were Daily 4 to 8 GB data for processing.•High level Data structure API used for analysis and used transformation in Spark with Scala•Scala Coding was used for ETL•Created staging tables in Hive and HBase Tables where Partitioned and bucketing used for query optimizing.•Full and Incremental data loading using Sqoop.•Worked on Stored Procedures, Views, Query Tuning Skills•Worked on Temporary Tables and Queries for Report Generation