Project- Data Door Dash• Developed and implemented data quality checks (null, count, and freshness) across multiple stages, ensuring data integrity from Amazon S3 to staging and target systems.• Utilized Scala to automate the data quality validation process, integrating results into JSON strings for efficient reporting.• Configured data pipelines using YAML files for file locations and formats, enhancing the robustness of data handling and processing workflows.• Collaborated with cross-functional teams to optimize data ingestion processes, leveraging Amazon SQS for streamlined communication of data quality metrics.• Spearheaded the implementation of multi-level data quality assurance processes within Amazon S3 environments, utilizing Scala for automated checks on null values, record counts, and data freshness, enhancing overall data integrity.

Project- RRP Treasury CE• Proficient in Hadoop Ecosystem including HDFS, MapReduce, Sqoop, Hive, Spark, Scala.• We have all our data on Oracle DB we use Sqoop to import data from oracle DB to HDFS environment this usually happen in 2 steps 1st from oracle to persistent and then from persistent to into HDFS.• I have created data pipeline using scala which takes data from HDFS environment applied different-different transformation like aggregation, joins and filtering based upon the business requirements and store into HDFS. And read data/tables using HIVE.• I have used many optimization methods like broadcast variable, accumulator, broadcast join, cache and persist which leads to increase 40% of processing speed.• Create partitions and buckets in Hive to handle structured and semi-structured data.• Process files of different format like CSV, JSON, PARQUET into HDFS using SparkSQL and Hive.

Trained on Bigdata and completed one project where, I have sqooped hundreds of tables from oracle to HDFS simultaneously creating the table definition in MetastoreDB of Hive using hive-import clause. Also analyzed and visualized data by using Hive and Hue.