Shreshth Shrivastava Email & Phone Number

> Experience in Designing, Developing and maintain scalable data pipelines to extract, transform, and load (ETL) data from various sources into storage systems.>Implementing data ingestion processes, data quality checks, and data validation mechanisms.Data Storage and Management:>Evaluate and select appropriate storage technologies and databases to efficiently store and manage large volumes of data.>Optimize data storage and retrieval for performance, reliability, and cost-effectiveness.Data Transformation and Integration:>Transforms and cleanse raw data to conform to defined data models and schema structures.Integrate data from multiple sources and ensure data consistency and integrity.Data Monitoring and Maintenance:>Monitor data pipelines, data flows, and data processing jobs to identify and resolve issues in a timely manner.>Perform routine data maintenance tasks such as data backup, data recovery, and system optimization.Collaboration and Documentation:>Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and provide appropriate solutions.>Document data engineering processes, workflows, and technical specifications.Continuous Learning and Research:>Stay updated with emerging trends and advancements in big data technologies, data engineering frameworks, and cloud platforms.>Explore new tools and techniques to enhance data engineering processes and optimize data workflows.> Collaborated with various team & management to understand the requirement & design the complete system> Implemented the complete Big data pipeline with real-time processing> Developed Spark code in Python/Scala using Spark SQL & Data Frames for aggregation> Worked with Sqoop to ingest & retrieve data from various RDBMS like Oracle DB & MySQL> Created schema in Hive with performance optimization using bucketing & partitioning> Written Hive queries to transform data for further downstream processing> Created Scheduler jobs for scheduling queries & actions> Involved in the end to end deployment process *Technical functionalities* >Knowledge of Hadoop ecosystem and different frameworks inside it – HDFS, YARN, MapReduce, Hive, Sqoop, Kafka>Real-time processing Framework (Apache Spark),Pyspark,Database architectures, ETLSQL-based technologies (e.g. MySQL/ Oracle DB)NoSQL technologies (e.g. Cassandra and MongoDB)Programming Languages: Scala /python Knowledge of different Operating System – UNIX/LINUXKnowledge of Git , CI/CD Pipeline.

• 

  Dr. A.P.J. Abdul Kalam Technical University

  71.56