Kapil Malviya Email and Phone Number

As a Data Engineer, I played a pivotal role in driving a successful data migration initiative for a prominent bank, facilitating the seamless transition of their data pipelines to Google Cloud Platform (GCP). Leveraging my expertise in Pyspark, I designed and implemented efficient data migration jobs, ensuring the secure and accurate transfer of critical data to GCP.Additionally, I led the development of a sophisticated transaction categorization module using DBT and BigQuery, tailored specifically for the bank's commercial BB and SME clients. This module significantly enhanced data processing capabilities, enabling precise categorization and analysis of transactional data, ultimately leading to more informed business decisions and enhanced client experiences.Furthermore, I spearheaded feature engineering efforts for machine learning models used in credit rating and affordability assessment. By leveraging cutting-edge techniques, I empowered the bank to build accurate and reliable ML models, which played a crucial role in optimizing lending processes and risk assessment for the bank's clients.Throughout the project, I collaborated closely with cross-functional teams, aligning business requirements with technical solutions and ensuring seamless integration with existing systems. My focus on data governance and data quality assurance throughout the migration process enabled the bank to maintain the highest standards of data integrity and security.As a result of these efforts, the bank successfully migrated their data infrastructure to GCP, unlocking new possibilities for data-driven decision-making and advanced analytics. The introduction of sophisticated categorization and ML capabilities positively impacted the bank's operational efficiency, client satisfaction, and risk management strategies.#DataEngineer #DataMigration #GCP #Pyspark #DBT #BigQuery #MachineLearning #DataAnalytics #DataQuality #DataGovernance

As a Big Data Engineer for client Experian, I have been entrusted with transforming daily data for various downstream systems, leveraging my expertise in Spark and Scala. My role involves automating big data jobs using AWS services, optimizing Spark jobs in EMR through Managed Resizing, and implementing other EMR techniques for enhanced efficiency.Working extensively with diverse data formats, including Mainframe EBCDIC encoded files, Parquet, Text, and CSV, I have demonstrated a strong grasp of Spark concepts such as RDDs, Dataframes, accumulators, and broadcast variables. This proficiency has enabled me to handle complex data processing tasks with precision.In my pursuit of process automation, I have leveraged Amazon S3 events to trigger Step functions using prefix and suffix patterns, efficiently orchestrating data workflows. I designed and deployed automated workflows using Step Functions, and proficiently wrote CloudFormation templates to create essential resources for workflow execution using Amazon CloudFormation service.Furthermore, I contributed to the automation of EMR cluster creation by writing lambda functions in Java, streamlining the provisioning of required resources.To ensure smooth operations, I utilized Amazon Simple Notification Service to promptly send mail/SMS notifications in case of any workflow failures, enabling quick resolution of potential issues. Additionally, I implemented Amazon Simple Mail Service to deliver Daily Validation mails with attached XLS sheets.Taking pride in my commitment to quality, I employed ScalaTest and scoverage tools for unit test automation and test coverage statistics, validating the robustness of my solutions.

Worked in various Bigdata application phases like data cleaning,data ingestion,data analytics and performance tuning using various BigData technologies like Spark,Hadoop,YARN,HDFS.