. Instrumental in building data platform and improved developer productivity by 5X.. Introduced and led various data products like Fivetran for data ingestion, HVR for replication to cloud, DBT for data transformations and Atlan for Data catalog. This helped build seamless experience for internal customers.. Trained and enabled 40 Citizen Data Engineers to use data platform products to solve business use cases across 12 domains.. Build custom materializations and macros in DBT that are used across the firm.

• Piloted and implemented E2E Data pipeline using pyspark, Amazon EMR, Python and Amazon RDS.• Developed a brand-new reporting suite at a rapid pace and helped customers adapt the new reporting stack. • Reduced the overall time to provision the cloud resources by deploying the artifacts using Terraform and Jenkins. • POC of Snowflake DB set up and evaluated its feasibility for all the reporting use cases.• Reduced cloud costs by 30% by automating the turn off/on of the EC2 machines and auto scaling the database cluster.

• Developed a generic framework using Spark/ Scala to process Dynamic Json Data.• Source streaming and batch data from disparate data sources like S3, Rest API, Kafka Queue and ingest into Hadoop/Hbase Tables using Apache NIFI.• Worked on migrating the code from on premise Hadoop cluster to Cloud(AWS).

• Involved in History Migration Project capacity planning, development, deployment, tuning and benchmarking the workflow pipeline with Operations team.• Worked with Cloudera team in designing the Unified data repository for end users to access 21yrs of hundreds of Terabytes of history data.• Experience in writing Oozie workflows and writing Autosys JIL scripts for job automation.• Hands on experiencing in developing spark application using Scala functional features.• Created Hive tables to store the processed results in a tabular format and written Hive scripts to transform and aggregate the disparate data.• Worked with modeling team to migrate SAS Macro/Programs from Teradata to Cloudera Impala.• Automated the process for extraction of data from warehouses and weblogs into HIVE tables by developing workflows jobs in Oozie.• E2E design in integrating Cloudera Impala with various BI tools like SAS, R, Matlab and Toad for Model Development and ad-hoc querying.• Worked on migrating the business users from Teradata to Hadoop and thus involved in Teradata decommission that in turn helped the organization by cost cutting.• Developed and productionalized the SaS Macro to write data to impala tables.• Developed a shell utility to move the data from production to lower lanes using Distcp.• Experience in using Avro, Parquet, RCFile and JSON file format and developed UDFs using Hive and Pig.• E2E development of the ETL process by sourcing the data from upstream, perform complex transformations and export the data to Teradata.

Designed and developed a framework to source the data from Kafka(exposed through APIGEE) using a informatics ETL tool. This approach enabled the client to perform the date ingestion without having to buy a license for Power exchange which infact saved the organization and team 330K.Have worked on multiple POC for data pipelines and evaluating the performance and usage of Big data tools like Kafka, storm, Hive and impala.Have worked on E2E ETL application development and productionalized them. Extract valuable information from the Raw data and present them to the business users using Tableau. The insights where very help for the users to take important cost saving descisions

Develop Pentaho Kettle Graphs to cleanse and transform the raw data into useful information and load it to a Kafka Queue and Neo4J database for further processing.Continous Integration and Continous Deployment(CICD) of the applications using Chef, Maven and Jenkins.

Interacted with design and process teams which are spread across various companies to explore solutions for business Well acquainted with agile methodology, sprint wise development and release wise delivery.Collaborated extensively with test teams spread across UK and India to provide rapid turnarounds in analyzing and resolving potential showstoppers during test phases.Exhibited sound object oriented programming skills in completing the assigned development tasks well within the deadlines.