• Involved in various phases of the software development life cycle (SDLC) of the application, like requirement gathering, design, development, and documentation.• Developed ETL processes in AWS Glue to migrate data from external sources like S3, ORC, parquet, and text files.• Created a data pipeline for different events of ingestion, aggregation, and loading of consumer response data in the AWS S3 bucket into Hive external tables in the HDFS location to serve as feeds for Tableau dashboards.• Implemented Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark, SQL, Data Frame, Pair RDD's, and YARN.• Developed and implemented SSIS, Azure Cloud Services created complex ETL packages using SSIS, which upset data from staging tables to database tables.• Upgraded existing pipelines to become more real-time for faster reporting.• Collaborated with stakeholders and data scientists, including data, product, and executive teams, and assisted with data-related technical issues.• Conducted performance tuning exercises and capacity planning to ensure the scalability and efficiency of the data infrastructure.• Experienced in Agile methodologies, participating in daily stand-ups and internationally coordinated PI Planning.• Utilized JIRA for project reporting, creating sub-tasks for development, QA, and partner validation.

• Developed and maintained robust data pipelines, ETL processes, data integrations, data structures, and data models.• Designed efficient ETL processes to move and integrate data from different sources into the enterprise warehouse.• Performed ETL/ELT and SQL coding using AWS cloud-based solutions and downstream enablement in analytics.• Implemented strategies for partitioning, indexing, and caching in Snowflake and AWS services to improve query performance and reduce processing time.• Created data flows by sourcing data from multiple disparate sources utilizing SSIS packages, SQL, and REST API’s.• Converted Hive/SQL queries into Spark transformations using Spark SQL and Scala.• Developed new and improved existing data pipelines, extracting from external API sources or internal events.• Created (CI/CD) continuous integration and deployment pipelines.