• Modern Data Stack Development: Built and maintained a modern data stack using Fivetran, Airflow (MWAA), Snowflake, DBT, and PowerBI. Implemented configuration-driven data ingestion, data quality, error logging, and data orchestration frameworks leveraging AWS services.• Data Ingestion and Pipeline Management: Designed, deployed, and maintained data ingestion frameworks for both streaming and batch data using Fivetran, AWS Lambda, DynamoDB, S3, and AWS Glue. Orchestrated data pipelines with Airflow and deployed them using Terraform, including an error logging framework to capture and alert the team about issues.• Custom Data Processing: Developed custom Python scripts to parse CSV and Excel data before ingestion into Snowflake, ensuring data integrity and quality.• Snowflake Administration and Optimization: Managed Snowflake as an admin, creating and granting roles using RBAC. Optimized cost and performance by creating dashboards to monitor costly warehouses, long-running queries, and idle times, and implemented guard rails to limit query durations.• Hackathons and Innovation: Participated in hackathons involving Amazon Bedrock/Chat-GPT for image recognition and web scraping data analysis, demonstrating adaptability and innovative problem-solving skills.

• Consulted with Startups: Interacted with various startup teams to understand their challenges and provided consultation on building data pipelines, gaining insights into the startup ecosystem and culture.• Product Ideation: Brainstormed and evaluated potential product ideas, identifying opportunities to develop them into viable product companies.• Healthcare Solution Development: Collaborated with a hospital management team to address their data challenges by developing solutions using Python.

• ETL Management and Support: Handled, maintained, and supported ETL data processes for Redshift, transitioning to a Data Lake architecture and managing thousands of ETL jobs, ensuring data integrity and reliability.• Data Transformation and Migration: Offloaded and transformed data from Redshift to Parquet format using Apache Spark on EMR, loading it into Redshift Spectrum to implement a Data Lakehouse solution.• AWS Glue Catalog Infrastructure: Worked on an in-house built AWS Glue catalog sharing infrastructure to minimize data duplication and enhance data consistency, leveraging AWS Glue, Lake Formation, Resource Access Manager (RAM), and Redshift Spectrum.• Data Handling at Scale: Managed and supported a centralized data warehouse handling 45-50 petabytes of data, providing on-call support to resolve issues and advising teams on data integration to facilitate stakeholder data sharing.

Data Engineer:• Data Lake Development: Designed and developed a comprehensive Data Lake on AWS using AWS Glue Jobs, Glue Crawler, Data Catalog, Athena, and EMR. Created PySpark code with RDDs and UDFs for complex data transformations.• Workflow Orchestration: Implemented and managed computational workflows and data processing pipelines with Apache Airflow, ensuring efficient data operations and timely data availability.• Data Deletion Framework: Built an in-house data deletion framework for CCPA/GDPR compliance within the Data Lake environment using Athena’s CTAS feature after doing a POC on Apache Hudi, Delta Lake and SnowflakeSenior Software Development Engineer in Test• Big Data and Automation Testing: Designed and developed testing frameworks using AWS Glue, EMR, Spark, and Python. Conducted thorough verification and validation of ETL processes in databases, data warehouses, and AWS S3.

• Data Pipeline Development: Designed and deployed data pipelines using Python, PySpark, MySQL, AWS platforms, Ganglia, Ansible, NiFi, Docker, and GitHub.• Big Data/Automation Testing: Conducted comprehensive testing of Spark processing using PySpark, Zeppelin, PyCharm, Jenkins, Perforce, and AWS platform. Utilized Python in Jupyter Notebook, Splunk, and AWS CLI for end-to-end validation of big data pipelines.• Hackathon Innovations: Pitched, implemented, and presented five new ideas involving testing tools, Android integration, voice integration using Android and Python, and PySpark at internal hackathons.

Research Assistant (Data Mining and Software Development) - co-advisor was the President of the International Educational Data Mining Society.• Automated Detector and Feature Engineering: Developed structured Java code using RapidMiner and Weka APIs to automate machine learning detector building and engineered complex features from foundational data sets.• Data Extraction and Behavior Detection: Created Java code to extract features from log files and relational databases for intelligent tutoring systems (e.g., ASSISTments ITS) and configured automated detectors to identify student behaviors such as boredom, frustration, confusion, and concentration.

Performance Testing Engineer - Bell Client at CGI Group Inc (bangalore, India)• Test Plan Development and Script Creation: Involved in requirements gathering and test plan preparation. Developed scripts using Web (HTTP/HTML), Web (Click and Script), and Ajax (Click and Script) protocols in HP LoadRunner, embedding C and Java code for error handling.• Performance Metrics and Testing: Set up monitors using Sitescope to collect performance metrics on application and web servers. Conducted various performance tests (load, stress, spike, and endurance), analyzed metrics and test results, and prepared reports. Performed tuning activities to identify and eliminate bottlenecks, enhancing overall performance.