Research Data Engineer
Current• Develop, maintain, and monitor Databricks job, automated ETL pipelines with a focus on reliability, scalability and data integrity using Pyspark• Designed and implemented scalable data pipelines in Snowflake, utilizing its multi-cluster shared architecture to enable parallel processing• Developed ETL pipelines leveraging Snowflake’s micro-partitioning to integrate data from various sources into AWS Redshift, with data accuracy and consistency• Proficient in setting up and managing data lakes using AWS Lake Formation• Ensured secure, scalable, and efficient data storage and access, leveraging Glue Catalog for metadatamanagement and seamless data integration across various AWS services• Developed and orchestrated robust ETL pipelines using AWS Glue and EMR• Automated data extraction, transformation, and loading processes, enabling efficient data processing and analysis. Leveraged Glue Catalog for maintaining an organized and accessible data repository• Developed Airflow orchestrated core python Databricks jobs in collaboration with Dynamo DB and terraform.• Collaborate with data visualization analysts and learning engineers to develop and maintain interactive dashboards for visualizing and interpreting trends in user data using Power BI• Collaborate with research scientists and learning engineers to develop and maintain data modelling tools for shared understanding of data definitions, lineage and access control.• Extract, parse, analyze, investigate, interpret and extract knowledge from high volume, velocity, and variety of data obtained from online and blended instructional applications and associated outcome measures• Performed advanced queries on relational and non-relational operational databases using sql and other query tools as needed• Part of 4-member team that worked on successful and accepted NCME (National Council on Measuring in Education) research paper on “Improving equity in AI-based learner modelling using aggregated demographic data”.