Data Engineer Intern
👥 Team background:The Insurance Corporation of British Columbia (ICBC) is a provincial Crown Corporation responsible for insurance, driver licensing, and road safety for British Columbians. I was part of the Data Services team, which works on building and maintaining the data pipelines and deploying predictive analytics solutions to business problems.📝 Responsibilities:• Identified and investigated data quality issues using Spark and SQL to help the company’s machine learning models be trained on high-quality data and make more accurate predictions.• Implemented data ETL pipelines using Python and Spark to serve data for analysis and predictions that support stakeholders’ business decisions.• Performed data quality testing using PyDeequ to ensure that the required data characteristics are maintained after ETL pipeline transformations.• Wrote an article on how to use SVN, a version control tool, which guided the developers in the team to make their contribution to the correct versions of the codebase and collaborate on projects more effectively.• Documented for future maintainers the data conversion logic from raw to ready-to-use format using both word and visual explanation.👩💻 Tech stack: Python, SQL, Spark, Jenkins, Control-M, SVN, Jira