Junior Data Engineer
Current• Joined to focus on software testing, utilising ISTQB certification and testing methodologies.• Transitioned to a data engineering role, driven by a growing interest in the data field and the desire to align with the company's primary focus, further supported by completing a Data Engineering Bootcamp and obtaining a certification.• Collaborated with senior data engineers to develop and validate SQL queries, ensuring accurate user billing and data transfer in a migration project to Azure cloud services for a Water Supply ClientStatefulDB Operational Data Store as a Service:• Participated in API testing activities to verify data flow and integrity using knowledge of Postman and Python.• Conducted basic HTTP requests using Postman and began automating simple test cases using Pytest to support development efforts.BOOTCAMP PROJECTSPinterest Data Pipeline● Technologies used: Kafka, AWS MSK, MSK Connect, AWS API Gateway, AWS S3,Spark, Spark Structured Streaming, Databricks, Airflow, AWS MWAA, AWS Kinesis.● Developed an end-to-end data processing pipeline hosted on AWS based onPinterest’s experimental processing pipeline.● Developed using Lambda architecture taking advantage of both batch processingand stream-processing methodsMultinational Data Centralisation● Technologies used: Python, APIs, PostgreSQL, Pgadmin4, Pandas, AWS RDS● Developed a robust system capable of efficiently extracting and refining retaildata from diverse sources, including APIs, PDF documents, a Cloud database, andJSON files.● Thoroughly processed and cleansed a substantial volume of 100k+ records,preparing the data for modelling within a star-based database schema.● Conducted in-depth analysis of the processed data, unveiling valuable insightsrelevant to the retail industry. These insights hold significant potential forenhancing retail business operations and decision-making processes.