Data Engineer Jan 2024 till date Cathmich Consulting Ltd (Remote)• Design, build, and maintain efficient and scalable data pipelines.• Automate data ingestion processes from multiple sources and ensured timely delivery.• Work with data scientists and analysts to ensure data pipelines are robust and support their needs for modeling and analysis.• Optimize existing data pipelines to improve performance, scalability, and reliability.• Ensured smooth operation of ETL processes and resolved any data-related issues promptly.• Design and implement low-cost and efficient data storage solutions (data lakes, marts and warehouses) for clients to support transactional databases and increase availability, reliability and performance.• Design and manage both SQL and NoSQL databases for different business applications.• Maintain and evolve data architecture to support business expansion needs.• Ensure data integrity, accuracy and security through thorough data management practices.• Implement data governance frameworks and drive compliance with internal policies and external regulations.• Set up data monitoring and auditing procedures to track data usage and identify possible risks and deviations.• Ensure data privacy and security protocols are maintained, and adhere to local and other relevant data protection regulations.• Collaborate with cloud solution architects to optimize cloud infrastructure.

Junior Data Engineer June - Dec 2023 Cathmich Consulting Ltd (Remote)• Designed data ingestion pipelines that ingested data from different sources into company’s data lake storage. • Implemented triggers on data pipelines based on frequency and/ or requirement for incoming data to automate pipelines. • Implemented workflows in data pipelines to transform incoming data based on data warehousing requirements and needs of data analysts and other data consumers. • Implemented dataflows that moved transformed data into data warehouse based on triggers. • Delivered data in the form of external tables and views adequately staged for the consumption of other members of the data team. • Created delta lake tables in Azure that served as sinks for streaming data originating from clients’ internet sales outlets. • Set up Azure Stream Analytics to process streaming data in the delta lake tables. • Used Spark Streaming Structure with various windowing functions to analyze (streaming) data in delta lake tables. • Delivered output of streaming analytics through Microsoft PowerBI to produce real-time visualization of customer data from application usage activities, in order to provide business intelligence and drive decisions in real-time.

• Developed CLI programming skills including Bash, PowerShell, pgcli, which serve as a faster alternative to GUI for working with software, and containerization with Docker. • Applied advanced Python programming skills to develop ETL pipelines for extracting data into local RDBM systems like PostgreSQL using pgAdmin, Docker and Docker-compose. • Applied advanced Python programming skills to develop ETL pipelines for extracting, transforming and loading data into GCS (Bucket). • Applied SQL skills to transform data in data lakes and migrate them to data warehouses like BigQuery where they are deployed for consumption. • Orchestrated and implemented workflows to manage and monitor data pipelines setup as DAGs in Apache Airflow to load data into local and cloud-based data warehouses.• Used PySpark to write complex SQL queries to run data transformation jobs on Apache Spark.• Optimized number of Spark clusters used for running Spark jobs.• Used Data Build Tool (dbt) to perform data transformation and delivered output data for visualization in Google Studio.• Set up Apache Spark Clusters in Azure Databricks to implement data engineering and data science workloads.

• Developed intermediate to advanced Programming skills in Python necessary for big data processing and analytics. • Developed complete and deployable Python scripts for Grocery Store Sales and Inventory Management Application. • Used versatile python libraries such as pandas, matplotlib, seaborn to analyze and visualize complex, public datasets and discover crucial relationships that exist between target variables and other features in diverse datasets. • Used powerful python sk-learn library to train Regression and Classification models using publicly available datasets including diabetes dataset, heart disease dataset, house price dataset.• Carried out hyper-parameter tuning to improve performance of standalone models and built pipeline objects that preprocessed and trained several models together and selected one with best performance.

• Joined a team developing a mobile application that will serve as a digital payment solution unique for providing credit services to users booking rides on ride-hailing platforms. • Designed and executed prospective customers survey to gather relevant consumer data required to understand product relevance, level of acceptance, probability of usage and other market indices vital to the success of the product at incubation. • Collected and analyzed data (qualitative and quantitative) gathered from surveys in Excel. • Designed interactive dashboards from survey analysis in Excel and PowerBI, to guide product designs, marketing campaigns and other consumer enlightenment campaigns that will give successful product acceptance in the market. • Pitched insights from analysis of market surveys in easy-to-understand terms to non-technical management audience. • Interfaced with other teams within the organization to provide detailed and accurate knowledge in the form of trainings and orientations about product. • Interfaced with vendors and third-party firms on all communications related to the product with respect to data usage policies including data privacy and data protection.

• Developed intermediate to advanced Excel skills as introductory data analysis tools.• Worked with complex datasets to perform exploratory data analysis, and deliver actionable insights via interactive dashboards built in Ms-Excel and PowerBI. • Developed basic SQL skills and worked on publicly available databases such as AdventureWorks to implement SQL queries in Microsoft SQL Server and PostgreSQL. • Evaluated Sales trends in order to understand competitive environment, product performance and effectiveness of strategies; • Synthesized current business intelligence data to produce reports and polished presentations, highlighting findings and recommending actions; • Created databases and developed database objects, including tables, views and materialized views using SQL.