João Victor Da Silva Mendes Email & Phone Number

Campinas

• Utilization of manual parallelism in Data Factory pipelines to optimize the ingestion of millions of records by distributing data across multiple threads in the Integration Runtime.• Data modeling according to business rules to ensure consistency and efficiency in data delivery to the Data Warehouse in analytical applications.• Development of custom operators in Apache Airflow using object-oriented programming to (1) interact with system APIs and generate files in Azure FileShare, and (2) integrate data from these files for Business Intelligence reporting across departments (Finance, Logistics, Operations, Sales).• Development of a DAG for data extraction from the Green Offer API to provide analytical dashboards on sustainability metrics related to Sonepar's green product offerings.• Analysis of Execution Plans of queries to optimize the data ingestion and transformation process in dimension and fact tables to maximize the daily update performance for the organization's Data Warehouse and Data Marts.• Implementation of Slowly Changing Dimension Type 2 (SCD-Type 2) to store historical item costs in analytical billing reports for the Controllership team.• Quality testing of Excel data sheets using Pydantic before loading them into the corporate Data Lake.• Creation and maintenance of a data catalog for the Control Tower data project to store technical and business metadata on data assets available for consumption by the software engineering team.

• Data Engineer in the Control Tower data project at Sonepar.• Development of pipelines using Data Factory for ingesting and transforming data from various sources, integrating it into a Data Warehouse to make it accessible for the Business Intelligence (BI) development team.• Use of Apache Airflow to orchestrate the entire data ingestion and pipeline execution process for the Data Warehouse.• Creation of dashboards in Grafana with KPIs, indicators, and charts that provide data-driven insights for strategic decision-making.• Use of SQL in relational databases (MySQL, SQL Server) to build advanced analytical queries that deliver business insights to support decision-making.• Creation of SQL procedures and CTEs in relational databases to automate data cleaning and processing tasks within the MySQL relational database.• Experience in optimizing the execution of SQL queries, views, and procedures to provide data to end users as quickly as possible.• Experience in object-oriented programming in Python to develop data validation, processing, and cleaning pipelines, ensuring data quality for storage in the Data Lake.