Implementation of a developer-first engine for collecting behavioral data platform (25 million events/day) and a platform to process the data having a pipeline that extracts raw data, applies ETL processes to prepare it for analysis, stores it in the Delta format on Amazon S3(IAM Roles), and utilizes PySpark within a Jupyter environment running in a Docker container managed by Jenkins

Main Duties:Data Strategy Consulting:Develop tailored data strategies aligning with business objectives, focusing on near real-time analytics, architecture optimization, and governance.Data Engineering & Architecture:Build scalable data pipelines using ClickHouse, Superset, and Kubernetes, ensuring data quality across batch and streaming environments.Cloud & Big Data Solutions:Architect cloud-native solutions on AWS, deploying scalable data workflows and managing Kubernetes clusters for seamless integration.Advanced Analytics & BI:Develop interactive dashboards using Superset, enabling clients to visualize trends and make informed decisions.Technical Leadership:Mentor teams in adopting cutting-edge technologies and data-driven decision-making, fostering continuous improvement.Major Successes & Projects:Near Real-Time BI Solution:Implemented a pipeline using Snowplow, PySpark, Jenkins, AWS S3, ClickHouse, and Superset, enabling quicker decision-making by analyzing user behavior data.Automated ETL Pipeline:Designed an ETL pipeline on Jenkins for data enrichment, saving results as JSON. Automated the workflow, improving data accuracy and reducing processing time.Kubernetes Deployment:Deployed Kubernetes clusters on AWS, enhancing data processing and scalability. Integrated data services efficiently, improving uptime and resource use.Superset Dashboard for KPIs:Developed a Superset dashboard monitoring critical metrics for a major client, enabling real-time analytics and faster operational decisions.Cloud-Native Data Infrastructure:Led the migration of data workflows to a cloud-native infrastructure on AWS, reducing costs and boosting processing speed and scalability.

Develop and improvement of the data architecture, install and evolve the data extraction and preparation infrastructure, and build views that show relevant information to the business;Plan and manage data pipelines, perform database maintenance, build APIs for data consumption, with constant monitoring and testing of systems.

Content writer for:Data Viz (Gestalt), Data Storytelling, Machine Learning, Power BITeaching the following modules and contents:Big Data Ecosystem:• Professions;• Data Engineer Roadmap;• Types of analysis;• Data Lake x Data Warehouse x Data Mesh;• Extraction/Transformation/Loading;• Storage and operational processing;• OLTP x OLAP;• Hadoop ecosystem (HDFS/YARN/MapReduce/Hive);• Apache Spark.data ingestion• Batch and Streaming• Lambda architecture;• Metadata management;• Automating data flows;• Workflows• Error handling;Cloud Computing• AWS S3;• Azure Blob Storage;• Google Cloud Storage;• Settings;• Managing storage;• Access management;• Cloud data processing;• Analysis and insights;Learn more about the courses at: https://escolakoru.com.br/lp-dados/

Development of a global platform to support teams of data scientists, focused on maximizing the impact, speed and quality of the analytics product implementation.The flow includes steps such as exploratory data analysis, implementation of the model as a class for consumption as a library, source code management, metrics and model evaluation, traceability of experiments, production of artifacts, registration of the model for consumption in production both in batch and via API.Used Technologies:* CI/CD: GitHub Action* Languages: Python, pySpark e Spark* Engineering Service: Azure ML & Databricks* ML lifecycle manager: MLFlowUnderstanding and application of best practices in machine learning, software engineering, and/or production deployment of ML servicesExperience in CI/CD automation and ML OpsExperience of putting data science models into productionExperience with Azure Databricks and MLFlowStrong experience with AI/ML frameworks (scikit-learn, Pytorch, Tensorflow, …) and MLOps frameworks (Docker, MLFlow, …).Experience with setting up & logging/monitoring of machine learning model metricsExperience with Azure data, Machine Learning and Artificial Intelligence solutionsExperience designing and implementing ML pipelines in Azure ML/DatabricksKnowledge of new feature store platforms (Databricks, Feathr)

Interaction:Generate business impact acting in a multidisciplinary squad and leading the data initiative for the marketing area of ​​a Global Beverage CPG Company;Develop the data architecture, install and evolve the data extraction and preparation infrastructure, and build views that show relevant information to the business;Ensure that data is received, verified, transformed, stored, and accessible to the team;Plan and manage data pipelines, perform database maintenance, build APIs for data consumption, with constant monitoring and testing of systems.Design and implement extraction, transformation and loading and visualization solutions;1) Extraction through services or jobs;2) Transformation: through ETLs, pipelines, streamings, realtime or batch.3) Load: in databases, warehouses, data lakes, and various repositories.4) Visualization: through services or query interfaces;Challenges:Connect to different databases, understand their structure and content;Handle large-scale data (Cloud and Big Data);Gather and prepare data to be analyzed and worked on, correcting errors;Add value by building, structuring, and optimizing data to be used by the Analytics team, Analysts/Scientists/Data Engineers, in addition to supporting business leaders;

Management of two value streams:1. Analytics, Engineering and Data Science team driven to product construction;2. Development of a platform for simulating an e-commerce application

Global Project of a Platform for Sales Management and Uplifting (Cross and Up-Sell) in Beverage Industry with a Data Science teamOutcomes: - Continuous Product Discovery; - KPI Design and Dashboard; - Machine Learning Models;

Feature Manager for Mobile REST API application using Java Spring Framework.

Global Data Unification Project (Data Lake and EDW) - Microsoft Azure

Global Data Unification Project (Data Lake and EDW) - Microsoft Azure

Data today has high added value for an organization. The path of data analysis is long and complex, ranging from the way data is collected to the insights that the teams managing this huge amount of data can draw from it. On the flow path to digital transformation, you will learn basic data concepts, processes and tools for analysis, current ethical aspects, how to look for relevant insights, and how to present results to interested peers to support actions and gains.1. Data analysis introductory aspects: the importance of statistical aspects for data analysis. Spreadsheet programs and data analysis. Spreadsheets and data analysis. Use of data in user centric solutions.2. Big data: architecture, data modeling and the science of data analysis. Skills for data analysis. Data management aligned with ethics and legislation. The impact of assumptions on data study. The impact of exponential technologies for data analysis: machine learning.3. Practical aspects of data analysis: Understanding the problem mapping the available data. Assumptions definition. Structuring the data for analysis (cleanliness and definition of what to use). ETL process application. Correlating data. Analysis, visualization and presentation of results.https://www.fnodt.com.br/transform-tracks%C3%A7%C3%A3o-para-a-transforma%C3%A7%C3%A3o-digital

Responsible for the Data Science Manager course (Based on DMBoK)

Creation of a Data Lake in AWS environment (Financial sector company)

• Architecting distributed systems;• Create reliable pipelines;• Combine data sources;• Create solution architecture;• Collaborate with the Data Science team and build the right solutions for these teams;• Collect data from various sources (crawling);• Store in an organized and scalable way on a data lake;• Creation of tools to access information;• Product Development Sprint assembling and managing the resources

• Project Management using CRISP-DM methodology;• Creation / maintenance of Data Lake (AWS);• Integration of different data sources / formats;• AWS Environment: S3, SageMaker, EMR e Redshift;• Data modeling;• Pipelines construction using distributed systems (Spark);• Development in Python;• Development ETL layer in Python;• Application Deployment for Data Science;• Creation and Deployment of Dashboard (Power BI);• Orchestrating data.

Main activities: - Motivation of agile principles in teams. - Ensure that deliveries are made within Sprint by removing any impediments.- Mapping of internal and external dependencies - Facilitation with agile metrics and Scrum rituals. - Facilitation in internal and external communication promoting transparency and sharing information. - Support and assist Product Owners with the maintenance and prioritization of the Backlog. - Refinement of user stories to add greater value to the business and risk mapping.

The area of Academic Intelligence aims to mine and transform the available data into knowledge that generates value, being an indispensable part to provide support to the direction of decision making in all academic scope for the company.* Study the process together with the business teams and architect the database of the projects;* Develop and ensure a flow of information for control and management of reliable data;* Automation and Optimization of processes for work optimization and reduction of operational cost;* Acting in customized systems so that it can manage improvements to guarantee the reliability of the information;* Development of KPI and critical analysis of data;* Create data intelligence aiming to generates knowledge of the information analyzed for decision making;* Develop smart data;* Data Structure;* Development of robots for process automation;* Architecture and Structuring of Servers for Data Processing;

Acting with business intelligence through the elaboration of reports and dashboard based on KPIs for strategic decision making. Responsible for creating and implementing the support process. Management and training of employees to provide support and after sales. Customer training.Also responsible for:- Securing satisfaction with current clients and growing relationship- Monitoring results and acting when those were below expected.- Internal product development participation- Coordinating operations with other departments- Software implementation and project management- Follow up of product implementation- Strategy creation and development- Creating and developing process standards- Achieving Goals- Presenting/analysing results based on data

Market analysis for Implementation of the Astronomus / Gaspra Software, based on the patent protected computer program by Unicamp.Training in business modeling methodologies (Business Model Canvas / Lean Canvas) and development of technological entrepreneurship.

Indexing and analysis of database.

Development of an ultrasonic system for measuring the propagation velocity of acoustic waves;

Assistance to students with difficulties in their respective disciplines.