Near Real-Time Data Platform Project (Petrobras)- I'm currently working on a solution for log data ingestion, search, and organization using OpenSearch on AWS, ensuring data is processed and available in near real-time.- Developed efficient pipelines to process and store log text files, using Logstash for data transformation and loading into the system.- Automated various processes by creating APIs and scripts in Python, which helped streamline data flow and maintain data integrity.

End to end Data Mesh project (Itaú bank)- Led the project of Data Mesh architecture on aws.- Developed python and PySpark code for extractions using Databricks from Redshift, lambda scripts to send files cross account, Glue ETL jobs to feed the Data Mesh layers.- Developed ETL processes using Databricks to connect to an AWS account, extract data into Parquet files, and send them to another AWS account to feed the Data Mesh layers.- GitHub Actions was used as the CI/CD pipeline.Technology: Python, PySpark, Databricks, GitHub, Terraform, Some AWS stacks utilized: RDS, RedShift, Roles, Lambda, Event Bridge, S3, KMS, Glue Workflow, Trigger, Job, Crawler, Athena.Development of Oi's new marketplace (Oi Telecom)- Developed ETL process from the Mirakl's APIs to support the new Marketplace.- Developed ETL process to extract data from relational and no relational data bases.- Focused on raw zone data engineering. - Deployment support.Technology: Informatica Power Center, Cloudera (CDP), Spark, HDFS, Hive, Python, Shell

- Developed ETL process from Postgres to an on-premises database.- Used shell scripts to connect and extract data files according to the rules entered via SQL. From there, it was inserted into the internal database and made available in materialized views.- Data modeling of fact and dim tables for the new DW.- Deployment support.Technology: Git, Shell, Python, Airflow