I am responsible to build and design a performant Data Warehouse. I spent most of my days creating new models, learning about current data models and debugging issues in the data to improve data quality across the board.

Cloud Data Engineer responsible to create data pipelines, manage workflows (through Apache Airflow/GCP Composer) and small real time data pipelines.Made an impact by developing a pipeline that improved a long running (over 24h runtime) pipeline to be reduced by 60% thus making our data analysts feedback loop a lot better. I have modified a CI pipeline to improve our data engineering experience using GCP composer by making sure our data engineers could task and modify our DAGs independently without interfering with one another.Tech Stack: GCP (Google Cloud Platform), Python, Apache Airflow, GCP Pubsub, GCP Storage, GKE, SQL Server, Docker.

Most of my activities are related to plan and implement cloud architectures that can handle small loads of data as well as provisioning GCP resources in the cloud, all of that while maintaining costs in check.Projects usually involve Google Cloud Platform as the Cloud Provider and the main services used are: BigQuery, Google Cloud Storage, GKE (Google Kubernetes Engine), IAM, Virtual Machines. I use Terraform to provision the cloud resources in a reproducible and traceable way.In order to write the data pipelines needed by my users I leverage Python, Apache Airflow and Docker.

Create and maintain data pipelines using a whole range of AWS services (Route 53, IAM, EC2, CloudWatch, S3, ECS, ECR, AWS Athena, AWS Glue).I also had experience interacting with React code, enough to be able to talk about it and discuss the best approach to communicate data between frontend and data.Another great part was that I used a lot of Terraform to provision AWS resources and separate DEV and PROD environments of a simple application.

1. Develop simpler ETL jobs using ECS tasks and docker using pandas, pyathena and Google Sheets + Klipfolio for visualization.2. Develop Data Lake Architecture containing Apache Airflow (for automation), ECS/ECR (Task launch), AWS EMR and Apache Spark (Python3/Scala). Also used boto3 a lot for comunnicating and integrating with AWS. Besides that, maintained AWS Glue and AWS Athena services for the data science team.3. Analyze data using pandas, matplotlib and Jupyter notebooks.4. Develop event source architecture using Spark Structured Streaming and AWS Kinesis using Parquet as well as ORC for the storage format.

Working at the data team providing1. API solutions, mostly in Java.2. Machine Learning Models using Scala, Spark (Big Data), Amazon EMR, Spark ML (Machine Learning), Pandas and Scikit-Learn mostly deployed by a combination of AWS Data Pipeline, AWS EMR and Spark.3. Analyzed datasets using Apache Zeppelin, Spark, Pyspark.4. Data Engineering and ETLs jobs using Spark, Kafka, Docker.

I worked using Django framework, Pandas and TDD to develop Open Source Solutions for the Public Educational System in Brazil. I also used Jquery, CSS3 and d3.js for charts.

Develop Neural Networks using Java; Pre-Processing Data using Python, Pandas, MatplotLib;

I worked that as a software developer, mostly on Angular.JS , SalesForce and Google AppEngine(Java)