Created a fully cloud native spark enviroment for data pipelines using Dataproc, Cloud Build, gitLab and Cloud Composer Airflow and also created the master scripts for data transfer from landing to trusted zone using said enviroment.I was responsible for the maintenance, improvement and optimization of the entire Hering CRM, sales and inventory data pipelines according to business needs. These data pipelines consisted of 15 Airflow Cloud Composer DAGs and over 35 ETL scripts.

Used Python and PySpark framework to create 10+ ETL incremental analytical data pipelines using the tables available at “trusted zone” to create tables “sandbox zone” on Google Cloud Storage using both queries already created by data analysts or queries I have created myself based on business needsCreated Python ETL scripts using PySpark framework to create incremental data pipelines to be used with 20+ tables coming from “raw zone” to “trusted zone” on Google Cloud Storage.Executed maintenance and bug corrections of various databases

Development and optimization of scripts in VBA languageExecution of important everyday tasks such as Validation of the fund portfolio

Statistical treatment and data entry optimization using Excel and VBA code.Development of Excel VBA scripts and templates that will be used throughout the project for multiple statiscal analysis and for the creation of charts