- Utilization of Spark to process and analyze large volumes of data, implementing complex transformations with high performance- Configuration and maintenance of clusters on Dataproc for running distributed jobs, adjusting memory and CPU resources to reduce execution times- Creation of pipelines using Spark SQL, DataFrames, and RDDs to enable flexible processing and seamless integration with other Big Data tools- Optimization of Spark tasks, including data partitioning, execution parameter tuning, and caching to maximize efficiency- Implementation of data processing routines with Java to execute jobs on Dataproc, ensuring compatibility with Spark and scalability in cluster environments- Integration of Java pipelines with other Google Cloud services, such as Cloud Storage and Bigtable, for data ingestion and storage- Development of Airflow DAGs (Directed Acyclic Graphs) in Python to orchestrate complex data workflows and ensure execution in logical, dependent steps- Implementation of version control to ensure traceability and rollback of pipelines and data scripts using GitLab- Use of Google Cloud Platform (GCP): BigQuery, Cloud Storage, Dataflow, Dataproc, Bigtable.

- Creation of DAGs in Apache Airflow using Python to handle the data extraction process, as well as its transformation, loading, and distribution via the MQTT protocol.- Development of a Machine Learning model for semantic search using cosine similarity with BERT transformers, PyTorch, NumPy, and Pandas. Optimized spreadsheet completion and significantly reduced manual work, delivering satisfactory results.- Experience with relational databases: SQL Server and PostgreSQL. Non-relational databases: Redis.- ETL execution using Power Query, as well as the creation and maintenance of dashboards in Power BI.- API maintenance and development in Python using the FastAPI framework, along with Pandas, NumPy, and SQLAlchemy. Implemented classes and methods to promote more concise and efficient code.- Experience with AWS (Lambda, S3, InfluxDB, ECS, RDS, Kinesis, IoT).- Application deployment with containers orchestrated by Docker in both Linux and Windows environments.- Utilization of the MQTT protocol for real-time data communication with the AWS environment, processed through Amazon Kinesis Data Streams.- Participation in international projects for clients in the USA and France.