Data engineer/cloud engineer consultant:- Data ingestion pipeline design and implementation (AWS) - Batch (event-driven: s3, eventBridge, lambda, glue, rds) - Streaming (event-driven: event mapping, eventBridge, kafka, lambda, kinesis data streams)- Docker image creation for glue and lambda unit tests- CICD integration (Docker image and SonarQube as part of the CI section)- Operational monitoring via Opensearch (event-driven, eventBridge, Firehose, lambda, Opensearch Domain)- Orchestration (airflow)- Data analysis (Snowflake, Microsoft server)- ETL via AB Initio on EC2 instances- Data pipeline generation- Data processing via databricks (SparkSQL)

Working as part of the data migration team. My mission is:- to migrate data from Hadoop 2.6 platform to Hadoop 3.1 platform- to use Sqoop to transfer data (with shell script and json data)- to use a PUT framwork composed of: * copy CLI from hadoop fs (hdfs dfs) * small data processing with spark- to use Hive to check on the data transferred- to use Tidal to schedule the job and monitor it in Hadoop 3.1- to use JIRA, confluence, sharepoint, excel for documentation and agile

Conducted application assessment as a consultant for airbus. Shared Data Management/engineering advicesTools used:PalantirPythonActiveMQJiraExcel

Performed data engineering and management missionsConducted lectures at engineering schools' conferencesParticipated in pre-sale mission by defining KPI and compared cloud providers' solutionsSupervised junior consultants, assisted in their career aspirations, and reported to the career committee on their performanceTools used:Spark and scalaSpark StreamingDockerPySparkElasticsearchKafkaAnsibleAWS S3AWS lambdasAWS EC2AWS EMRAWS Glue jobsGitConfluenceJiraGCP cloud functionsGCP cloud storage

Performed Data cleaning and data preparation on AWSCreated a Data dictionary on AWSParticipated in data architecture solutionPerforming tests on mongoDBTools used:AWS S3AWS EC2Jupyter on AWS EC2AWS Glue JobsOpen DataSoftAWS EMRAWS lambdaMongoDB Compass

Performed comparative studies on cloud providers solutions (AWS, Azure, GCP, ...)

Created Ansible playbook to create Kafka topics to consume Invoices data.Worked with SparkStreaming to create Invoices referential data sets and store those data sets in Cassandra.Worked with SparkStreaming to index the needed sample of elements in Invoices data to work with Elasticsearch.Created a web service to expose the data indexed in Elasticsearch with a REST API.Created Ansible playbook to start the webservice. Assisted Carrefour to improve the invoice scanning policyReported every step done on gitHub, Confluence and JIRA

Participated in the data ecosystem creation via Hadoop ecosystemManaged data with Apache Pig and Java in order to create referential data sets for the marketing department and automated the process via shell scripts, allowing Credit Mutuel to launch a customer loyalty campaign and anticipate customer behavior based on the referential dataStored data sets in HDFS, created DDL with schema description to use Hive on those data sets as external tablesCreated a verification process that raises an error if the data is not within expectationsSolved incidents that we created on Service Now such as workflow breakdownTested other data processing software such as Apache SparkTested NoSQL Databases like MongoDBTested data transfer through Sqoop

Recherche sur les bases NoSQL, le système Hadoop.Exposé sur ces technosAxe de travail sur une base orientée graphe (Neo4J) afin de vérifier l'utilité et les performances d'une telle base de données afin d'indexer les données géospatiales de l'outil développé, pour la sécurité des événements supervisés informatiquement par Egidium.

Evolution d'interface graphique sur un projet de déploiement de réseau