Infracommerce is a Brazilian retail company that delivers value by facilitating logistics, payment methods, and integrations. In this challenge was developed a data processing pipeline to monitor mainly the Sistem Service agreement (SLA) of integrations between her partners. For collecting the data between different systems was used Debezium, a Change Data Capture plugin of Kafka-connect. after imputed in Kafka topics, this data was processed with KsqlDB.In KsqlDB all the flows, business rules, and SLA were calculated and sent to another topic. After finishing KSQL processing, this data was sent to Elasticsearch and showed as a metric on Kibana dashboards. The infrastructure of Kafka was used in an on-premise environment where was implemented dashboards to monitor all machines and applications' health.A big challenge in this experience was discovering and documenting how integrations work between partners' systems, for this was necessary to know other teams and understand how we can integrate theirs systems. Managing the infrastructure of all applications. Furthermore was needed to maintain and manage all infrastructure of these tools, besides that, each partner system had its own data schema, which needed to be dealt with before applying the calculation rules.On the Elasticsearch database side, some performance improvements were applied like caching dashboards data, purge policies, and custom fields to show updates in real-time.

S3 Bank: Among the developments carried out, I highlight here the data enrichment service that asynchronously consulted data from a bureau, which caused some inconvenience because this data took time to be loaded, RabbitMQ was used as a message broker to process and separate the responsibilities and even organization of the enriched data and after carrying out the due validations.