Senior Data Engineer
CurrentWorking in tech, I've had my fair share of challenging and rewarding projects, but nothing quite like the data ingestion system I got to build recently. It was a complex setup using Apache NiFi and Python to handle over 10TB of data each month. This included everything from transaction data from MySQL to user logs from MongoDB, and even raw files from AWS S3. To keep up with the real-time data we needed for analytics and fraud detection, I set up Apache Kafka, which now flawlessly processes more than 2 million data points a day.When it came to improving our data systems, I led a major shift to Databricks and Snowflake, which cut our data processing time and made our data much easier to access. I spent quite a bit of time fine-tuning our databases with ER/Studio and SQL Developer too, getting our query response times down significantly.I've also been diving into data visualization. Using Tableau and Power BI, I put together some interactive financial dashboards that not only look great but are also really insightful.I managed a team that worked on integrating and ensuring the quality of our data with tools like Informatica and Talend. We refined our ETL processes so well that data errors dropped from 1,000 incidents a year to just 10.Machine learning is another area I’ve explored a lot. I've been working with everything from segmentation algorithms to object detection and have even set up a microservices architecture with AWS Lambda to keep things running smoothly and quickly.One project I really enjoyed was using Apache Airflow to manage data workflows for a global e-commerce platform. Setting up a custom monitoring system there really helped us cut down on the time it takes to handle critical failures.And finally, transforming our AWS data lake and building a unified analytics platform with Databricks was a major milestone. This hasn't just made our day-to-day operations smoother but also helped everyone make quicker, better-informed decisions.