Senior Software Engineer, Big Data
Current• Designed and maintained fast and cost-effective ETL pipelines for processing terabytes of data using Apache Spark, Apache Airflow and AWS EMR• Pioneered and maintained real-time streaming data systems using Apache Flink.• Propose and initiate projects on big data systems, data lakes; and fault-tolerant and scalable platforms to the Head of Engineering and CTO.• Performed enhanced monitoring, load testing and capacity planning for new and existing data systems.• Researched and evaluated big data solutions such as real-time analytics databases, data storage systems, NoSQL databases, and search engines.• Performs various data analysis tasks using Pandas and Plotly in Jupyter.