Senior Technical Lead
Current• Promoted to Technical Lead, overseeing a team of big data engineers while continuing to contribute as a hands-on developer, ensuring project success through technical guidance, code reviews, and architectural decisions.• Developed and implemented automation tools using Scala, Presto JDBC, Hive, and Spark to enhance Flare pipeline performance• Developed comprehensive ETL Automation Application for scheduled daily, weekly, and monthly ETLs, incorporating multi-query summaries, data trend checks, status notifications, and automated file transfer to edge nodes for consumer access.• Led migration of CVM campaign generation process from Oracle to HDFS, reducing processing time by 80%. Converted key summaries from Oracle to Presto, managed the migration team, and integrated the process with the ETL Automation Application for seamless execution.• Designed and built API Data Extractor to process JSON data from various endpoints.• Engineered Partition Rewrite Application to optimize queries performances and resolve Hadoop small file issues• Developed Hive tables duplicated records removal tool to improve data integrity and system efficiency.• Developed a distributed application to automate the hive repair tables process.• Led POC projects for ELK Stack, Apache Presto, Impala, and Spark SQL technologies.• Authored custom database extractors using Scala JDBC and Apache Sqoop for diverse data sources including Oracle, Informix, SQL Server.• Conducted root cause analysis on data quality issues and developed permanent solutions• Led testing efforts for Ligadata's near real-time Decisioning engine, Flare.• Delivered comprehensive training sessions on pipeline components, including data extraction, ingestion modules, and in-house API.• Supported customer change requests using nPario platform for clients including Schibsted Sweden, MBC, and Aramex.
