Senior Java Developer | Scala Developer
Bidmind by Fiksu is a Demand Side Platform with Real Time Bidding to show advertisement in Web, Mobile, Desktop, and CTV applications. Company uses accurate real-time machine learning models to win in real-time price auction in successful and profitable way. Model is based on user’s device type, platform, browser, location, time, interests, previous activities and etc. Response time restriction is 100 ms. Distribution over the world. Hundreds of thousands of auctions per second. Hundreds of terabytes of data for machine learning and aggregated real-time reports. There are also various user audiences build on sophisticated algorithms. System incrementally actualises billions of mobile profiles. My responsibilities: Developed, optimised Scala ETL Apache Spark flows to extract, transform, validate, enrich, join, aggregate data for reporting, analysing, investigation, uploading to warehouse and machine learning model usage. Implemented Apache Zeppelin notebooks for reporting and analysing. Supported Apache Spark meta catalog to query parquet Amazon S3 data with SQL. Supported Clickhouse warehouse: optimise queries, configure, pre -aggregated cube building with dimensions and measures. Developed different Apache Kafka data collectors, Kafka Streams applications for audience/mobile profiles. Designed and developed Apache Kafka Streams scraper to collect application data from different app stores (Google Play, Apple and etc) and to gather authorised advertisers data located on developer domains. Developed Federated Warehouse based on Apache Calcite to have single SQL entry point with querying data from different DB, S3 and other sources. Added optimisation rules. Design: different technologies and data investigation to choose right architecture, integration. Support: optimisation tasks, bug fix, monitoring Grafana charts, feature production deployment, cluster scaling, alerts monitoring, Kibana logs analysing, Kubernetes yaml Docker files set-up, code reviews.
