Senior Software Engineer
CurrentDevelopment for the AMRA music collection society. Processing music streaming usage reports from providers such as Spotify and Apple Music, to collect streaming royalties for music publishers. Files varied in length from a few MB up to several GB, receiving batches amounting to hundred of GB each month.Ingestion projectSpark, Scala, Python, Parquet, Terraform, Sqoop, AWS (EMR, S3, SSM, CloudWatch, SQS, SMS, Batch, ECR) • Design and development of an application to process large quantities of files, from a range of formats and providers, such that royalties can be claimed. • Writing Spark transformers in Scala to convert the files into a common format, running the jobs on EMR clusters, creating a datalake on S3, and syncing the data to Oracle using the existing data model. Using a mix of BDD and TDD to implement features. • This removed a large amount of processing from the database and enabled us to several billion rows of data and move it to cheaper storage. • Implementation of CI pipeline using Bamboo and GitHub. Writing e2e tests • Designing and implementing a system to process files automatically as they arrived, using CloudWatch Event rules to detect new objects on S3, SQS to store the events, CloudWatch to trigger the Spark jobs on EMR, through running a custom Python utility in Docker on AWS BatchMatching serviceJava 8, Airflow, Sqoop, AWS (EMR, EC2, ECR, Fargate), Docker, Terraform, Dropwizard • Design and development of an application to match usages against assets, to determine which tracks (songs) belong to our publishers. The improved matches enabled us to claim more money for our publishers. • Running the application on EC2, tuning the Java VM and performing runtime analysis to find bottlenecks. Fixing issues with hashcode implementations and caching to improve the performance by 100 times.Legacy Application Java Swing, PL/SQL, Oracle 12.1