Associate Software Engineer
CurrentTeam: MindbendersResponsible for Streaming Data Platform for millions of records and files.- Migration of records from SQL tables to data lake and AWS Snowflake- File processing achieved through multi-part or file pull. During time on team, created orchestrator that would handle file pull submissions- Lambda is orchestrator that creates EMR cluster- Apache Spark is then used to process large files.Also worked on functionality of spark engine- Adapt to different file formats, such as parquet, Avro, fixed width, and csv- Bug fixes to handle various error handling such as invalid file formats and permissionsOur application is essential to migrate data and tokenization to other teams throughout the company to perform analysis Canary testing.- Scala is the main programming language for this application