Data Scientist
CurrentAs a Data Engineer at Global Atlantic Financial Group, I specialize in developing robust Spark applications using Scala to handle data from various RDBMS and streaming sources, optimizing performance on EMR clusters through effective configuration and tuning. My role involves creating tables, executing complex queries, and implementing schema extraction for file formats, ensuring optimized storage and query performance using AWS S3. I also developed complex data pipelines using Pandas for data cleaning and aggregation, utilized NumPy for numerical computations, and implemented data transformations with Spark and PySpark