Growth Specialist
CurrentDesigned and established data pipelines for diverse data types (CSV, Parquet, XML, JSON, Excel) sourced from various platforms (S3, SQS, BigQuery, APIs) into Databricks' "bronze" tables.Transformed both streaming and batch data, shaping it into "silver" and "gold" zones, ensuring customized data access for end-users.Implemented Data Quality Checks, Validation, and Testing processes to guarantee the reliability of data, facilitating precise business decision-making.Provided mentorship to the team, focusing on Python, Spark, Databricks, and Software Engineering, while also reviewing and optimizing their code.Collaborated with non-technical stakeholders to gather project requirements and design data flow strategies.Constructed impactful data pipelines that handled CMR data transformation and loading.Optimized the migration process from SQL Server to Databricks, resulting in a cost savings of £40k per year by utilizing AWS DMS and Spark Streams.Tech Stack: Python, PySpark, Databricks (Workflows, Delta Live Tables, Autoloader, CDC, Z-ordering, Data Quality Checks, Unity Catalog, Delta Tables, etc.), AWS, Pandas, Git, Azure DevOps.
