Data Engineer
Current- Designed and developed data pipelines in Databricks workflows to enable efficient data processing.- Migrated data pipelines from on-premise databases (Vertica) to Delta Live Tables on the Databricks platform.- Utilized Pyspark to consume data from Kafka for batch data processing in Databricks, enabling streamlined data ingestion and transformation.- Collaborated with the data science team to establish data pipelines that effectively support the AI project.- Built datamarts to support business requirements for dashboards, AI, etc.- Used the behave framework for unit and integration ETL testing.- Communicated with the data source team to create specs of the data pipeline to support business requirements.