Data Engineer
Current- Designed and implemented data models using the medallion architecture on Databricks, including dimension and fact tables, utilizing DBT for efficient data transformation.
- Updated and optimized semantic models using Tabular Editor to enhance data accessibility and analysis.
- Developed and managed end-to-end data pipelines and sensors using Airflow and Dagster, ensuring seamless data workflows and integrations.
- Utilized MLflow and AutoML on Databricks for tracking experiments, managing machine learning models, and automating model selection and hyperparameter tuning.
- Performed comprehensive data cleaning to ensure data quality and accuracy.
- Extracted and processed structured data from PDFs using OCR techniques, and developed RAG-based prompts for LLM models like DBRX and Llama to convert unstructured data into actionable insights.
May 2024 - Present
