Data Engineer
Current• Deployed 30+ various types of Databricks jobs, selecting appropriate compute resources, and ensuring their operation through monitoring and issue resolution.• Managed Databricks compute resources and user access controls to ensure security and compliance.• Engineered a Python framework to streamline data ingestion processes from Azure Blob Storage, S3, SharePoint, and DBMS, applying knowledge of authentication techniques to enhance data security.• Handled large datasets in various formats (CSV, Parquet, Avro, JSON, Binary, etc), performed pre-transformation and developed efficient data loading strategies.• Built and maintained CI/CD pipelines for deployment of Databricks resources and dbt projects, ensuring validation, testing.• Utilize basic BI tools to create ad-hoc visualizations for data verification and analysis.• Domain Knowledge: Retail Sales
