Lead Data Engineer
Current* Migrated on prem (Red Hat, manually deployed) applications to AzureCloud (Docker/Kubernetes/Jenkins CICD) reducing new engineer onbording time, access to on prem servers containing sensitive data and skill level requirements.* Moved application from batch to event driven using Python Kafka (confluent) which increase business decision making speed.* Complimented ingestion calabilities where Azure Data Factory is not supported* Greatly increased the efficiency of legacy python custom parsers to accomodate our modern (exponentially bigger) data loads.* Improved and migrated existing Databricks pipelines from Azure 1.9 to 2.0* Daily use of Linux, Python, Kafka and Synapse SQL pool* Experienced with Databricks, ElasticSearch, Cosmos, Azure Data Lake, PII handling