Data Engineer
Current● Led the design and deployment of scalable data lakes using Apache Kafka, consolidating varied data sources for enhanced real-time analytics and streamlining data access for cross-functional teams.● Orchestrated the implementation of a robust data governance strategy using Databricks, safeguarding data quality, consistency, and compliance, and leading to a 30% reduction in data discrepancies.● Optimized SQL queries, procedures, and window functions using Numpy and Oracle Data Integrator, enhancing overall system performance.● Identified diverse data sources, constructed efficient data pipelines with Apache Airflow, and ensured data integrity, facilitating seamless data consumption and analysis in formats like Parquet and JSON.● Leveraged advanced tools like Hue-Hive, Presto, and REST APIs to analyze large datasets, driving insightful decision-making.● Implemented continuous delivery (CI/CD) practices using Docker and Kubernetes, automated end-to-end processes, and monitored using Cloudwatch, reducing manual efforts.● Developed impactful business insight interactive visualizations with Power BI, Tableau, and integrated real-time data using REST, providing key metrics and trends to stakeholders.● Utilized S3, S3 Glacier Deep Archive, and managed permissions with IAM for optimized storage and backup processes, ensuring data availability and security with a 99.9% reliability rate.● Collaborated closely with engineering and cross-functional teams, utilized optimal Jira, Confluence, and Oracle Data Integrator for efficient data integration and project tracking, resulting in a 20% reduction in project delivery time.
