Data Engineer
CurrentAs a Data Engineer at BMO, I’ve been integral in enhancing data operations, focusing on cloud technologies and data engineering. My role encompasses:• Data Migration and Cloud Integration: Led the migration of data pipelines to Azure Synapse and ADLS from Hadoop systems, ensuring seamless integration and data transfer.• Real-Time Data Processing: Implemented Kafka and Spark Structured Streaming, significantly boosting real-time data analytics and responsiveness.• Cloud Data Management: Managed Azure Data Lake and Blob Storage, optimizing these platforms for scalability and data accessibility.• Data Analysis and BI Tools: Proficient in tools like Tableau and Splunk, conducting regression analyses and developing insightful visual representations for business intelligence.• Databricks and Azure Management: Automated Databricks notebooks with SQL and Python, and managed high concurrency clusters in Azure Databricks, enhancing data preparation and quality.• Data Pipeline Architecture: Developed and maintained data pipeline architecture in Azure, using Data Factory and Databricks, and implemented solutions in Azure SQL Data Warehouse.• Team Collaboration and Project Management: Collaborated with SMEs, data scientists, and developers through all stages of data product development, aligning projects with business objectives.• Data Design and Systems Development: Led data design and managed large datasets, actively engaged in developing OLTP and OLAP systems from the ground up.• Agile Practices: Embraced Agile methodology for project execution, ensuring effective team communication and alignment with project goals.• Data Security and Compliance: Maintained stringent data security and compliance standards.This role has been a significant platform for growth in data engineering, cloud technologies, and business intelligence, enabling me to contribute effectively to BMO’s data strategy.