Data Engineer
CurrentDesigned, implemented, and maintained scalable data pipelines to collect, process, and transform structured and unstructured data from various sources.Worked on modifications of the SQL code as per the requirement to build the pipeline.Monitoring of the Jobs scheduled in Control M.Successfully ingested the job end-to-end using microservices.Changes performed on the jobs in both Control M and the Registration portal as per the requirement.Implemented data transformations and aggregations using SQL.Investigating the cause of the job failure and fixing the issues.Performed data ingestion.Performed history load of the data in EDH and validated.Changes performed both on data and metadata of the objects.Worked on creating the feeds for FISERV data ingestion.Worked on databricks and Azure services.Deployed and managed data processing systems using CI/CD pipelines in Azure Devops.Implemented the changes in the microservices while transitioning to the Unity catalog.Created and managed clusters in databricks, installation of necessary libraries on the cluster, declaring the global parameters and modifying the changes accordingly in the required microservices.Interacted with databricks team on a regular basis.Well-versed with Confluence, JIRA and Linux commands.Mentored and guided team members and new joiners.Unit testing has been performed on every modification to the code.Maintained documentation and Issue trackers.Conducted data analysis on data marts utilizing SQL, providing valuable insights to the data modeling team.Responsible for Data Modelling and Data Mapping.Gather metadata information for the necessary data tables.Performed Data profiling and provided valuable inputs by identifying data redundancies, spelling mistakes across different layers.Collected business names and definitions for all column-table pairs through collaboration with clients and utilization of internal tools.Validated reports to ensure accuracy and consistency.