● Involved in the entire Software Development Life Cycle – Analysis, Design, and Development – and coordinated with the team using Agile Methodologies.● Implemented PySpark for data processing to handle data from various RDBMS and streaming sources; used Redshift for the Data Warehouse service.● Built an end-to-end data pipeline and performed analytics using the AWS stack (EMR, EC2, S3, RDS, Lambda, Glue, SQS, Redshift).● Implemented end-to-end data quality checks and monitoring… Show more ● Involved in the entire Software Development Life Cycle – Analysis, Design, and Development – and coordinated with the team using Agile Methodologies.● Implemented PySpark for data processing to handle data from various RDBMS and streaming sources; used Redshift for the Data Warehouse service.● Built an end-to-end data pipeline and performed analytics using the AWS stack (EMR, EC2, S3, RDS, Lambda, Glue, SQS, Redshift).● Implemented end-to-end data quality checks and monitoring alarms for ETL pipelines, minimizing data delay friction.● Solely maintained Amazon Redshift cluster and AWS architecture, automating data pipelines and supporting data lakes.● Developed data pipelines to extract data from various sources, transformed it as per the business requirements, and loaded it into Redshift for analysis.● Employed AWS Kinesis to capture real-time data from various sources, handling terabytes of data.● Worked on CI/CD solution, using Git and Jenkins to set up and configure the big data architecture on AWS cloud platform.● Responsible for writing Unit Tests and deploying production level code with the assistance of Git version control. Show less

● Served as a Data Engineer, responsible for Data modelling, Data migration and Designing ETL pipelines for both cloudand on-premises environments.● Possess strong knowledge of Apache Spark components including Spark CORE, Spark SQL, and Spark Streaming● Configured Spark Streaming to receive real time data from Apache Kafka and store streamed data to HDFS using Scalaand PySpark.● Involved in loading and transforming large datasets of Structured, Semi-Structured and… Show more ● Served as a Data Engineer, responsible for Data modelling, Data migration and Designing ETL pipelines for both cloudand on-premises environments.● Possess strong knowledge of Apache Spark components including Spark CORE, Spark SQL, and Spark Streaming● Configured Spark Streaming to receive real time data from Apache Kafka and store streamed data to HDFS using Scalaand PySpark.● Involved in loading and transforming large datasets of Structured, Semi-Structured and Unstructured data and analyzed them using Hive queries.● Performed Extract, Transform and Load operations from Source Systems to Azure Data Storage services leveragingAzure Data Factory, Spark SQL and Azure Data Lake Analytics.● Executed Data Ingestion to Azure Services like Azure Data Lake, Azure Storage, and Azure SQL DW, processing data in Azure Databricks.● Developed Pipelines in ADF using Linked Services, Datasets, and Pipelines to Extract, Transform, and load data from various sources like Azure SQL, Blob storage, Azure SQL Data warehouse.● Imported data from multiple sources and created automated reports in Power BI after cleaning and preprocessing the data. Show less

- Facilitated both in-person and virtual meetings to gather and document business requirements. Secured signoffs from stakeholders to ensure alignment with project goals.- Effectively converted business requirements and user expectations into detailed specifications using Unified Modeling Language (UML). Utilized tools like ERWIN and MySQL Workbench to create accurate data models and diagrams.- Developed ad-hoc reports using Teradata and Tableau to address critical business needs… Show more - Facilitated both in-person and virtual meetings to gather and document business requirements. Secured signoffs from stakeholders to ensure alignment with project goals.- Effectively converted business requirements and user expectations into detailed specifications using Unified Modeling Language (UML). Utilized tools like ERWIN and MySQL Workbench to create accurate data models and diagrams.- Developed ad-hoc reports using Teradata and Tableau to address critical business needs. Assisted business partners across various lines of business by delivering actionable insights and data-driven recommendations.- Managed data analysis tasks, including understanding business data requirements, performing ETL processes, and ensuring data accuracy in reports through rigorous testing.- Interacted with project coordinators and end-users to identify and resolve data issues within specified timeframes. Communicated with clients as needed to address concerns and provide solutions.- Ensured that Business Requirement Documents (BRD) and Technical Design Documents (TDD) accurately reflected the requirements. Conducted thorough reviews to confirm alignment with project objectives. Show less