Senior Data Engineer
Current• Worked on AWS Data pipeline to configure data loads from S3 to into Redshift.• Using AWS Redshift, I Extracted, transformed and loaded data from various heterogeneous data sources and destinations• Created Tables, Stored Procedures, and extracted data using T-SQL for business users whenever required.• Performs data analysis and design, and creates and maintains large, complex logical and physical data models, and metadata repositories using ERWIN and MB MDR• I have written shell script to trigger data Stage jobs.• Assist service developers in finding relevant content in the existing reference models.• Like Access, Excel, CSV, Oracle, flat files using connectors, tasks and transformations provided by AWS Data Pipeline.• Utilized Spark SQL API in PySpark to extract and load data and perform SQL queries.• Worked on developing Pyspark script to encrypting the raw data by using Hashing algorithms concepts on client specified columns.• Responsible for Design, Development, and testing of the database and Developed Stored Procedures, Views, and Triggers• Developed Python-based API (RESTful Web Service) to track revenue and perform revenue analysis.• Compiling and validating data from all departments and Presenting to Director Operation.• KPI calculator Sheet and maintain that sheet within SharePoint.• Created Tableau reports with complex calculations and worked on Ad-hoc reporting using PowerBI.• Creating data model that correlates all the metrics and gives a valuable output.• Worked on the tuning of SQL Queries to bring down run time by working on Indexes and Execution Plan.• Performing ETL testing activities like running the Jobs, Extracting the data using necessary queries from database transform, and upload into the Data warehouse servers.• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.