Rizwan S Email and Phone Number
10 years experience in Data Engineering field with a powerhouse blend of Big Data, ETL, Architect, Automation, Analytics, Visualization and a robust Python application development background, I bring expertise in PySpark, AWS, Azure, Databricks, and Snowflake. My proven track record includes crafting intricate PySpark scripts, architecting ETL and ML pipelines, and engineering automation applications. Proficiency extends to AWS Data Pipeline, Azure Data Factory, and beyond, enabling precision and scalability.• Designed and implemented end-to-end data architectures encompassing acquisition, integration, storage, processing, and analysis using AWS, Spark, Fivetran, Dbt, Airflow, and Snowflake.• Built robust Spark applications in Databricks for data extraction, transformation, and aggregation, driving valuable business insights.• Developed PySpark scripts for Auto Loader, Delta Live Tables, and spark streaming applications, processing vast data from Kafka, S3, and Kinesis.• Pioneered automated data ingestion in Azure, transferring web services data to Azure SQL DB.• Optimized Spark efficiency using techniques like z-ordering, caching, and optimized joins.• Proficiency in Hadoop ecosystem components including HDFS, YARN, MapReduce, Sqoop, Hive, and Kafka.• Expertise in Hive data warehousing, table building, partitioning, optimizing HiveQL queries.• End-to-end Data Warehouse architecture proficiency for OLAP/OLTP data models in BI applications like Tableau.• Understand end-to-end Machine Learning flow and closely work with ML engineers to deploy automated solutions. • Extensive work with Python libraries like Pandas and Numpy for data wrangling and numerical computations.• Mastery of AWS and Azure cloud services for ETL pipelines, data engineering components, handling various file formats, and data modeling.• Extensive knowledge of AWS services including S3, EC2, SQS, RDS, EMR, Kinesis, Lambda, Glue, Redshift, DynamoDB, Elasticsearch, CloudWatch, IAM, and Azure services including Blob Storage, Data Lake, Synapse Analytics, Azure Functions, Azure Data Factory, Cosmos DB.• Proficient in designing custom reports using Tableau and Power BI, including algorithm development based on business cases.• Proficient in CI/CD and version control systems like Git, GitHub, GitLab, SVN, Bamboo, and Bitbucket.• Strong communication skills for mentoring junior data engineers and effective stakeholder communication.Domain- Financial Services Health CareTelecommunications Risk
-
Federal Home Loan Bank Of ChicagoUnited States -
Senior Data EngineerFederal Home Loan Bank Of Chicago Feb 2023 - PresentChicago, Il, UsAs a Senior Data Engineer Consultant, I led the design and implementation of end-to-end cutting-edge Data Lakehouse in Databricks for multiple business units by transforming on-premises traditional architecture to medallion architecture for a project that involves 200+ data sources. Collaborated with Machine Learning and data analysts to define optimized data schemas and structures for efficient querying and reporting in Tableau, resulting in a reduction in report generation time. This architecture optimized query performance by facilitating immediate Business Intelligence insights and enabling Machine Learning experimentation with a reduction in model development time. This unified platform established a scalable foundation for ongoing data-driven innovation, positioning the organization at the forefront of adaptability to future data demands.> Designed and developed Pyspark and Python scripts in Databricks to pull data from cloud-based SAAS application’s API and apply transformations to them. > Designed and architected multiple solutions for Data Lakehouse automated pipelines in Databricks. > Developed PySpark and Python scripts to build Databricks notebooks and Delta Live Tables, employing query optimization, parallelization, and distributed computing clusters for efficient and huge data processing.> Developed and automated Spark streaming pipelines using Delta Live Tables and Auto Loader functionality in Databricks. > Orchestrated data transformation pipelines using dbt, enabling efficient and scalable modeling of analytics-ready datasets.> Developed and configured Fivetran data connectors to establish real-time data pipelines, facilitating near-instantaneous data availability for time-sensitive analytics needs.> Integrated Fivetran connectors for automated data ingestion, incorporating change data capture (CDC) techniques to ensure near-real-time data availability while reducing ingestion development time.
-
Senior Data EngineerMorgan Stanley Oct 2020 - Jan 2023New York, Ny, Us> Developed data processing tasks using PySpark such as reading data from external sources, merging the obtained data, performing data enrichment, and loading into data warehouses for risk analytics. > Performed the transformations and actions on the imported data from AWS S3 using PySpark. > Scheduled Apache Airflow DAGs to export the data to AWS S3 buckets by triggering to invoke an AWS lambda function. > Built S3 buckets, managing policies for S3 buckets and Glacier for storage and backup on AWS. > Migrated data from various data sources to Snowflake using AWS Database Migration Service (DMS) through Amazon S3 Bucket to Snowflake.> Performed and developed sophisticated SQL queries, views, functions, and reports that qualify customer requirements and built a dashboard from data stored in Snowflake. > Designed data model in Snowflake Datawarehouse and virtual warehouse’s credits usage monitoring, calculating query load and checking for excessive credit usage by using load monitor charts. -
Senior Data EngineerHca Healthcare Jun 2018 - Oct 2020Nashville, Tn, Us> Developed PySpark Scripts to process streaming data ingested from data lakes using Spark Streaming. > Developed data processing pipelines using PySpark such as reading data from external sources, merging the obtained data, performing data enrichment, and loading into data warehouses. > Developed Pyspark applications in Azure Databricks data extraction, transformation and aggregation from multiple file formats & transformed the data to uncover business insights. > Implemented Spark optimizations for an effective and efficient process in Azure Databricks. > Developed an automated process in Azure cloud by Azure functions which can ingest data on day-to-day basis from web service and load into Azure SQL DB.> Developed robust pipelines in Azure Data Factory to move data from on perm to Azure SQL Datawarehouse.> Extended the capabilities of Data Frames by User Defined Functions in PySpark. > Performed transformations, read, write operations from various data sources by optimizing the Spark SQL queries, and saved the results in HDFS. > Developed aggregate functions using SparkSQL to build interactive querying. > Performed transformations, cleaning and filtering the imported data using Spark Data Frame API and loaded final data into Hive. -
Data EngineerT-Mobile Sep 2015 - May 2018Bellevue, Wa, Us> Developed applications using Python web frameworks like Flask, Web2py, and Python Servlet Engine. > Used AWS Data pipeline for Data Extraction, Transformation, and Loading from homogeneous or heterogeneous data sources and built various graphs for business decision-making using Python matplot library. > Scheduled & monitor the Apache oozie to run multiple Hive and Spark jobs, which independently run with time and data availability.> Performed multiple MapReduce jobs in Hive for data cleaning and pre-processing and worked with HDFS file formats like Avro, Sequence File, and various compression formats like Snappy and gzip. > Worked on Kafka REST API to collect and load the data on HDFS. > Strong working experience in importing data using Sqoop from various RDMS like Teradata, Oracle to HDFS and performed transformations on it by using Spark.> Extensively used Apache Sqoop for efficiently transferring the bulk data between Apache Hadoop and relational databases. -
Data AnalystSwisslog Aug 2014 - Aug 2015Buchs/Aarau, Ag, Ch> Interacted with Business Analysts and Developers in identifying the requirements, designing, and implementing the Datawarehouse Schema. > Documenting and maintained database system specifications, diagrams, and connectivity charts. > Participated in T-SQL code reviews and technical quality standards reviews with the development teams. > Developed and optimized Stored Procedures, Views, and User-Defined Functions for the Application. > Identified Relationships between tables and enforce referential integrity using foreign key constraints. > Created Functional Design Documents and Transaction Definition Documents. > Implemented metadata standards, data governance and stewardship, master data management, ETL, ODS, data warehouse, data marts, reporting, dashboard, analytics, segmentation, and predictive modelling > Designed dashboards and reports, parameterized reports, predictive analysis in Power BI.
Rizwan S Education Details
-
Jawaharlal Nehru Technological UniversityEngineering
Frequently Asked Questions about Rizwan S
What company does Rizwan S work for?
Rizwan S works for Federal Home Loan Bank Of Chicago
What is Rizwan S's role at the current company?
Rizwan S's current role is Sr Cloud Data Analytics Engineer | Architect | Spark | Databricks | Snowflake | Azure | AWS | Python | Airflow | Big Data | DataOps | Automation.
What schools did Rizwan S attend?
Rizwan S attended Jawaharlal Nehru Technological University.
Who are Rizwan S's colleagues?
Rizwan S's colleagues are Christopher Cummings, Layal Khouri, Mba, Ma, Maria Wynne Lascarro, Tommy Payne, Jack Luzzo, Wayne Sochacki, Nancy Schachman.
Free Chrome Extension
Find emails, phones & company data instantly
Aero Online
Your AI prospecting assistant
Select data to include:
0 records × $0.02 per record
Download 750 million emails and 100 million phone numbers
Access emails and phone numbers of over 750 million business users. Instantly download verified profiles using 20+ filters, including location, job title, company, function, and industry.
Start your free trial