Rupesh K Email and Phone Number
With 7 years of experience as a Data Engineer, I bring a robust set of skills and accomplishments to benefit Corporations. Below are just a few highlights- • Design and deploy scalable data infrastructures on AWS and Azure, optimizing processing speed.• Develop optimized Spark pipelines and data models using Data Bricks for analyzing Big Data.• Experienced in Data Ingestion projects to inject data into Data Lake from multiple source systems using Denodo.• Design and develop multiple batch processing frameworks to move data from source to gold zone for analytical consumption.• Develop data lakes with ETL Tools and lead the end-to-end implementation of Big Data projects, with comprehensive experience as a Cloud developer in AWS and Azure Ecosystem.• Develop Spark Programs in Data Bricks for analyzing Big Data as per the requirements.• Assist companies in migrating data to Snowflake and AWS using Python, Spark, DBT, and SQL.• Resolve complex database performance issues, including optimizing recursive SQL query.• Proficient in Apache Spark, Kafka, Hadoop, cybersecurity, and Hive for efficient ETL solutions.• Skilled in Python, Scala, Java, UNIX, and LINUX systems.• Architect of cloud-native data warehousing solutions using Snowflake, Redshift, and Azure Synapse Analytics.• Develop interactive data visualizations using Tableau, Power BI, and Quick Sight.• Implement Agile/Scrum methodologies, improving team productivity and project delivery.• Proficient in Git, Bitbucket, and other version control tools for streamlined code collaboration.• Worked on Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, and Pair RDDs.• Leveraged advanced SQL operations for optimized query performance and database maintenance.• Implemented Docker-based CI/CD pipelines orchestrated with Kubernetes for efficient application delivery.• Expertise in using Linear & Logistic Regression and Classification Modeling, Decision-trees, Principal Component Analysis (PCA), and Cluster and Segmentation analyses.• Mitigated risk factors through careful analysis of financial and statistical data. Transformed and processed raw data for further analysis, visualization, and modeling.• Assist in determining the full domain of the MVP, create and implement its relevant data model for the App, and work with App developers integrating the MVP into the App and any backend domains
-
Senior Data EngineerNational Exchange Carrier Association, Inc -
Senior Data EngineerNational Exchange Carrier Association, Inc Apr 2021 - Present• Worked on AWS Data pipeline to configure data loads from S3 to Redshift.• Extract, transform, and load data from various heterogeneous data sources and destinations, through AWS Redshift.• Created tables, stored procedures, and extracted data using T-SQL for business users whenever required.• Perform data analysis and design, create, and maintain large, complex logical and physical data models, and metadata repositories using ERWIN and MB MDR.• Direct experience in writing shell scripts to trigger data stage jobs.• Assist service developers in finding relevant content in the existing reference models like Access, Excel, CSV, Oracle, and flat files using connectors, tasks, and transformations provided by AWS Data Pipeline.• Utilized Spark SQL API in PySpark to extract and load data and perform SQL queries.• Developed PySpark script to encrypt the raw data by using hashing algorithms concepts on client-specified columns.• Responsible for the design, development, and testing of the database and Developed Stored Procedures, Views, and Triggers.• Developed Python-based API (RESTful Web Service) to track revenue and perform revenue analysis.• Compiling and validating data from all departments.• KPI calculator Sheet and maintain that sheet within SharePoint.• Created Tableau reports with complex calculations and worked on Ad-hoc reporting using PowerBI.• Creating data models that correlate all the metrics and give a valuable output.• Worked on the tuning of SQL Queries to bring down run time by working on Indexes and Execution Plans.• Performing ETL testing activities like running the jobs, extracting the data using necessary queries from database transform, and uploading it into the Data Warehouse Servers.• Pre-processing using Hive and Pig.• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.
-
Data EngineerCapital One Oct 2019 - Mar 2021 -
Data EngineerCapital One Oct 2019 - Mar 2021• Achieved seamless migration of on-premises data infrastructure to AWS, leveraging a wide array of tools including Amazon S3, EC2, RDS, Redshift, Glue, EMR, Lambda, Data Pipeline, Athena, QuickSight, CloudFormation, CloudWatch, Step Functions, SQS, and SNS, resulting in a 70% reduction in infrastructure costs, 50% faster data processing, and enhanced scalability to accommodate a 5x increase in data volume.• Achieved successful implementation of a robust data infrastructure on Azure, leveraging key tools including Azure Blob Storage, Azure SQL Database, Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Functions, and Azure Power BI.• Achieved successful migration to Cloudera, leveraging Solace messaging, enabling real-time data processing, enhanced scalability, and advanced analytics capabilities, driving actionable insights for business growth.• Achieved seamless integration of ETL processes by implementing a scalable solution utilizing Hadoop-based tools such as Apache Spark, Apache Kafka, Apache Hadoop, and Apache Hive, in conjunction with Azure services including Azure Data Factory, Azure HDInsight, and Azure Databricks.• Highlighted a prominent level of expertise across various programming languages, encompassing Python, Scala, and Java, as well as extensive familiarity with UNIX and LINUX environments.• Demonstrated leadership in implementing a cloud-native data warehousing solution, harnessing the capabilities of Snowflake, Amazon Redshift, and Azure Synapse Analytics.• Achieved streamlined code collaboration and version control efficiency with Git, Bitbucket, and other tools, resulting in improved workflows and higher code quality.• Maximized query performance and data analysis by utilizing advanced SQL techniques.• Engineered high-performance ETL pipelines, ensuring accurate data extraction, transformation, and loading with a reduction in processing time, facilitating data-driven decision-making.
-
Data AnalystT-Mobile Jan 2016 - Sep 2019 -
Data AnalystT-Mobile Jan 2016 - Sep 2019• Developed a new data model in Sisense to replace the traditional reporting system. Embedded the Sisense dashboards within the application to provide operational reporting.• Built strong relationships with the data scientist, administration, Stakeholders, and customers to provide weekly presentations, KPI reporting, and dashboards.• Designed, built, assessed, and maintained data pipelines, data integration scripts, ETL processes, and data management within Airflow and GCP data warehouse environment using Databricks.• Convert JSON format from AWS S3, Spark, Glue, and Kafka storage facilities into Snowflake data warehouse.• Utilized Python programming to construct DataBase Transformer (DBT) scripts that converted JSON documents into structured table arrangements within designated Snowflake tables, ensuring compatibility between input and output forms.• Employed Python scripting skills to design programs efficiently loading JSON content from buckets into Snowflake repositories. • Implemented validation steps using Python scripts to verify the cohesion of original records against transformed results before entry into specified Snowflake containers.• Developed highly engaging and informative data visualizations using Sisense, effectively presenting complex datasets in a visually compelling manner.• Optimized code collaboration and version control using Git, Bitbucket, and other tools, resulting in improved workflows and elevated code quality.• Utilized advanced SQL techniques to optimize query performance and enable informed decision-making for efficient operations.• Architected efficient ETL pipelines, achieving precise data extraction, transformation, and loading with a decrease in processing time, enabling data-driven decision-making.• Implemented Docker and Kubernetes for seamless CI/CD pipelines, driving streamlined deployment, enhanced scalability, and accelerated release cycles for effective application delivery.
Rupesh K Education Details
-
Master Of Science - Ms -
Bachelor Of Pharmacy - Bpharm
Frequently Asked Questions about Rupesh K
What company does Rupesh K work for?
Rupesh K works for National Exchange Carrier Association, Inc
What is Rupesh K's role at the current company?
Rupesh K's current role is Senior Data Engineer at NATIONAL EXCHANGE CARRIER ASSOCIATION, INC.
What schools did Rupesh K attend?
Rupesh K attended Saint Louis University, Nalla Narasimha Reddy Education Society Group Of Institutions.
Who are Rupesh K's colleagues?
Rupesh K's colleagues are Gregory Stevens, Lindsey Short, Cae, Katie Rogers, Susan Casareale, Alan Schwane, Angie Teten, Susan Grube.
Not the Rupesh K you were looking for?
-
2hotmail.com, assurant.com
2 +180599XXXXX
Free Chrome Extension
Find emails, phones & company data instantly
Aero Online
Your AI prospecting assistant
Select data to include:
0 records × $0.02 per record
Download 750 million emails and 100 million phone numbers
Access emails and phone numbers of over 750 million business users. Instantly download verified profiles using 20+ filters, including location, job title, company, function, and industry.
Start your free trial