• Data Engineer with 9+ years of combined experience in the building data solutions using Azure Services like Azure SQL Db, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Databricks and Big Data implementations like Spark, Hive, Kafka and HDFS including programming languages such as Python, Scala and Java. • Experience in Big Data ecosystems using Hadoop, Pig, Hive, HDFS, MapReduce, Sqoop, Storm, Spark, Airflow, Snowflake, Teradata, Flume, Kafka, Yarn, Oozie, and Zookeeper. • High Exposure on Big Data technologies and Hadoop ecosystem, In - depth understanding of Map Reduce and Hadoop Infrastructure. • Expertise in writing end to end Data processing Jobs to analyze data using MapReduce, Spark and Hive. • Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and Migrating On premise databases to Azure Data Lake store using Azure Data factory. • Experience in working with Azure Cloud Space using Azure Components like Azure Data Factory, Azure Databricks, Azure Data Lake store, Azure storage accounts (Generation1 &Generation2), Logic Apps, Azure Key Vault. • Good understanding of Big Data Hadoop and Yarn architecture along with various Hadoop Demons such as Job Tracker, Task Tracker, Name Node, Data Node. • Experience with Apache Spark ecosystem using Spark-Core, SQL, Data Frames, RDD's and knowledge on Spark MLLib. • Hands on experience on Google Cloud Platform (GCP) in all the bigdata products BigQuery, Cloud Data Proc, Google Cloud Storage, Composer (Air Flow as a service). • Strong experience in Business and Data Analysis, Data Profiling, Data Migration, Data Integration, Data governance and Metadata Management, Master Data Management and Configuration Management. • Experience in implementing various Big Data Engineering, Cloud Data engineering, Data Warehouse, Data Mart, Data Visualization, Reporting, Data Quality, and Data virtualization Solution • Experience in analyzing data using Python, R, SQL, Microsoft Excel, Hive, PySpark, Spark SQL for Data Mining, Data Cleansing, Data Minging and Machine Learning. • Hands - on experience in Azure Cloud Services (PaaS & IaaS), Azure Synapse Analytics, SQL Azure, Data Factory, Azure Analysis services, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake.
Mastercard
View- Website:
- mastercard.com
- Employees:
- 42669
-
Workday Integration AnalystMastercardMontgomery, Al, Us
-
Senior Data EngineerExabeam Apr 2023 - PresentFoster City, California, United States• Worked on various technologies like Azure Databricks, Azure Data Factory, Azure Synapse Analytics, HDInsight, and other technologies in Azure Services. • Have In-depth knowledge of Databricks Architecture. • Integrated Azure Databricks with ADLS Gen1, Gen2, Cosmos Db, EventHub’s, DevOps to analyze the data and to do transformations • Deployed Databricks clusters using POST Man tool. • Setup end to end environment based on the requirements. • Assisted other team members towards the resolutions of the issues in Azure. • Invested time to understand Databricks product from the ground up collaborating with our Databricks partner and engineering team. • Created Azure Data Factory for copying data from Azure BLOB storage to SQL Server. • Developed ADF Pipelines using the Data Factory Activities like Copy Data, Data Lake Analytics • Created Azure Data Factory, setting up the integration Runtime to connect customer On-Prem environment to Azure Cloud. • Worked on creating the Azure Data Factory Pipelines to process the files from On-Premises. • Designed end to end scalable architecture to solve business problems using various Azure Components like HDInsight, Data Factory, Data Lake, Storage and Machine Learning Studio. • Designed and architected scalable data processing and analytics solutions, Including technical feasibility, integration, development for Big Data storage processing and consumption of Azure data, analytics, big data (Hadoop, park), business intelligence (Reporting Services, Power Bl), NoSQL, HDInsight stream Analytics, Data Factory, Event Hubs, and Notification Hubs. • Participated in daily stand-up meetings to update the project status with the internal Dev team. • Writing Python scripts to load data from Web APIs to staging DB. • Reverse-engineered existing data models to incorporate new changes utilizing Erwin. -
Senior Data EngineerFifth Third Bank Jul 2021 - Mar 2023Evansville, Indiana, United States• Involved in converting Hive/SQL queries into transformations using Python • Performed complex joins on tables in hive with various optimization techniques • Created Hive tables as per requirements, internal or external tables defined with appropriate static and dynamic partitions, intended for efficiency. • Developed Python application for Google Analytics aggregation and reporting and used Django configuration to manage URLs and application parameters. • Initiated the development and implementation of website user clickstream data analytics in Hadoop/Hive. • Worked extensively with HIVE DDLS and Hive Query language (HQLs) • Involved in loading data from edge node to HDFS using shell scripting. • Understand and manage Hadoop Log Files. • Develop and deploy the outcome using spark and Scala code in Hadoop cluster running on GCP. • Private Cloud (VPC), Cloud Formation, Lambda, Cloud Front, Cloud Watch, IAM, EBS, Security Group, Auto Scaling, Dynamo DB, Route53, and Cloud Trail. • Worked with Impala for massive parallel processing of queries for ad-hoc analysis. Designed and developed complex queries using Hive and Impala for a logistics application. • Worked on Bigdata technologies like Hive, Impala, HDFS, Oozie workflows for ingesting data from different sources to audit layer and tan to harmonized layer • Created Bash scripts to add dynamic partitions to Hive staging tables. Responsible for loading bulk amount of data into HBase using MapReduce jobs. • Involved in various Transformation and data cleansing activities using various Control flow and data flow tasks in SSIS packages during data migration • Build data pipelines in airflow in GCP for ETL related jobs using different airflow operators both old and newer operators.• Manage Hadoop infrastructure with Cloudera Manager. • Involved in moving all log files generated from various sources to HDFS for further processing through Flume. -
Data EngineerMacy'S Dec 2018 - Jun 2021New York, United States• Analyse, design and build Modern data solutions using Azure PaaS service to support visualization of data. Understand current Production state of application and determine the impact of new implementation on existing business processes. • Implemented Proof of concepts for SOAP & REST APIs • REST APIs to retrieve analytics data from different data feeds • Developed various shell scripts and python scripts to address various production issues. • Extracted the needed data from the server into HDFS and Bulk Loaded the cleaned data into HBase. • Designed and maintained databases using Python and developed Python based API (RESTful Web Service) using Flask, SQL Alchemy, PLSQL and PostgreSQL. • Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analysing & transforming the data to uncover insights into the customer usage patterns. • Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark data bricks cluster. • Enhanced scripts of existing Python modules. Worked on writing APIs to load the processed data to HBase tables. • Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines • Hands-on experience on developing SQL Scripts for automation purpose. • Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS). -
Data EngineerCybage Software Private Limited Apr 2016 - Sep 2018Hyderabad, Telangana, India• Analyse, design and build Modern data solutions using Azure PaaS service to support visualization of data. Understand current Production state of application and determine the impact of new implementation on existing business processes. • Implemented Proof of concepts for SOAP & REST APIs • REST APIs to retrieve analytics data from different data feeds • Developed various shell scripts and python scripts to address various production issues. • Extracted the needed data from the server into HDFS and Bulk Loaded the cleaned data into HBase. • Designed and maintained databases using Python and developed Python based API (RESTful Web Service) using Flask, SQL Alchemy, PLSQL and PostgreSQL. • Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analysing & transforming the data to uncover insights into the customer usage patterns. • Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark data bricks cluster. • Enhanced scripts of existing Python modules. Worked on writing APIs to load the processed data to HBase tables. • Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines • Hands-on experience on developing SQL Scripts for automation purpose. • Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS).
-
Sql Data AnalystHudda Infotech Private Limited Oct 2014 - Mar 2016Hyderabad, Telangana, India• Designed and created Data Marts in data warehouse database Implementations of MS SQL Server Management studio 2008 to create Complex Stored Procedures and Views using T-SQL. • Collecting the data from many resources and converting into flat text files with comma delimiter separator and importing the data to the SQL server for data manipulations. • Responsible for deploying reports to Report Manager and Troubleshooting for any errors during the execution. • Scheduled the reports to run on daily and weekly basis in Report Manager and emailing them to director and analyst to be reviewed in Excel Sheets. • Created several reports for claims handling which had to be exported out to PDF formats. • Analyzed business requirements and provided excellent and efficient solutions. • Contributed to JRD sessions to gather requirements and define functional requirements. • Utilized Python for recurring reports automation and visualized them on the BI platform. • Worked collaboratively within and across development and project teams in a fast-paced work environment, utilizing Agile BI design. • Designed complex T-SQL queries and user-defined functions in SQL server. • Utilized T-SQL queries and views based on business reporting requirements, performance tuning, and various complex query optimization. • Implemented dynamic SQL to develop customizable queries, answerable by the OLTP Server. • Removed duplicate records by cleansing data utilizing ranking functions and CTEs in SQL. • Created ETL package for data conversion using various transformations tasks. • Worked on Dimensional modeling, Data migration, Data cleansing, and ETL Processes for data warehouses. • Incorporated SSIS to load data into the data warehouse’s data mart with star schemas. • Created summary and detail reports utilizing drill down and drill through functionalities in SSRS. • Optimized SQL queries with execution plans to enhance performance.
Venkat Sai Education Details
-
Computer Science
Frequently Asked Questions about Venkat Sai
What company does Venkat Sai work for?
Venkat Sai works for Mastercard
What is Venkat Sai's role at the current company?
Venkat Sai's current role is Workday Integration Analyst.
What schools did Venkat Sai attend?
Venkat Sai attended Jntuh College Of Engineering Hyderabad.
Who are Venkat Sai's colleagues?
Venkat Sai's colleagues are Gabriel Scarabelli, Steen Bo Hansen, Pawan Vyas, Siddharth Vadgama, Joan Mccraw, Jaya Dubey, María Martínez López.
Not the Venkat Sai you were looking for?
-
-
Venkat Sai
Senior Java Full-Stack Developer At Pepsico ||Java 11 || Mongo Db || Sql Loader || Angular Js || React Js|| Nodejs || Jquery || Bootstrap|| Jenkins|| Github|| Angular 10 & 12|| Aws Sns/Sqs ||Mckinney, Tx -
Venkat Sai
Azure Data Engineer | Ai & Machine Learning Enthusiast | Actively Looking For C2C Remote RolesAtlanta, Ga -
-
Venkat sai
Lake St Louis, Mo
Free Chrome Extension
Find emails, phones & company data instantly
Aero Online
Your AI prospecting assistant
Select data to include:
0 records × $0.02 per record
Download 750 million emails and 100 million phone numbers
Access emails and phone numbers of over 750 million business users. Instantly download verified profiles using 20+ filters, including location, job title, company, function, and industry.
Start your free trial