Shashank Gupta Email & Phone Number

Idaho, United States

• Leading a team of 7 to develop and test python automations using PySpark, to carry out validation automation.• Worked on identifying root cause and solving the issue collaborating with cross-functional teams.• Worked on building user stories and gathering requirements to create stories in JIRA and documenting the processes in Confluence.• Experience in working with day-to-day ad-hoc tasks, using GCP and airflow for smooth processing of the deliverables.• Experience in working with teams both off-shore and on-shore to plan and execute the sprint as per Agile requirements.• Gathering of Requirements from various systems business users.• Communicating with the business users to understand business requirements clearly• Conducting meetings with the data architects to understand the source system elements• Provided support to other developers in accurately mapping source attributes into the Teradata Financial Services Logical• Data Model (FSLDM) and in interpreting business requirements• Worked closely with the Data Modeler and Data Architect in enhancing the FSLDM• Development of scripts for loading the data into the base tables in EDW using FastLoad, MultiLoad and BTEQ utilities of Teradata.• Implemented logical and physical data modeling with techniques using Erwin in Data Mart• Extracted data from various source systems like Oracle, Sql Server and flat files as per the requirements.• Worked on testing primary index and skew ratios before populating data into tables using sampling techniques, explain plan in Teradata before querying large tables with several joins.• Performed tuning and optimization of complex SQL queries using Teradata Explain.• Created a BTEQ script for pre population of the worktables prior to the main load process.• Used various Teradata Index techniques to improve the query performance• Writing UNIX shell scripts for processing/ cleansing incoming text files.

• Defined the vision and strategy for data governance implementation for clients.• Led teams of Software Engineers, Data Engineers, Data Quality Analysts, and Data Analysts to build complex and innovative solutions. • Managed the data infrastructure, overseeing the planning and execution of testing efforts to ensure accuracy and performance measures are being met, in securing the overall success of business intelligence for the client.• Designed, built, deployed, and customized complex data governance and analytics solutions with the goal of creating a customized and complex data ecosystem to influence test design, manual and automated test execution, and reporting. • Established data strategy, data policies, data standards, organization, and administration. • Synthesized and documented dataflows for critical business processes to maintain data policies.• Established data quality policy, procedures, and prepared reports to communicate trends and identify new opportunities for data quality improvement and compliance.• Coordinated projects across multiple operations teams including development, testing, infrastructure, data, and other teams.• Built data technology integration blueprints including distributed data architectures.• Developed data governance models including data source profiling, data Modeling, data quality, master data management processes, dashboards, and metrics.• Developed data quality control models for effective measurement of data quality efforts.• Translated business requirements into technical specifications, established and define details, definitions, and requirements of critical applications, components, and enhancements.• Developed operational definitions and managed data integrity on mandated report requirements.• Participated in project planning, identifying milestones, deliverables, and resource requirements to track activities and execution of tasks.

New Jersey, United States

• Designing and Developing Apache NiFi jobs to get the files from transaction systems into data lake raw zone. • Analyzed the user requirements and implemented the use cases using Apache NiFi. • Worked on reading and writing multiple data formats like JSON, ORC, Parquet on HDFS using PySpark. • Experienced with Azure Micro Services, Azure Functions and Azure Solutions. • Hands on experience on Azure VPN-Point to Site, Virtual networks, Azure Custom security, end point security and firewall. • Caching, SQL Azure, NoSQL, Storage, Network services, Azure Active Directory, API Management, Scheduling, Auto Scaling, and PowerShell Automation. • Involved in developing the Azure Solution and Services like IaaS and PaaS. • As a Hadoop Developer, my role is to manage the Data Pipelines and Data Lake. • Have experience of working on Snow -flake data warehouse and have hands-on experience working with SnowSQL. • Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data. • Designed custom Spark REPL application to handle similar datasets. • Used Hadoop scripts for HDFS (Hadoop File System) data loading and manipulation. • Performed Hive test queries on local sample files and HDFS files. • Developed the application on Eclipse IDE. • Worked on Configuration of Internal load balancer, load balanced sets and Azure Traffic manager.• Developed Hive queries to analyze data and generate results. • Used Spark Streaming to divide streaming data into batches as an input to Spark Engine for batch processing. • Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, hive, HBase, Spark and Sqoop. • Exported data from HDFS to RDBMS via Sqoop for Business Intelligence, visualization, and user report generation. • Used Scala to write code for all Spark use cases. • Analyzed user request patterns and implemented various performance optimization measures including implementing partitions and buckets in HiveQL.

Jacksonville, Florida, United States

• Have actively taken part in Data Profiling, Data Cleansing, Data Migration, Data Mapping and actively helped developers to Compare data with original source documents and validate Data accuracy.• Worked on Tableau, to create dashboards and visualizations.• Data profiling and data quality checkup on the data for multiple units to ensure it met the specified business rules and standards set within the organization.• Partitioning and bucketing ideas were implemented in the Apache Hive database, which increases query retrieval performance.• Implemented indexes on OLTP/OLAP tables to improve navigation performance within large datasets• Configured Policy Center with tight document/imaging support, new screens, revised assignment logic and other standard rules/screen/UI changes.• Migrated data from multiple sources into QlikView applications, performed detailed analysis of source system data and model that data in QlikView.• Performed end- to-end Architecture & implementation assessment of various AWS services like Amazon EMR, Redshift, S3, Athena, Glue and Kinesis.• Fixed software bugs via Javascripts and CSS.• Designed, developed, and tested QlikView scripts to import data from source systems and test QlikView reports/dashboards to meet customer requirements.• Designed configuration of MS Dynamics CRM as per the business user's requirements and needs.• Assisted in migration of current CRM to MS Dynamics CRM with added features and functionality.

Raleigh, North Carolina, United States

• Written Hive queries for data analysis to meet the business requirements • Migrated an existing on premises application to AWS. • Developed PIG Latin scripts to extract the data from the web server output files and to load into HDFS • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting • Created many Sparks UDF and UDAFs in Hive for functions that were not preexisting in Hive and Spark SQL. • Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala. • Implementing different performance optimization techniques such as using distributed cache for small datasets, partitioning, and bucketing in hive, doing map side joins etc. • Good knowledge on Spark platform parameters like memory, cores, and executors • By using Zookeeper implementation in the cluster, provided concurrent access for Hive Tables with shared and exclusive locking. • Using Sqoop to import and export data from Oracle and PostgreSQL into HDFS to use it for the analysis. • Migrated Existing MapReduce programs to Spark Models using Python. • Migrating the data from Data Lake (hive) into S3 Bucket. • Done data validation between data present in Data Lake and S3 bucket. • Used Spark Data Frame API over Cloudera platform to perform analytics on hive data. • Designed batch processing jobs using Apache Spark to increase speeds by ten-fold compared to that of MR jobs. • Used Kafka for real time data ingestion. • Created different topic for reading the data in Kafka • Read data from different topics in Kafka. • Involved in converting the HQL's in to spark transformations using Spark RDD with support of python and Scala • Moved data from S3 bucket to Snowflake Data Warehouse for generating the reports.

Hyderabad, Telangana, India

• Recommended structural changes and enhancements to systems and Databases. • Conducted Design reviews and technical reviews with other project stakeholders. • Was a part of the complete life cycle of the project from the requirements to the production support • Created test plan documents for all back-end database modules • Used MS Excel, MS Access, and SQL to write and run various queries. • Worked extensively on creating tables, views, and SQL queries in MS SQL Server. • Worked with internal architects and assisting in the development of current and target state data architectures. • Created Hive target tables to hold the data after all the PIG ETL operations using HQL. • Analysed the data by performing Hive queries and running Pig scripts to know user behaviour. • Worked on migrating existing mainframe data and reporting feeds to Hadoop. • Integrated Hadoop with Tableau and SAS analytics to provide end users analytical reports • Worked with Hadoop in Stand-alone, pseudo and distributed modes. • Coordinate with the business users in providing appropriate, effective, and efficient way to design the new reporting needs based on the user with the existing functionality. • Remain knowledgeable in all areas of business operations to identify systems needs and requirements.

Hyderabad, Telangana, India