Vikas. . Email & Phone Number

United States

• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.• Working extensively on Hive, SQL, Scala, Spark, and Shell.• Involved in creating ETL pipelines to ingest data into Azure datalake.• Experience in designing data-driven solutions.• Involved in building new data sets, and products helping support business initiatives.• Built production quality ingestion pipelines with automated quality checks to help enable the business to access all the data sets in one place.• Automate the pipeline using Airflow by creating dependencies between each job and schedule them based on daily/weekly/monthly and perform quality checks based on consistency, accuracy, completeness, orderliness, uniqueness, and timeliness and use them to determine ETL job.• Developed Spark core and Spark SQL scripts using Scala for faster data processing. Transformed the data using Spark applications for analytics consumption.• Worked on importing data from SQL Server and Oracle Server into Azure Data Lake using Sqoop.• Worked on creating incremental spark jobs to move data from Azure to Snowflake.• Experience working with delta tables in Azure for loading incremental data into Azure.• Experience in using Azure blob storage for storing various data files.• Developed Managed File Transfer jobs for data exchange to and from third-party vendors and Optum.• Developed Scala script for ingesting flat files, csv, json into Data Lake• Worked on giving production support for developed applications to resolve Critical/Priority-1 issue.• Used Kafka connector for reading CDC feed from Cosmos Db, mongo DB in real time into Kafka• Worked on multiple streaming jobs to read data from Kafka, transform and write into Azure data lake• Support our Data Scientists by helping enhance their modeling jobs to be more scalable when modeling across the entire data set.• Experience in productionize and optimizing data science models.

Irving, Texas, United States

• Experience in working with Azure cloud platform (HDInsight, Databricks, DataLake, Blob Storage, Data Factory, Synapse, SQL DB, SQL DWH and Data Storage Explorer).• Involved in building an Enterprise Data Lake using Data Factory and Blob storage, enabling other teams to work with more complex scenarios and ML solutions.• Used Azure DataFactory, SQL API and Mongo API and integrated data from MongoDB, MS SQL, and cloud (Blob, Azure SQL DB). • Developed Spark Scala scripts for mining data and performed transformations on large datasets to provide real time insights and reports.• Supported analytical platform, handled data quality, and improved the performance using Scala’s higher order functions, lambda expressions, pattern matching and collections. • Implemented scalable microservices with Scala and Akka to handle concurrency and high traffic. Optimized existing Scala code and improved the cluster performance.• Performed data cleansing and applied transformations using DataBricks and Spark data analysis.• Used Azure Synapse to manage processing workloads and served data for BI and prediction needs. • Designed and automated Custom-built input adapters using Spark, Sqoop and Airflow to ingest and analyze data from RDBMS to Azure Datalake.• Reduced access time by refactoring data models, query optimization and implemented Redis cache to support Snowflake. • Involved in developing automated workflows for daily incremental loads, moved data from RDBMS to Data Lake. • Monitored Spark cluster using Log Analytics and Ambari Web UI. Transitioned log storage from MS SQL to CosmosDB and improved the query performance.

Hyderabad, Telangana, India

• Orchestrated the setup and configuration of Azure Databricks workspaces, clusters, and jobs, ensuring optimized performance and cost management.• Developed robust ETL pipelines using Databricks to process, clean, and transfer multi-terabyte datasets to and from Azure Data Lake Storage, improving data processing speeds by 50%.• Leveraged Delta Lake's capabilities to build a high-performance, reliable, and simplified data lake, ensuring ACID compliance and efficient upsert operations.• Implemented structured streaming in Databricks for real-time analytics, enabling actionable insights from live data feeds.• Conducted performance tuning of Spark jobs in Databricks, utilizing best practices like broadcasting, caching, and partitioning, achieving a 15% reduction in processing times.• Used Databricks' integrated monitoring tools and integrated with Azure Monitor and Log Analytics to track job performance, failures, and resource utilization.• Implemented auto-scaling and termination policies for Databricks clusters, optimizing resource usage and reducing Azure costs by 20%.• Transformed raw data into a more efficient and optimized data format such as Avro and Parquet to speed up data processing and improve data transfer through networks. • Conducted data analysis, auditing, forecasting, programming, research, report generation, and software integration to gain an in-depth understanding of the current BI platform architecture and support the implemented solution. • Designed, implemented, and tested various performance tests to ensure the successful delivery of a project. Utilized data-driven performance analytics to produce reports and dashboards with valuable insights. • Designed, developed, and tested ETL applications with different sources.• Constructed and optimized SQL queries in Hive through HUE. Executed MapReduce jobs in Hive by querying available data.

Hyderabad, Telangana, India

• Responsible for identifying, gathering, exploring, and cleansing data for modeling.• Created and exhibited models to fund managers that resulted in 20% improved returns compared to prior performance and 25% improved stock price predictions over traditional methods. • Collaborated with data compliance and governance teams to maintain data models, metadata, and dictionaries while defining source fields and their definitions. • Utilized HIVE to execute transformations, event joins, and some pre-aggregations before storing the data in HDFS.• Authored SQL and PL/SQL scripts to extract data from databases and for testing purposes.• Designed a Spark batch job to automate the creation/update of external Hive tables from datasets stored in HDFS.• Worked with HDFS file formats such as Avro, Sequence File, and various compression formats such as Snappy. • Created Spark SQL scripts to optimize query performance. • Implemented Spark scripts using Spark Session, Python, and Spark SQL to access Hive tables for more efficient data processing. • Partnered with data engineers and operation team to implement ETL processes, Snowflake models, and optimized SQL queries to extract data to meet analytical requirements. • Applied Dimensional Data Modeling to deliver Multi-Dimensional STAR schemas and developed Snowflake schemas by normalizing dimension tables as needed.• Revamped ETL procedures, added data management and data query components, and updated existing ones. • Employed Redshift and S3 within AWS along with Informatica Cloud Services to load data into an S3 bucket. • Conducted data analysis and visualizations using Python and Tableau.

Master'S Degree, Management Science, 3.8

Bachelor'S Degree, Electronics And Instrumentation Engineering, 74