Vishal .. Email & Phone Number

Jacksonville, Florida, United States

*Designed and implemented an Enterprise Data Lake on Snowflake, utilizing Snowpipe for continuous streaming data ingestion.*Developed end-to-end architectures for data ingestion pipelines using AWS services and Databricks with PySpark for transformation and storage. *Azure DevOps for continuous integration and continuous deployment (CI/CD*Configured secure AWS environments and implemented cloud policies while enriching data pipelines with Azure Databricks tools for efficient management.*Optimized data processing frameworks, achieving significant runtime reduction and cost savings.*Integrated Kafka with Spark Streaming for real-time processing and ensured data quality through PySpark and Scala checks.

Minneapolis, Minnesota, United States

*Architected scalable, automated data pipelines for structured/unstructured data on AWS, leveraging Spark DataFrames for efficient processing.*Championed serverless data processing with AWS Lambda and implemented GDPR rules for data encryption.*Designed and managed data engineering workflows using Databricks tools for orchestration and automation.*Built real-time stream processing pipelines integrated with AWS Kinesis and monitored clusters using CloudWatch for proactive troubleshooting.*Conducted data cleaning, pre-processing, and modeling using Spark and Python on AWS.

*Integrated on-prem and cloud data using Azure Data Factory, Azure Synapse, and Databricks, creating data pipelines and migrating large datasets.*Developed Databricks notebooks for data curation and mounted blob storage for streamlined business use cases.*Managed Azure data engineering workflows, including transformations and integration runtimes, while ensuring data security with Azure Key Vaults.*Led comprehensive migration strategy from on-premises Hortonworks clusters to Azure, implementing Azure services for optimized cloud-based processing.*Provided post-migration support and conducted performance monitoring and optimization in the Azure environment.

India

*Developed and maintained data ingestion pipelines using Apache NiFi and Sqoop to bring in data from various sources into HDFS.*Implemented data processing workflows with Spark, MapReduce, and Hive for transformations and analytics, while ensuring data security through encryption.*Conducted performance tuning activities to optimize Hadoop job performance and addressed performance bottlenecks.*Collaborated with business stakeholders to understand data requirements and provided technical guidance and support.*Created comprehensive documentation and conducted knowledge-sharing sessions to enhance the skillset of the IT team and end-users.