Parth Patel Email & Phone Number

Toronto, Ontario, Canada

• Developed data processing ETL pipelines using PySpark, involving data reading from external sources, merging, data enrichment, and loading into Azure Sql data warehouse.
• Developed PySpark scripts to process streaming data from data lakes using Spark Streaming, enabling real-time data processing capabilities, and performed data transformations, cleaning, and filtering with Spark Data.
• Leveraged Hive Meta data store backup, partitioning, and bucketing techniques to optimize Spark job performance and tuning.
• Possess a strong understanding of Spark Architecture, including Spark Core, Spark SQL, Data Frames, Spark Streaming, and related components such as Driver Node, Worker Node, Stages, Executors, and Tasks.
• Handled complex Hive queries, conducting table joins for extracting meaningful insights related to Spark jobs.
• Converted Hive SQL queries into Spark transformations using Spark Data Frames and Python, analyzing SQL scripts to design efficient PySpark solutions. Conducted performance tuning of Spark Applications, optimizing.

Toronto, Ontario, Canada

• Developed PySpark applications for encrypting raw data using hashing algorithms on client-specified columns, ensuring data security and privacy.
• Utilized Spark SQL API in PySpark for data extraction, loading, and performing SQL queries, transforming existing Hive-QL queries into Python-based transformations with optimized techniques.
• Responsible for creating on-demand tables on S3 files using Lambda Functions and AWS Glue with Python and Spark, streamlining data storage and access.
• Leveraged Spark-Streaming APIs to perform necessary data transformations and actions on Kafka-sourced data, persisting the results into Aws Data Lake.
• Developed Spark applications capable of handling data from RDBMS (MySQL, Oracle Database) and streaming sources, facilitating efficient data processing.
• Led the development of a data pipeline with AWS to extract data from weblogs and store it in Amazon EMR, ensuring seamless data integration.

India

• Imported Legacy data from SQL Server and Teradata into Amazon S3.
• Created consumption views on top of metrics to reduce the running time for complex queries.
• Exported Data into Snowflake by creating Staging Tables to load Data of different files from Amazon S3.
• As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to snowflake.
• Developed SQL scripts to Upload, Retrieve, Manipulate and handle sensitive data (National Provider Identifier Data I.e. Name, Address, SSN, Phone No) in Teradata, SQL Server Management Studio and Snowflake Databases.
• Implemented Defect Tracking process using JIRA tool by assigning bugs to Development Team

India

• Involved in the entire lifecycle of the projects including Design, Development, and Deployment, Testing and Implementation and support.
• Developed web-based applications using PHP, XML, JSON, and MVC3, utilizing Python scripts for database updates and file manipulation, enhancing application functionality.
• Built database models, APIs, and views with Python to create interactive solutions, contributing to seamless user experiences.
• Proficient in designing and developing the presentation layer of web applications, leveraging technologies such as HTML, CSS, JavaScript, JQuery, AJAX, and Bootstrap, ensuring visually appealing and responsive.
• Developed XML schema documents and implemented frameworks for efficient parsing of XML documents, streamlining data processing.
• Utilized Python for XML and JSON processing, facilitating data exchange and seamless business logic implementation.