Pavankumar M Email & Phone Number

Bengaluru, Karnataka, India

Hyderabad, Telangana, India

Hands-on development and implementation experience in Big Data Management Platform (BMP) using Databricks, S3, DynamoDB, HDFS, Hive, pyspark, Apache airflow and HUE.Experience in Data transfer from structured data stores to HDFS using pyspark JDBC connection.Developed Batch Processing jobs using pyspark, SparkSQL, and Hive.Depth of working experience.

Hyderabad, Telangana, India

Implemented the python and pyspark scripts to parse the nested xml’s and convert them into Json format.Developed spark coding using python scripting to analyse the data we are getting from different sources. Created data frames in a Spark SQL from data in HDFS and did transformations analyse the data and stores back into the HDFS and used the msck.

Bengaluru, Karnataka, India

Created Hive External tables with partitioning and bucketing logic. Tracking the corrupted records for each table and updating those corrupted records separately in error table. Experience in dealing JSON, CSV, TSV and pipe delimiter files. Experience in loading data from landing area to RAW layer (HDFS) and from Raw to Stage and Conformed layers.

Bengaluru, Karnataka, India

As per the project requirements we are fetching the data from Dynamic 365 to s3 pre-landing layer using databricks auto loader.Loading the prelanding delta tables into landing by partitioning on year, month, day, hour and minutes after checking the datatype check, source and target count check, Then applying the incremental merge in the full delta table.