Pavankumar M Email & Phone Number

Bengaluru, Karnataka, India

Hyderabad, Telangana, India

Hands-on development and implementation experience in Big Data Management Platform (BMP) using Databricks, S3, DynamoDB, HDFS, Hive, pyspark, Apache airflow and HUE.Experience in Data transfer from structured data stores to HDFS using pyspark JDBC connection.Developed Batch Processing jobs using pyspark, SparkSQL, and Hive.Depth of working experience on different file formats like parquet, JSON, Orc..etc.Experience in extracting and loading data to databases like Oracle, MS… Show more Hands-on development and implementation experience in Big Data Management Platform (BMP) using Databricks, S3, DynamoDB, HDFS, Hive, pyspark, Apache airflow and HUE.Experience in Data transfer from structured data stores to HDFS using pyspark JDBC connection.Developed Batch Processing jobs using pyspark, SparkSQL, and Hive.Depth of working experience on different file formats like parquet, JSON, Orc..etc.Experience in extracting and loading data to databases like Oracle, MS SQL Server, and SQL Server, SAPBW.Experience in understanding and managing Hadoop Log Files, and cloudWatch logs.Experience with AWS Analytical services like Aws Glue, Lambda, S3, Secrete Manager, Athena, and Cloud Watch.Implemented Pyspark scripts in AWS Glue and databricks.Implemented separate ETL-Framework to automate the spark jobsImplemented the code to trigger the Databricks job from Lambda on event bases.Experience Databricks cluster and also implemented the job cluster to trigger the airflow jobs concurrently. Show less

Hyderabad, Telangana, India

Implemented the python and pyspark scripts to parse the nested xml’s and convert them into Json format.Developed spark coding using python scripting to analyse the data we are getting from different sources. Created data frames in a Spark SQL from data in HDFS and did transformations analyse the data and stores back into the HDFS and used the msck repair to merge the partition files with hive tables on daily bases. Worked on broadcast variable to put the data on single variable… Show more Implemented the python and pyspark scripts to parse the nested xml’s and convert them into Json format.Developed spark coding using python scripting to analyse the data we are getting from different sources. Created data frames in a Spark SQL from data in HDFS and did transformations analyse the data and stores back into the HDFS and used the msck repair to merge the partition files with hive tables on daily bases. Worked on broadcast variable to put the data on single variable and used it for entire processDeveloped UDFs for different logics based on requirement.Closely worked with testing team to fix all the bugs and made the code more stabilized.Experience in SVN code check-in and checkouts. As per the AWS POC, worked on AWS Glue, Athena and Cloud Watch etc. Show less

Bengaluru, Karnataka, India

Created Hive External tables with partitioning and bucketing logic. Tracking the corrupted records for each table and updating those corrupted records separately in error table. Experience in dealing JSON, CSV, TSV and pipe delimiter files. Experience in loading data from landing area to RAW layer (HDFS) and from Raw to Stage and Conformed layers (HIVE). Generated hash codes for each record to remove duplicate records with in the data. Developed UDFs for different… Show more Created Hive External tables with partitioning and bucketing logic. Tracking the corrupted records for each table and updating those corrupted records separately in error table. Experience in dealing JSON, CSV, TSV and pipe delimiter files. Experience in loading data from landing area to RAW layer (HDFS) and from Raw to Stage and Conformed layers (HIVE). Generated hash codes for each record to remove duplicate records with in the data. Developed UDFs for different logics based on requirement. Experience in manual execution of DAG’s using Airflow to load data to Raw, Stage and Conform layers Show less

Bengaluru, Karnataka, India

As per the project requirements we are fetching the data from Dynamic 365 to s3 pre-landing layer using databricks auto loader.Loading the prelanding delta tables into landing by partitioning on year, month, day, hour and minutes after checking the datatype check, source and target count check, Then applying the incremental merge in the full delta table based on the DML_actions that are coming from change feeds in D365.As per the data model we are merging the tables in… Show more As per the project requirements we are fetching the data from Dynamic 365 to s3 pre-landing layer using databricks auto loader.Loading the prelanding delta tables into landing by partitioning on year, month, day, hour and minutes after checking the datatype check, source and target count check, Then applying the incremental merge in the full delta table based on the DML_actions that are coming from change feeds in D365.As per the data model we are merging the tables in unifcation layer, then moving them in snowflake external tables. Implemented the logic to write dynamo-DB tables from s3 file systemImplemented the lambda code to trigger the Databricks job cluster once the glue job has completed using cloud-watch Rules.Experience on cloud-formation stacks and templates. Show less