Pradeep T Email & Phone Number

Overland Park, Ks, Us

• Developed Spark Applications by using Python and Implemented Apache Spark data processing Project to handle data from various RDBMS and Streaming sources. • Responsible for building scalable distributed data solutions using Hadoop• Extensive experience in working with AWS Cloud Platforms (EC2, S3, EMR, Redshift, Lambda and Glue).• Built NiFi dataflow to consume data from Kafka, make transformations on data, place in HDFS and exposed port to run spark streaming job.• Migrated an existing on-premises application to the AWS platform• Experienced in Maintaining the Hadoop cluster on AWS EMR • Worked with Spark for improving performance and optimization of the existing algorithms in Hadoop.• Work on Spark RDD, Data Frame API, Data set API, Data Source API, Spark SQL, and Spark Streaming.• Used Spark Streaming APIs to perform transformations and actions on the fly for building common.• Developed Kafka consumer API in python for consuming data from Kafka topics. • Consumed Extensible Markup Language (XML) messages using Kafka and processed the XML file using Spark Streaming to capture User Interface (UI) updates.• Created Spark Streaming jobs using Python to read messages from Kafka & download JSON files from AWS S3 buckets.

Woonsocket, Ri, Us

• Worked with Hadoop Ecosystems components like HBase, Sqoop, Zookeeper, Oozie, Hive, and Pig with Cloudera Hadoop distribution. • Involved in Agile development methodology active member in scrum meetings. • Worked in AWS environment for development and deployment of Custom Hadoop Applications. • Designed and implemented scalable Cloud Data and Analytical architecture solutions for various public and private cloud platforms using AWS. • Developed ADF Pipelines to load data from on-premises to AWS S3 and databases.• Involved in start to end process of Hadoop task that used various technologies such as Sqoop, PIG, Hive, MapReduce, Spark, and Shells scripts.• Extracted and loaded data into the Data Lake environment by using Sqoop which was accessed by business users.• Developed PIG scripts to transform the raw data into intelligent data as specified by business users. • Utilized Apache Spark with Python to develop and execute Big Data Analytics.• Installed Hadoop, Map Reduce, and HDFS to develop multiple MapReduce jobs in PIG and Hive for data cleansing and pre-processing. • Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive. • Improved the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDDs, and Spark YARN.

Pune, Maharashtra, In

• Developed real-time data ingestion/analysis using Kafka / Spark-streaming.• Created Hive External tables and loaded the data into tables and query data using HQL. • Worked on ingesting data from RDBMS sources like - Oracle, SQL Server, and Teradata into HDFS using Sqoop.• Imported the data from various formats like JSON, Sequential, Text, CSV, AVRO, and Parquet to HDFS cluster with compressed for optimization. • Used AWS S3 to store a large amount of data in an identical/similar repository. • Loaded all datasets into Hive from Source CSV files using Spark and Cassandra from Source CSV files using Spark• Created environment to access Loaded Data via Spark SQL, through JDBC & ODBC (via Spark Thrift Server).• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying of the log data.

Hyderabad, Telangana, In

• Responsible for gathering requirements from business analyst/operational analyst and identifying the data sources required for the request• Created value from data and drive data-driven decisions by performing advanced analytics and statistical techniques to determine to deepen insights, optimal solution architecture, efficiency, maintainability, and scalability which make predictions and generate recommendations.• Enhancing data collection procedures to include information that is relevant for building analytic systems.• Worked closely with a data architect to review all the conceptual, logical, and physical database• Design models concerning functions, definition, maintenance review, and support data Analysis, Data quality, and ETL design that feeds the logical data models.