Sai P Email & Phone Number

Austin, Tx, Us

United States

• Designing and architecting multiple data pipelines, that includes stages like Data Ingestion, Data Transformation and Data Storage.• Develop ETL processes to move data through the pipeline efficiently.• Implement orchestration tools like Apache Airflow and Apache NiFi to manage the flow and dependencies of tasks.• Worked on AWS Data Pipeline to configure data loads from S3 into Redshift.• Responsible for creating on-demand tables on S3 files using Lambda Functions & Python.• Subscribed to the Kafka topic with Kafka consumer client and processed events in real-time using Spark.• Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performed necessary Transformations and Aggregation on the fly to build the common learner data model and persisted the data in HDFS.• As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to Snowflake.• Highly implemented a generic ETL framework with high availability for bringing related data for Hadoop & Cassandra from various sources using Spark.• Utilized Python in Spark to extract data from Snowflake and upload it to Salesforce daily.• Designed Kafka producer client using Confluent Kafka and produced events into Kafka topic.• Used Spark SQL to load JSON data and create Schema RDD, then loaded it into Hive Tables and handled structured data using Spark SQL.• Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data.• Developing Spark programs with Python and applied principles of functional programming to process the complex structured data sets.• Working on DBT (Data Build Tools) to architect and implement data transformations, ensuring efficient and scalable analytics pipelines.

San Diego, California, United States

• Migrate data into RV Data Pipeline using Databricks, Spark SQL and Scala.• Used Databricks for encrypting data using server-side encryption.• Worked with the Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, PySpark, Impala, Tealeaf, Pair RDD's, DevOps, Spark YARN• Create and maintain optimal data pipeline architecture in cloud Microsoft Azure using Data Factory and Azure Databricks• Worked with PowerShell and UNIX scripts for file transfer, emailing and other file related tasks.• Implement IOT streaming with Databricks Delta tables and Delta lake to enable ACID transaction logging• Exposed transformed data in Azure Spark Databricks platform to parquet formats for efficient data storage• Writing PySpark and spark Sql transformation in Azure Databricks to perform complex transformations for business rule implementation• Developed various Oracle SQL scripts, PL/SQL packages, procedures, functions, and java code for data.

Houston, Texas, United States

• Worked on SQL Server concepts SSIS (SQL Server Integration Services), SSAS (Analysis Services) and SSRS (Reporting Services). • Using SSIS, SPSS, SAS to extract transform & load source data from transaction systems.• Used AWS Data Pipeline to schedule an Amazon EMR cluster to clean and process web server logs stored in Amazon S3 bucket.• Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency which is important for decision making in the process.• Worked on different data formats such as JSON, XML and performed machine learning algorithms in Python.• Application development using Hadoop Ecosystems such as Spark, Kafka, HDFS, HIVE, Oozie and Sqoop.• Involved in support for Amazon AWS and RDS to host static/media files and the database into Amazon Cloud.

Bengaluru, Karnataka, India

• Used Hive to implement data warehouse and stored data into HDFS. Stored data into Hadoop clusters which are set up in AWS EMR. • Performed pig script which picks the data from one Hdfs path and performs aggregation and loads into another path which later pulls populates into another domain table. Converted this script into a jar and passed as parameter in Oozie script• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.• Performed Data Preparation by using Pig Latin to get the right data format needed. • Utilized the clinical data to generate features to describe the different illnesses by using LDA Topic Modelling. • Performed Data Visualization and Designed Dashboards with Tableau and generated complex reports including chars, summaries, and graphs to interpret the findings to the team and stakeholders. • Used Hive and created Hive tables and involved in data loading and writing Hive UDFs.• Used Sqoop to import data into HDFS and Hive from other data systems.• Used Git for version control with Data Engineer team and Data Scientists colleagues.

Hyderabad, Telangana, India

• Involved in creating Hive tables, loading with data and writing hive queries on top of data present in HDFS.• Loaded data from MySQL server to the Hadoop clusters using the data ingestion tool Sqoop.• Worked on tuning the performance Pig queries. Involved in Developing the Pig scripts for processing data.• Written Hive queries to transform the data into tabular format and process the results using Hive Query Language.• By using Apache Flume loaded real time unstructured data like xml data, log files into HDFS.• Processed large amount both structured and unstructured data using MapReduce framework.• Designed solution to perform ETL tasks like data acquisition, data transformation, data cleaning and efficient data storage on HDFS• Developed Spark code using Scala and Spark Streaming for faster testing and processing of data.• Store the resultant processed data back into Hadoop Distributed File System.