Sai Reddy Email & Phone Number

* Proficient in Apache Spark development using PySpark for scalable data processing and analysis.* Extensive development experience in Apache Spark using PySpark and Scala for large-scale data processing and analytics and developed and optimized Spark jobs for real-time and batch processing, enabling data-driven insights.* Extensive experience with Azure Databricks platform, leveraging its collaborative environment to build and optimize Spark workflows.* Designed and implemented Spark jobs for complex data transformations, data enrichment, and machine learning tasks.* Expertise in utilizing Azure Data Factory to orchestrate end-to-end data workflows.* Designed data pipelines to extract, transform, and load (ETL) data from diverse sources into Azure Databricks and other target destinations.* Leveraged Azure Data Factory's data movement activities to efficiently transfer data between Azure services.* Implemented data storage solutions using Azure Storage, including Blob storage and Data Lake Store.* Leveraged Databricks Delta for managing large-scale datasets, enabling ACID transactions, and optimizing data storage and querying performance.* Developed data archival strategies to optimize storage costs and ensure data availability as per business requirements.* Proficient in Git-based version control, maintaining clean and organized repositories for codebase management.* Exposure to Azure DevOps for setting up continuous integration and continuous deployment (CI/CD) pipelines.*Automated deployment processes using Azure DevOps, ensuring efficient and reliable application releases.

Kolkata, West Bengal, India

• Well versed in working with Spark RDD, Data Frame API, Spark SQL and Spark Streaming Spark Context, Spark-SQL, Data Frame, Pair RDD and Spark YARN. • Worked on building centralized Data Lake on AWS Cloud utilizing primary services like S3, EMR, Redshift and Athena.• Developed Spark Applications by using Python for data processing Project to handle data from various RDBMS and Streaming sources. • Experienced in data manipulation using python for loading and extraction as well as with python libraries such as NumPy, SciPy and Pandas for data analysis and numerical computations.• Configured & monitored the Apache Airflow dags to migrate the data from S3 bucket to Snowflake data warehouse.• Developed Lambda functions to create ad-hoc tables to add schema and structure the data in S3 and performed data validation, filtering, sorting and transformation for every data change in Dynamo DB table and load the transformed data into PostgreSQL database. • Designed and developed ETL processes in AWS Glue and python to migrate campaign data from external sources like S3, ORC/Parquet/Text files into AWS Redshift.• Developed Spark applications using SPARK-SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights.• Configured Snow pipe to pull the data from S3 buckets and stored incoming data in the Snowflakes staging area and worked with micro batching to ingest millions of files on Snowflake cloud when files arrive to staging area.• Maintained Data Marts in Data Warehouse consisting of Star Schema Schemas (Facts/Dimensions) and Snowflake Schemas (Extended Star) utilizing Type II Slowly Changing Dimensions (SCD) for History Retention.