Susmitha K Email & Phone Number

United States

● Orchestrated and automated 1+ data pipelines using Airflow, ensuring timely data availability for downstream applications.● Use Spark SQL to perform complex data analysis on massive datasets, improving query performance compared to traditional methods.● Successfully integrated Azure Data Lake Storage and Data Lake Analytics to streamline data processing workflows, resulting in a 25% reduction in data processing times.● Execute a robust data streaming architecture using Kafka… Show more ● Orchestrated and automated 1+ data pipelines using Airflow, ensuring timely data availability for downstream applications.● Use Spark SQL to perform complex data analysis on massive datasets, improving query performance compared to traditional methods.● Successfully integrated Azure Data Lake Storage and Data Lake Analytics to streamline data processing workflows, resulting in a 25% reduction in data processing times.● Execute a robust data streaming architecture using Kafka, efficiently handling the ingestion and distribution of real-time data across diverse applications.● Implement and deploy Spark applications on Databricks clusters for data processing, achieving a 10% improvement in data processing speed.● Working on MapReduce platform team for performance tuning of the Hive and spark jobs of all users. Show less

India

● Developed efficient Snowflake queries to analyze large datasets, improving query performance by 15%.● Used Spark to accelerate data processing, achieving a 25% increase in processing speed for daily batches of up to 50GB, enhancing the team's data analysis capability.● Exploited AWS Redshift Spectrum for cost-effective processing of large datasets stored in S3, achieving a 20% reduction in data warehousing costs.● Industrialized and maintained ETL pipelines using AWS Glue… Show more ● Developed efficient Snowflake queries to analyze large datasets, improving query performance by 15%.● Used Spark to accelerate data processing, achieving a 25% increase in processing speed for daily batches of up to 50GB, enhancing the team's data analysis capability.● Exploited AWS Redshift Spectrum for cost-effective processing of large datasets stored in S3, achieving a 20% reduction in data warehousing costs.● Industrialized and maintained ETL pipelines using AWS Glue crawlers, transformers, and jobs, automating data ingestion and transformation for data lakes.● Scheduled and monitored Airflow DAGs (Directed Acyclic Graphs) ensuring timely data processing and alerting for failures. Show less

India

● Utilized Spark and Scala APIs to compare the performance of Spark with Hive and SQL, and Spark SQL to manipulate Data Frames in Scala.● Created a Power BI dashboard to visualize key business KPIs, resulting in a weekly time savings of 10 hours on manual reporting tasks.● Improved PySpark scripts to ingest data from various sources, including databases, data lakes, and streaming platforms to ensure efficient and reliable data extraction and loading processes.● Used Spark libraries… Show more ● Utilized Spark and Scala APIs to compare the performance of Spark with Hive and SQL, and Spark SQL to manipulate Data Frames in Scala.● Created a Power BI dashboard to visualize key business KPIs, resulting in a weekly time savings of 10 hours on manual reporting tasks.● Improved PySpark scripts to ingest data from various sources, including databases, data lakes, and streaming platforms to ensure efficient and reliable data extraction and loading processes.● Used Spark libraries like Spark SQL and MLlib for efficient data analysis and machine learning tasks.● Established complex HiveQL queries to extract and analyze large datasets, reducing query execution time by 10%. Show less