• Working in Cyber Security team, developed Spark applications using PySpark and Spark-SQL, leading to efficient data extraction, transformation, and integration into SQL databases. • Developed and maintained a robust ETL framework using Spark, incorporating daily runs, error handling, and logging, significantly improving data processing efficiency. • Wrote neo4j queries and utilized Spark for efficient data reading from neo4j databases as a crucial part of the ETL pipeline… Show more • Working in Cyber Security team, developed Spark applications using PySpark and Spark-SQL, leading to efficient data extraction, transformation, and integration into SQL databases. • Developed and maintained a robust ETL framework using Spark, incorporating daily runs, error handling, and logging, significantly improving data processing efficiency. • Wrote neo4j queries and utilized Spark for efficient data reading from neo4j databases as a crucial part of the ETL pipeline. • Gained extensive experience using Brinqa, a vulnerability risk management tool, enhancing organizational security and risk assessment capabilities. • Composed SQL queries to generate diverse reports, providing valuable insights and data-driven decision support. • Utilized Spark SQL for efficient data reading and writing from SQL databases, optimizing data pipeline operations. • Worked with Azure Data Lake and Azure SQL, adeptly creating tables and crafting queries on Azure SQL server, leading to enhanced data management. • Created a data pipeline in PySpark, integrating Azure Data factory and SparkSQL in Azure Databricks for the effective data extractions, transformations, and aggregations with multiple file formats.• Developed Python programs for data ingestion to various Azure services (Azure Data Lake, Azure Storage, and Azure SQL), enhancing data processing and transformation in Azure Databricks.• Have Experience in designing and developing Azure stream analytics jobs to process real time data using Azure Event Hub.• Vigilantly monitored Splunk dashboards for pipeline failures and conducted log analysis to rectify issues, maintaining high pipeline reliability. • Showcased expertise in developing a range of reports and dashboards using Tableau Visualizations, enhancing data interpretation and presentation, and created detailed Tableau bar graphs and scatter plots, providing comprehensive summary reports and dashboards for better data insights. Show less

• Developed ETL pipelines in and out of the data warehouse, creating advanced SQL reports that improved data accessibility and reporting accuracy. • Utilized HDFS's scalable architecture to efficiently store and manage large volumes of data, seamlessly handling the growth of data generated from diverse digital marketing.• ⁠Leveraged HDFS's built-in fault tolerance mechanisms, such as data replication and data block checksums, to ensure data reliability and availability even in the event… Show more • Developed ETL pipelines in and out of the data warehouse, creating advanced SQL reports that improved data accessibility and reporting accuracy. • Utilized HDFS's scalable architecture to efficiently store and manage large volumes of data, seamlessly handling the growth of data generated from diverse digital marketing.• ⁠Leveraged HDFS's built-in fault tolerance mechanisms, such as data replication and data block checksums, to ensure data reliability and availability even in the event of hardware failures.• Successfully managed S3 buckets creation, implementing robust policies on IAM roles, enhancing data security and access control. • Developed PySpark code for AWS Glue jobs and EMR, optimizing data processing and integration across AWS services. • Designed and developed an ETL process in AWS Glue using PySpark, effectively migrating data from sources like S3, flat files, RDBMS into AWS Redshift, and enabled efficient reporting using Athena. • Showcased professional knowledge in AWS Redshift by designing ETL jobs that enhanced data extraction and loading efficiency. • Involved in constructing ETL pipelines using AWS Data Pipelines, leading to streamlined data transfer and processing. • Designed Tableau bar graphs, scatter plots, and geographical maps, creating detailed summary reports and dashboards that provided deeper insights into data. • Established comprehensive monitoring systems using CloudWatch for Lambda functions, Glue Jobs, and EC2 hosts, ensuring high system reliability and performance. • Built Lambda and step function serverless orchestration from S3 to Redshift and RDS services, streamlining data flow and processing. • Wrote Spark SQL jobs to enrich data (joining, aggregating) and build KPI tables, providing crucial metrics for business analysis. • Designed Lambda functions using Python for various use cases, including creating dynamic DAGs and event triggers, enhancing automation and efficiency. Show less

• Streamlined the transfer of data from servers to HDFS using Apache Sqoop, improving the efficiency and reliability of data storage processes. • Executed complex MYSQL queries from Python, leveraging Python-MySQL connector and MySQL dB package, leading to more effective data manipulation and extraction. • Configured and automated AWS services (Glue, EC2, S3) using Python's Boto3 library, enhancing cloud infrastructure efficiency and operational agility. • Developed and managed… Show more • Streamlined the transfer of data from servers to HDFS using Apache Sqoop, improving the efficiency and reliability of data storage processes. • Executed complex MYSQL queries from Python, leveraging Python-MySQL connector and MySQL dB package, leading to more effective data manipulation and extraction. • Configured and automated AWS services (Glue, EC2, S3) using Python's Boto3 library, enhancing cloud infrastructure efficiency and operational agility. • Developed and managed pipelines for seamless data migration from on-premise databases to AWS services like Redshift, RDS, and S3, ensuring secure and efficient data transfer. • Implemented PySpark code in AWS Glue jobs for robust data extraction, transformation, and loading into S3, leading to improved data processing capabilities. • Used AWS glue catalog to get the data from S3 and perform SQL operations using AWS Athena.• Developed scripts using Spark which are used to load the data from Hive to Amazon RDS(Aurora) at a faster rate.• Experience with Apache Airflow, Python scripting for scheduling tasks and process automation. • Designed and developed insightful Power BI reports and dashboards, providing dynamic data visualization and enhanced reporting capabilities for better decision-making. Show less