• Optimizing Cloud Performance: Implemented solutions to optimize cloud performance for handling high-volume data, potentially using cloud services like AWS.• Collaboration with Business Process Managers: Collaborated as a subject matter expert to transform large data volumes and create analytical data products for BI reporting needs.• AWS Lambda: Created functions and assigned roles in AWS Lambda for event-driven processing with Python and Java.• Spark Development: Developed Spark applications using Scala, DataFrames, and the Spark SQL API for efficient big data processing.• ETL Integration with Spark: Designed and implemented ETL integration patterns using Python on Spark, analyzing SQL scripts and using PySpark for implementation.• Cosmos DB: Designed and implemented Cosmos DB database schemas for efficient storage and management of large-scale datasets.• Data Pipelines: Constructed data pipelines using Spark, Hive, Sqoop, and custom input adapters for ingesting, transforming, and analyzing operational data.• Spark-SQL and Hive: Utilized Spark-SQL to load JSON data, create Schema RDDs, and load data into Hive tables, possibly hosted on AWS.• Data Integration and Real-time Processing: Utilized Spark for interactive queries, streaming data processing, and integration with NoSQL databases to handle large data volumes. Integrated Cosmos DB with Azure Stream Analytics for real-time data processing.• AWS Glue: Designed and implemented ETL processes in AWS Glue to migrate data from external sources into AWS Redshift.• Elasticsearch Integration: Developed ETL parsing and analytics using Python/Spark to establish a structured data model in Elasticsearch for API and UI consumption.• IAM and Security: Utilized IAM for creating accounts, roles, groups, and policies for AWS resource management and integration with AWS services.• Snowflake: Authored complex Snow SQL scripts in the Snowflake cloud data warehouse for business analysis and reporting.

Implemented Cloud (Azure) Delta lake house platform infrastructure, including RBAC, using IAC (Terraform) following industry best practices.Collaborated with architecture and security teams to remediate infrastructure-related concerns and recommendations, specifically focusing on Azure Databricks, Azure PaaS, and Azure DevOps (VSTS).Designed and implemented end-to-end data pipelines using Azure Data Factory and Databricks, enhancing data accessibility and efficiency.Developed data models and schemas for Azure SQL Databases and Azure Cosmos DB, ensuring data consistency and query performance optimization.Constructed a scalable data warehouse using Azure Synapse Analytics, enabling efficient data analysis and reporting for business users.Implemented an automated data quality monitoring system using Azure Data Factory and Azure Functions.Executed ELT processes using Azure Data Factory and Databricks, leveraging big data technologies and distributed processing capabilities.Designed and implemented Change Data Capture (CDC) mechanisms to capture and synchronize real-time updates from source systems to Cosmos DB.Created pipelines in Azure Data Factory for data extraction, transformation, and loading from various sources, including Azure SQL and Blob storage.Developed Data Pipelines for Copy Activity in Azure Data Factory, enabling data movement and transformation for on-cloud ETL processing.Crafted Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from various file formats, uncovering insights into customer usage patterns.Conducted knowledge-sharing sessions and created comprehensive documentation about Cosmos DB design, data models, and deployment procedures, facilitating team skill development and seamless knowledge transfer.

• Spark Application Development: Developed Spark applications in Scala for data cleansing, validation, transformation, and summarization, showcasing expertise in big data processing.• Data Pipeline Construction: Created data pipelines using Spark, Hive, and Sqoop for ingesting, transforming, and analyzing operational data.• Real-time Data Processing: Streamed real-time data using Spark and Kafka for handling web server console log data, demonstrating real-time data processing skills.• Cloud Platform Expertise: Provided Infrastructure as a Service (IaaS) solutions on the Microsoft Azure cloud platform, highlighting cloud infrastructure knowledge.• SQL and Database Development: Designed database objects using T-SQL and developed processes for incremental data imports, showing database development skills.• Agile Collaboration: Collaborated within an Agile environment, utilizing tools like Jira and GitHub version control for continuous builds, emphasizing teamwork and agile methodologies.

• Data Extraction and Transformation: Extracted data from Oracle, DB2, and Teradata sources, performed transformations, and exported data as per business needs, demonstrating expertise in ETL processes.• Hive and MapReduce Processing: Imported data from diverse sources, applied Hive and MapReduce transformations, loaded data into HDFS, and used Sqoop for extracting from relational databases.• Hive Development: Created Hive tables, loaded data with HQL scripts, developed custom Java-based Hive UDFs, and designed Hive external tables with dynamic partitioning and buckets.• Data Processing Frameworks: Utilized Apache Spark with Scala and Python, integrated Kafka for real-time data processing, and employed NoSQL databases like HBase for semi-structured data storage.• Performance Optimization: Optimized performance with distributed caching, Hive partitioning, bucketing, and map-side joins, showcasing efficiency in big data processing.• Scheduling and Workflow Management: Scheduled data ingestions and transformations using Oozie and Maestro, highlighting workflow management skills.