I work with a range of data sources, including databases, APIs, and cloud storage, ensuring seamless data integration while maintaining consistency, quality, and security. I manage Kafka topics and integrate Kafka with Snowflake for real-time data ingestion. Additionally, I design and develop data transformation workflows using dbt and orchestrate data workflows through Airflow DAGs. I focus on optimizing performance and documenting processes to uphold high standards of data integrity.

I have designed and implemented scalable data models tailored to business needs, including star and snowflake schemas. I extracted data from AWS S3, Azure Blob Storage, and APIs, integrating it into Snowflake using the copy command. I developed Snowpipe for auto-ingestion, implemented ETL/ELT pipelines, and optimized Snowflake queries with clustering keys and materialized views. Additionally, I developed and maintained stored procedures, user-defined functions, and resolved issues related to data ingestion, transformation, and querying.

I connect with my team daily during standup calls and participate in weekly client meetings to discuss progress and gather inputs. I validate destination data against source data using business logic. I’ve extracted data from CSV and SQL Server sources to staging tables dynamically using Azure Data Factory (ADF). My work involves Azure Data Factory, Azure Databricks, Azure Data Lake, and SQL Data Warehouse, where I implement incremental and full data loads, create jobs, and develop pipelines for efficient data migration and transformation using PySpark, SparkSQL, and ADF.

I extracted data from various sources, including databases and APIs, and stored both raw and processed data in Amazon S3. I designed and implemented ETL jobs using AWS Glue, creating Glue scripts in Python or SQL for data transformation. I managed the Glue Data Catalog and Crawlers to organize and catalog data, ensuring proper schema and partitioning. Additionally, I automated and optimized ETL processes for performance and cost-efficiency, while implementing data validation and security measures using IAM roles and policies.