As an AWS Data Engineer at Amazon, I leverage AWS Glue to create scalable ETL processes that ensure efficient data integration across various systems. Utilizing the Hadoop ecosystem, including HDFS and MapReduce, I manage both structured and unstructured data for comprehensive analysis. I optimize data workflows on Databricks with Spark for high-speed processing and use Talend for seamless data integration while maintaining quality. My work involves engineering data pipelines on Impala and Hive for quick SQL queries, implementing workflows with Apache Airflow, and automating deployment processes using Jenkins for CI/CD. I orchestrate data flows with Nifi, develop data processing applications in Scala, and utilize NoSQL databases like MongoDB and Cassandra for flexible data modeling. Additionally, I integrate Hibernate and Spring frameworks, manage workflows with Oozie, and utilize various AWS services for resource optimization. My contributions have led to a successful migration of data processes to AWS Redshift, resulting in a 15% reduction in infrastructure costs.

I developed comprehensive data models and led the creation of end-to-end data solutions on Azure, including storage, integration, processing, and visualization. Using Azure Data Factory, I facilitated the migration of data from on-premises systems to the cloud, working extensively with both structured and unstructured datasets. I automated recurring reports with Python, ensuring data integrity through constraints and DML triggers, and performed data cleansing, profiling, and security tasks to enhance data quality and reliability. Additionally, I implemented data pipelines for Snowflake and PostgreSQL, optimized existing data models, and designed solutions to reduce manual efforts. I collaborated closely with the Enterprise Data Warehouse and BI Architecture teams, supporting business needs by mapping data sources and transforming data in alignment with OLTP and OLAP requirements. I also managed Power BI gateways, handled tokenization with PySpark, and developed security policies for BI reporting.

As a GCP Data Engineer for Hanover Insurance, I designed and developed cloud-based batch and streaming data pipelines using Google Cloud Platform (GCP) services, including BigQuery, Cloud Storage, Compute Engine, and Cloud Composer. I played a key role in migrating on-premises solutions to the cloud, establishing a Data Lake in Cloud Storage and a data warehouse in BigQuery. I converted Hive/SQL queries into Spark transformations with Spark RDD, Scala, and Python, collaborating with product teams, data analysts, and data scientists to build scalable data solutions. I also automated and orchestrated workflows using Airflow and Google Composer, creating end-to-end ETL pipelines with Snowflake and Redshift for data normalization. Working in an Agile environment, I contributed to sprint cycles, daily scrums, and design meetings, ensuring the success of the project from design to post-implementation support. Additionally, I implemented workflows, triggers, PySpark jobs, and Lambda functions to optimize data processing and reporting.

As a Hadoop Developer for Frontier Communications Corporation, I focused on data ingestion and processing, utilizing tools like Sqoop, Spark, and Hive to load data from various sources, including RDBMS, CSV, and XML files. I performed data cleansing and transformation tasks with Spark using Scala, ensuring the data was consolidated and formatted for auditing and reporting purposes. My responsibilities included designing and developing Spark SQL scripts based on functional specifications, optimizing existing algorithms for performance, and managing job scheduling with the Fair Scheduler and Oozie Workflow. I created Hive tables and executed queries that triggered MapReduce jobs, while also working with NoSQL databases like HBase to handle large sets of semi-structured data. I implemented automated workflows using Apache Oozie, optimized Spark applications for batch processing, and analyzed large datasets to enhance data aggregation and reporting capabilities across various file formats, including Text, Avro, ORC, and Parquet.

As a Data Consultant for Rivian Automotive, I focused on database design, data modeling, and T-SQL development while implementing ETL packages using SSIS. I contributed to the creation of data standards and policies to ensure long-term data quality and analyzed user requirements during JAD meetings to develop OLAP applications. I maintained data integrity by establishing constraints and DML triggers, and wrote Python scripts for manipulating user-defined objects. Utilizing Agile methodologies, I adapted to changing client requirements while developing complex ETL mappings for data migration. My responsibilities included creating SQL scripts for reporting, maintaining SSIS packages for data extraction and transformation, and implementing error handling techniques. Additionally, I managed dynamic data masking in Azure SQL Database and Azure Synapse Analytics, and designed periodic reports using SSRS to meet business needs. My work ensured efficient ETL processes and high data quality for large-scale data warehouses, enabling effective data-driven decision-making.