In my role as Senior Data Engineer at CVS, I'm deeply involved in various aspects of software solution development, from conception to delivery. My key contributions include:• Participating in requirement grooming meetings to understand business needs and providing estimates to translate these into actionable software solutions. This involves extensive use of Agile development methodologies and JIRA for project management.• Managing data integrity and system functionality across… Show more In my role as Senior Data Engineer at CVS, I'm deeply involved in various aspects of software solution development, from conception to delivery. My key contributions include:• Participating in requirement grooming meetings to understand business needs and providing estimates to translate these into actionable software solutions. This involves extensive use of Agile development methodologies and JIRA for project management.• Managing data integrity and system functionality across multiple platforms, including checking data within DynamoDB tables and ensuring EC2 instances are operational across development, quality assurance, certification, and production environments in AWS.• Developing and deploying Spark jobs in various environments, handling data loading to NoSQL databases such as Cassandra, Hive, and HDFS, and securing data with encryption protocols.Implementing comprehensive solutions on AWS, including configurations of EC2, S3, RDS, EBS, and Elastic Load Balancers, and utilizing Cloud Watch for monitoring and notifications.• Extending expertise to Google Cloud Platform services, enhancing infrastructure with Compute Engine, Cloud Functions, and Cloud Storage, among others.• Writing code in Apache Spark and Scala, and using tools like IntelliJ and Jenkins for continuous integration and deployment, alongside Docker and Kubernetes for container orchestration.Utilizing Kafka for streaming real-time data and Kibana for log monitoring to ensure optimal performance.• Scheduling and managing Informatica jobs through the Autosys tool, and extensively working on power center mappings, sessions, and workflows to enhance data flow and integration.• Leveraging SQL tools like TOAD and SQL Developer to run queries and validate data, enhancing system reliability and performance.Technologies: Apache Spark, Scala, Cassandra, AWS, GCP, Kubernetes, Jenkins, Kafka, Informatica PowerCenter, SQL Server, Salesforce, UNIX Show less

In my position as a Data Engineer, I was instrumental in building scalable distributed data solutions and driving significant technological migrations. My key contributions included:• Participating in Agile development processes such as Scrum and sprint planning, ensuring efficient workflow and project alignment with business goals.• Leading the migration of on-premise environments to Google Cloud Platform (GCP), which involved setting up and managing Hadoop clusters in a Windows… Show more In my position as a Data Engineer, I was instrumental in building scalable distributed data solutions and driving significant technological migrations. My key contributions included:• Participating in Agile development processes such as Scrum and sprint planning, ensuring efficient workflow and project alignment with business goals.• Leading the migration of on-premise environments to Google Cloud Platform (GCP), which involved setting up and managing Hadoop clusters in a Windows environment and transitioning data warehouses to the Snowflake Data Warehouse.• Handling the complex process of migrating existing on-premise Hive code and Oracle SQL ETLs to GCP, using BigQuery and Cloud Dataproc. This also included setting up Apache Airflow jobs triggered via Cloud Pub/Sub for streamlined data processing.• Extracting and analyzing data from data lakes and enterprise data warehouses to relational databases, utilizing SQL Queries and PySpark for advanced data analysis and insight generation.• Developing PySpark scripts to merge and cleanse data, enhancing the quality and usability of information across the organization.• Creating and managing workflows using Apache Airflow to automate services for Change Data Capture, and using Kafka and Spark streaming for real-time data ingestion into HDFS.• Building and maintaining robust data integration programs within a mixed environment of Hadoop and traditional RDBMS, significantly improving data accessibility and system performance.• Writing Sqoop scripts to facilitate the efficient transfer of data between RDBMS and HDFS, and setting up a Data Lake in Google Cloud using Google Cloud Storage, BigQuery, and BigTable.Technologies: Hadoop, GCP, BigQuery, BigTable, Spark, PySpark, Sqoop, ETL, HDFS, Snowflake Data Warehouse, Oracle SQL, MapReduce, Kafka, Apache Airflow Show less

At Servion Global Solutions, I leveraged extensive AWS cloud expertise and advanced data processing technologies to enhance data solutions significantly. My role encompassed a broad range of responsibilities:• Extensively worked with AWS cloud platform services including EC2, S3, EMR, Redshift, Lambda, and Glue, enhancing system capabilities and migrating an on-premises application to the cloud.• Developed Spark applications using Python, improving performance and optimization of… Show more At Servion Global Solutions, I leveraged extensive AWS cloud expertise and advanced data processing technologies to enhance data solutions significantly. My role encompassed a broad range of responsibilities:• Extensively worked with AWS cloud platform services including EC2, S3, EMR, Redshift, Lambda, and Glue, enhancing system capabilities and migrating an on-premises application to the cloud.• Developed Spark applications using Python, improving performance and optimization of existing Hadoop algorithms, and managed data integration from various RDBMS and streaming sources.• Utilized Spark Streaming and Kafka to develop real-time data processing systems. This included creating Kafka consumer APIs in Python to manage XML and JSON data, which streamlined data flow and improved user interface updates.• Implemented data ingestion and transformation pipelines using AWS Glue, PySpark, and Elasticsearch to manage data in S3 buckets, load it into Hive external tables, and stage it in Snowflake, ensuring efficient data handling and storage.• Consolidated data warehouses into Amazon Redshift and configured Snowpipe to enhance data transfers from S3 buckets into Snowflake tables, which significantly improved data accessibility and analysis capabilities.• Advanced data analysis and management tasks included using Hive QL on Parquet formatted tables, developing custom UDFs in Python, and utilizing Sqoop and Kafka for data loading, all within a secure environment maintained through Kerberos and monitored via Cloudera Manager.Technologies: Spark, AWS, Kafka, Cassandra, Snowflake, Hadoop, Hive, Pig, Python, PySpark, Jenkins, SQL, and more, within an Agile development framework. Show less

At Wissen Infotech, my role as a Software Engineer involved a broad range of technical duties, from database management to web development. Key aspects of my role included:• Designed and implemented interactive KPI dashboards using Excel Pivot Tables, Pivot Charts, Slicer, and Timeline, extracting data from multiple databases via ODBC connections.• Developed and optimized MySQL scripts for creating tables, sequences, triggers, views, and materialized views, enhancing data retrieval… Show more At Wissen Infotech, my role as a Software Engineer involved a broad range of technical duties, from database management to web development. Key aspects of my role included:• Designed and implemented interactive KPI dashboards using Excel Pivot Tables, Pivot Charts, Slicer, and Timeline, extracting data from multiple databases via ODBC connections.• Developed and optimized MySQL scripts for creating tables, sequences, triggers, views, and materialized views, enhancing data retrieval and storage processes.• Authored complex MySQL stored procedures and triggers to monitor operational metrics from sensor data across hundreds of sites, improving data integrity and operational efficiency.• Managed data collection, cleaning, and manipulation processes, compiling data from various sources into Excel and uploading it to database servers for centralized access.• Supported the design of Entity Relationship Diagrams (ERD) and flowcharts for supplier databases, and implemented 3NF normalization to eliminate data redundancy.• Analyzed data sets for anomalies and trends, producing detailed reports and visualizations in Excel to assist management in strategic decision-making.• Digitalized and optimized internal processes by developing web-based training materials, forms, and workflows, significantly reducing manual efforts and enhancing process efficiency.• Created and maintained internal web pages detailing processes, policies, and workflows, which served as a key repository of institutional knowledge.• Developed front-end UI modules using HTML, JSP, JavaScript, and CSS, and improved webpage interactivity with JavaScript and jQuery, ensuring a user-friendly experience.• Conducted unit testing, maintenance, and bug fixing for various applications, ensuring robust software performance and reliability.Technologies Used: MySQL, MS Excel, HTML, JSP, JavaScript, CSS, jQuery, C, C++ Show less