Anu P. Email and Phone Number

.Evaluating the current state of an organization's data systems, pinpointing areas for enhancement, and incorporating considerations related to Hadoop, ELK, CI/CD, and SIEM..Identifying and assessing innovative technologies and solutions for the modernization of data systems, integrating insights from Hadoop, ELK, CI/CD, and SIEM perspectives..Crafting a comprehensive data modernization strategy that aligns seamlessly with the overarching business goals, with a specific focus on Hadoop, ELK, CI/CD, and SIEM implementations..Executing seamless data migration from legacy systems to contemporary platforms, emphasizing data integrity and security throughout, with attention to Hadoop, ELK, CI/CD, and SIEM implications..Formulating and implementing robust data storage solutions on GCP, including but not limited to BigQuery, Cloud SQL, and Cloud Spanner, while considering integration with Hadoop and ELK ecosystems..Integrating data from diverse sources, encompassing on-premises systems and other cloud platforms, leveraging technologies such as Cloud Dataflow, Cloud Dataproc, and Cloud Data Fusion. .Orchestrating the creation and management of data pipelines for both real-time and batch processing, utilizing technologies like Cloud Dataflow, Cloud Composer, and Cloud Data Fusion, with seamless integration with Hadoop and ELK components..Ensuring the governance of data access, employing Cloud Identity and Access Management (IAM) and Cloud Key Management Service (KMS), while also considering security aspects related to Hadoop and ELK..Collaborating effectively with cross-functional teams, including data engineers, data scientists, and application developers, to guarantee proper utilization, integration, and security of data, while incorporating Hadoop, ELK, CI/CD, and SIEM principles..Vigilantly monitoring and optimizing the performance and cost efficiency of data storage and processing solutions, incorporating insights from Hadoop, ELK, CI/CD, and SIEM perspectives.

Orchestrated the migration initiative, transitioning from On-prem (Hadoop) and Teradata (IDW) to the GCP Data Engineering platform, integrating GCS, Dataproc, Datafusion, BigQuery, and Dataflow. This migration resulted in a substantial annual cost reduction in hardware infrastructure maintenance, along with a notable 25% improvement in performance.Provided valuable technical mentorship to junior team members, conducted thorough code and design reviews, and enforced coding standards and best practices, incorporating considerations related to Hadoop, ELK, CI/CD, and SIEM.Established a robust Cloud Warehouse in GCP's BigQuery, redirecting legacy data sources to BigQuery by reengineering existing ETL processes using PySpark/Talend, ensuring seamless integration with Hadoop and ELK ecosystems.Architected and implemented versatile frameworks for processing transactional data (30 million records/day) from SQLServer to the Cloud, utilizing Qlik Replicate and PySpark,Automated the ETL process across billion-row datasets, resulting in a 30% reduction in manual workload, with a focus on optimizing processes related to Hadoop, ELK, CI/CD, and SIEM.Ingested data from diverse sources, including SQLServer, Oracle, and Salesforce API, using Python to create data views for BI tools like Tableau/PowerBI, while ensuring compatibility with Hadoop.Leveraged Spark in Python to distribute data processing on large datasets, enhancing the speed of ingestion and migration, with a keen eye on Hadoop and ELK integration.Designed and executed migration pipelines from Hadoop to GCS/BigQuery, employing DISTCP utility and PySpark features for BigQuery-compatible conversion, while considering implications for Hadoop and ELK ecosystems.Implemented DevOps deployment methods for bulk DDL deployment and version management for BigQuery metadata, utilizing Terraform for maintaining object states, and integrating CI/CD principles, including considerations for Hadoop, ELK, CI/CD, and SIEM.

Our key objective is to develop and implement reusable frameworks, with a dedicated emphasis on Disaster Recovery and efficient Data Ingestion into Hive from diverse sources such as Mainframe systems, Teradata databases, and NAS files. This includes crafting frameworks and joblets such as File Capture, Snapshot Ingestion, SCD Ingestion, Load Status Joblet, DQ Check Joblet, ELI Joblet, Mainframe Files Ingestion, Mainframe Header Validation Joblet, and more.The MARS team consistently builds reusable frameworks and joblets tailored for constructing a Data Lake within the Big Data Hub. Additionally, we extend our expertise to Migelop various reusable frameworks on Talend Real-Time Big Data for both internal stakeholders and other client vendors. Our responsibilities encompass end-to-end integration and validation of these frameworks, incorporating considerations related to Hadoop, ELK, CI/CD, and SIEM principles.Our collaborative efforts focus on seamless integration with Hadoop ecosystems, implementing ELK solutions for effective log management and analysis, adhering to CI/CD practices for streamlined development and deployment processes, and ensuring the incorporation of SIEM principles for robust security monitoring. The MARS team consistently works towards enhancing the efficiency and security of data processing through a holistic approach that considers Hadoop, ELK, CI/CD, and SIEM aspects.