Implemented distributed ETL solutions using AWS (S3, EC2, Glue, EMR, PySpark) for alerting and reporting enhancements.Developed CI/CD pipelines with AWS CloudFormation and GitHub Actions, reducing deployment errors and downtime.Collaborated with data science teams to operationalize ML models using SageMaker and EMR for scalable training.Designed real-time streaming data pipelines using Amazon Kinesis, Lambda, and Kafka, processing over 1M events/day.Automated data migration from… Show more Implemented distributed ETL solutions using AWS (S3, EC2, Glue, EMR, PySpark) for alerting and reporting enhancements.Developed CI/CD pipelines with AWS CloudFormation and GitHub Actions, reducing deployment errors and downtime.Collaborated with data science teams to operationalize ML models using SageMaker and EMR for scalable training.Designed real-time streaming data pipelines using Amazon Kinesis, Lambda, and Kafka, processing over 1M events/day.Automated data migration from legacy systems using AWS Data Pipeline and DMS with zero data loss.Led a team in adopting DevOps practices, using IaC, automated testing, and monitoring via AWS CloudWatch and X-Ray.Utilized Databricks for collaborative data processing, analytics, and ML model development.Developed and optimized NLP algorithms for text analysis at scale.Built real-time fraud detection models using Spark MLlib in Databricks.Processed real-time data with Amazon Kinesis Data Analytics, identifying patterns in transactional data.Automated CI/CD pipelines with GitHub Actions, Docker, and Terraform, deploying on Kubernetes.Managed infrastructure with Terraform, ensuring secure data access via AWS IAM and KMS.Designed and developed Power BI reports, established data relationships, and performed data validation. Show less

Developed data integration and analytics solutions using Hadoop (Hortonworks) and big data technologies.Participated in Agile methodologies, including daily Scrum and Sprint planning.Performed data transformations in Hive, optimizing performance with partitions and bucketing.Created Hive external tables based on MapReduce outputs, enhancing query performance.Monitored system health using Amazon CloudWatch, set alarms, and logged system events.Designed AWS cloud… Show more Developed data integration and analytics solutions using Hadoop (Hortonworks) and big data technologies.Participated in Agile methodologies, including daily Scrum and Sprint planning.Performed data transformations in Hive, optimizing performance with partitions and bucketing.Created Hive external tables based on MapReduce outputs, enhancing query performance.Monitored system health using Amazon CloudWatch, set alarms, and logged system events.Designed AWS cloud infrastructure blueprints, including EC2, S3, Route53, Load Balancers, and VPC.Developed custom UDFs in Hive and Pig tailored to business needs.Led Spark performance optimization and COE initiatives at Cisco for data simplification.Implemented Spark with Scala/Java, leveraging DataFrames and Spark SQL for efficient processing.Spearheaded multi-terabyte AWS Redshift data warehouse design, improving query performance by 50%.Built a secure data lake in S3 integrated with Lake Formation and Glue, supporting ML workloads.Developed Spark-SQL and Spark Streaming code for fast data processing and testing.Built data pipelines using Kafka, HBase, Spark, and Hive for customer behavioral data analysis.Managed metadata and migration of applications to Hive and Spark.Extracted data from Oracle and MongoDB using Sqoop, transformed it with Spark/Hive, and loaded it into HDFS.Installed and managed Hadoop ecosystem components, including Spark, Hive, and HBase.Ensured data security in AWS Data Lake with IAM policies, adhering to regulatory standards.Developed daily reports on BigQuery tables using SQL and matplotlib, including failure detection.Collaborated with stakeholders to architect and deliver Big Data analytics solutions. Show less

Evaluated business requirements and prepared specifications for program development.Loaded and transformed structured, semi-structured, and unstructured data using Hadoop/Big Data concepts.Created automated build and deployment processes, implementing continuous integration systems.Developed and optimized MapReduce programs for unstructured data and HDFS utilization.Designed Big Data analytics platforms using Hadoop, Hive, Pig, and Cloudera for customer insights.Exported… Show more Evaluated business requirements and prepared specifications for program development.Loaded and transformed structured, semi-structured, and unstructured data using Hadoop/Big Data concepts.Created automated build and deployment processes, implementing continuous integration systems.Developed and optimized MapReduce programs for unstructured data and HDFS utilization.Designed Big Data analytics platforms using Hadoop, Hive, Pig, and Cloudera for customer insights.Exported data to relational databases via Sqoop for BI team reporting.Designed and developed Azure relational databases and created tabular models on Azure Analysis Services.Used Azure services (ADLS, Synapse Analytics) for data processing and transformations with HDInsight and Hive.Developed Spark and SparkSQL code on Azure HDInsight for data transformations.Created ETL pipelines in Azure Data Factory (ADF) from various sources like Azure SQL and Blob storage.Developed Spark scripts using Python and Scala for faster data processing.Designed dimensional data models using Star and Snowflake schemas.Configured monitoring systems with Nagios, MCS, and CloudWatch for performance tracking.Worked with relational databases (Oracle, SQL, PostgreSQL) and Snowflake for parallel data processing.Implemented DevOps pipelines using Openshift and Kubernetes for microservices architecture. Show less

• Implemented data ingestion and Data-at-Rest processing solutions using Hadoop, MapReduce Frameworks, HBase, and Hive.• Utilized Sqoop to transfer data efficiently between databases and HDFS, and Flume for streaming log data from servers.• Loaded and transformed large sets of structured, semi-structured, and unstructured data from relational databases into HDFS using Sqoop imports.• Implemented an enterprise-grade platform (MarkLogic) for ETL from mainframe to NoSQL… Show more • Implemented data ingestion and Data-at-Rest processing solutions using Hadoop, MapReduce Frameworks, HBase, and Hive.• Utilized Sqoop to transfer data efficiently between databases and HDFS, and Flume for streaming log data from servers.• Loaded and transformed large sets of structured, semi-structured, and unstructured data from relational databases into HDFS using Sqoop imports.• Implemented an enterprise-grade platform (MarkLogic) for ETL from mainframe to NoSQL (Cassandra).• Responsible for importing log files into HDFS from various sources using Flume.• Analyzed data using HiveQL to generate payer reports for payment summaries transmission.• Imported millions of structured data from relational databases using Sqoop import, processed it using Spark, and stored it in HDFS in CSV format.• Leveraged Scala's Data Frame API for converting distributed collections of data organized into named columns.• Conducted data profiling and transformation on raw data using Pig, Python, and Java.• Developed Shell, Perl, and Python scripts for automating and controlling flow in Pig scripts.• Created prototypes for Big Data analysis using Spark, RDD, Data Frames, and the Hadoop ecosystem, working with CSV, JSON, Parquet, and HDFS files.• Developed Hive SQL scripts for transformation logic and data loading from staging to landing and semantic zones.• Managed and led development efforts with a diverse team, including both internal and overseas members Show less

• Involved in end-to-end process of setting up Hadoop clusters, including installation, configuration, and monitoring.• Automated the setup of Hadoop clusters and implemented Kerberos security for various Hadoop services using Hortonworks.• Responsible for cluster maintenance, including commissioning and decommissioning of data nodes, cluster monitoring, troubleshooting, data backups, and managing Hadoop log files.• Installed and configured Hive, Pig, HBase, and Sqoop on the Hadoop… Show more • Involved in end-to-end process of setting up Hadoop clusters, including installation, configuration, and monitoring.• Automated the setup of Hadoop clusters and implemented Kerberos security for various Hadoop services using Hortonworks.• Responsible for cluster maintenance, including commissioning and decommissioning of data nodes, cluster monitoring, troubleshooting, data backups, and managing Hadoop log files.• Installed and configured Hive, Pig, HBase, and Sqoop on the Hadoop cluster.• Configured property files like core-site.xml, hdfs-site.xml, mapred-site.xml based on job requirements.• Performed data analysis, feature selection, and feature extraction using Apache Spark Machine Learning streaming libraries in Python.• Analyzed system failures, identified root causes, and recommended actions. Documented processes and procedures for future reference.• Collaborated with systems engineering team to plan and deploy new Hadoop environments and expand existing Hadoop clusters. Installed and configured Kerberos for user authentication and Hadoop daemons. Show less