Aravindh Madhavan Email & Phone Number

Aubrey, Tx, Us

Frisco, Texas, United States

Plano, Texas, United States

• Led the migration of the on-premises Hadoop data ecosystem to AWS cloud, resulting in improved scalability, reliability, and cost efficiency.• Built ETL/ELT data pipelines using AWS EMR, Glue, Lambda, EKS, and other AWS data analytics services, along with Spark, to transform data across various stages of the data lifecycle.• Configured and managed AWS EMR clusters tailored to specific job requirements, optimizing resource utilization and improving job performance.• Developed and implemented ETL processes using SparkSQL on EMR to extract, transform, and aggregate data from multiple sources and file formats, loading it into Amazon Redshift.• Implemented a hybrid automation strategy using Control-M for on-premises environments and Airflow for AWS, enhancing operational efficiency and ensuring seamless data transfers.• Developed and implemented custom shell scripts to automate data processing jobs, enhancing workflow efficiency and reliability.• Provided production on-call support, resolving data pipeline issues, system outages, and performance bottlenecks to ensure uninterrupted data operations.• Designed and implemented incremental data loading logic for fact and dimensional tables, reducing storage usage on S3 and minimizing redundant data storage.• Investigated and resolved root causes of EMR failures and performance issues, significantly improving job performance.• Utilized Jenkins CI/CD pipelines and Liquibase to provision, configure, and maintain modular and scalable ETL pipelines across multiple AWS regions.

United States

• Analyzed RAWLS datasets using Python libraries and SQL queries, creating bespoke tables in Tableau and visualizing results through interactive dashboards.• Worked on all phases of data integration development, including real-time and batch data pipelines design and implementation, as well as Big Data ETL and reporting support.• Migrated legacy Data Warehouse from Oracle/SQL Server to a central Data Lake on AWS using Informatica.• Developed real-time streaming applications using PySpark, Kafka, and Hive on a distributed Hadoop cluster.• Created serverless data pipelines using AWS Lambda, Glue Catalog, and Athena; managed workflows with Apache Airflow.• Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from various file formats.

Chennai, Tamil Nadu, India

•Designed ETL processes with AWS Glue and Apache Kafka for real-time data ingestion and processing in an Amazon S3 data lake, supporting advanced analytics.•Developed a data processing framework using Apache Spark and PySpark, integrating Spark Streaming and ElasticSearch for fast data retrieval and real-time analytics.•Used Python scripting and libraries like NumPy for complex data manipulation and analysis, configured Apache Airflow for workflow orchestration.•Utilized Cloudera Stack components for large-scale data management and implemented NiFi for automated data flow to HDFS.•Optimized Spark applications in Databricks with Spark-SQL for customer behavior analysis and trend aggregation.•Demonstrated proficiency in dimensional and transactional modeling, including managing Slowly Changing Dimensions (SCD).•Conducted extensive data cleaning, scaling, and implemented statistical models for diagnostics and predictions using Python libraries.•Performed A/B testing and SQL query performance analysis, reducing execution time by 40%.•Developed SQL-based reports and dashboards with Power BI for KPI monitoring and performance insights.•Created business intelligence dashboards with Tableau, MicroStrategy, and AWS Quicksight, integrating Snowflake and Redshift for scalable data warehousing.

Master'S Degree, Data Science

Btech - Bachelor Of Technology, Civil Engineering