Senior Big Data Architect
• Led a 12-person geographically distributed data engineering team in design and development of a highly scalable POS supply chain big data solution for Fortune 50 retailers using MapR Hadoop• Designed Data Lake pipeline that ingested over 900 TB of data from 15 external sources that was used to support predictive modeling, machine learning, targeted marketing, and inventory logistics• Collaborated directly with clients to understand business data requirements, owned resulting end-to-end solution (data cleansing, transformation, and ingestion into Hadoop Data Lake using HIVE SQL and UNIX/Python scripts), and led team in conversion of scripts to PySpark and SparkSQL• Migrated multi-terabyte datasets from on-premise cluster to Snowflake on AWS• Created Hadoop data model for raw, curated, intermediate, and reporting layers and influenced leadership to move from fair to capacity scheduler, improving job run time by 22%• Tuned 25+ long-running critical HIVE queries, orchestrating a 60% reduction in execution time • Developed and evangelized best practices for data integration, ingestion, transformation, quality and standards, governance, backup, retention, and security strategies for the organization