Hadoop Data Warehouse Architect
Indiana Department Of Education
Indianapolis, Indiana Area
Oversaw entire lifecycle of all Department data, including student ISTEP test scores and data used to evaluate teachers and school corporations. Directed data ingest, storage, ETL, analysis, and release workflows. Negotiated contracts with testing vendors. Coordinated public release of high-interest data such as student test scores. Ensured compliance with federal regulations (FERPA), statutory deadlines, and security best practices (NIST).Supervised build, configuration, and deployment of 21-node Hadoop cluster to centralize data from Oracle, SQL Server, proprietary files, and other data stores. Directed full-time and contract team members across multiple agencies and negotiated contracts for server hardware, Cloudera Enterprise, and the Talend product suite. Held weekly SCRUMs and coordinated training of existing development team in Big Data fundamentals, Hadoop, Hive, Sqoop, Hbase, Talend, Zeppelin, and Linux. Mentored team members and resolved technical and network issues to ensure continued productivity and quality work.Developed Hive-based data warehouse with customizable table schemas to meet data needs of diverse clients. Supervised development of Sqoop framework to update warehouse data from SQL databases incrementally and on-demand. Created standardized HBase “product sets” to aggregate complete data domains (e.g. student, teacher, school) into comprehensive profiles accessible through a common presentation layer.Developed workflows to ingest data from Oracle, SQL Server, Excel, numerous proprietary formats used by testing vendors, and other data stores into Hadoop data lake and Hive warehouse. Designed model ETL, ingest, analysis, and reporting jobs in Talend Big Data, Data Quality, and Data Integration, and supervised their implementation using standard and custom Talend components, Java modules, joblets, and Linux shell scripts. Managed repositories of re-usable Talend contexts and schemas for development, staging, and production environments.