• Implemented multiple end-to-end ETL processes in Shell scripting that extract batch data from AWS S3 bucket or SFTP to HDFS (Hadoop Distributed File System) and load into target tables using HiveQL in both static and dynamic partitioning for efficient data access and usage• Developed a data quality dashboard pipeline that monitors 100+ core data science ETL jobs by using Oozie workflow scheduler command-line APIs to extract workflow running status on a daily basis, securing production issue tracking and recognized by the leadership to be the best practice for data quality initiative• Optimized SQL query design for the highest priority data science ETL pipeline and migrated its HiveQL queries to PySpark and redesigning intermediate table loading processes, reducing runtime by 30% • Redesigned a critical incoming data pipeline with the upstream web development team to migrate data transmission process from multiple REST API calls to a single batch CSV data file ingestion in AWS S3 bucket, enhancing the pipeline efficiency and stability• Maintained and troubleshot critical data quality Tableau dashboards for data science model pipelines to correctly reflect and debug model running issues ranging from platform resource limitation to table loading errors• Automated ad hoc business requests such as downstream table dependency checking to ensure smooth changes to existing tables without unexpected downstream impact using Python and Shell scripting • Aggregated and analyzed various data sets to provide actionable platform insights