Terrell Tan Email & Phone Number

https://math.stackexchange.com/users/378786/thelast-cipherhttps://philosophy.stackexchange.com/users/25512/thelast-cipherhttps://biology.stackexchange.com/users/50863/thelast-cipher

• 

  Data Engineer

  Phitopolis

  Apr 2022 - Present

  Developed custom ETL scripts using Python, PySpark, Apache Airflow, and Iceberg to efficiently process and manage large datasets.Scraped data from academic, financial, and government websites, utilizing API endpoints, Selenium WebDriver, and BeautifulSoup for robust data extraction.Engineered data pipelines that leverage SGE (Sun Grid Engine) to submit and manage jobs on cloud-based clusters, optimizing memory-intensive data transformations.Conducted comprehensive data quality checks and exploratory data analysis on financial market datasets for quantitative research teams.Contributed to unit testing and refactoring of FASTAPI endpoints within an internal tool designed to scrape pricing information from multiple cloud service providers.Involved in the development of an internal tool that orchestrates cluster creation and management, enabling the execution of data pipeline workloads using Sun Grid Engine and Slurm.