Consultant, Investment Risk Management Technology
CurrentPython/Pandas/Airflow, PostgreSQL, Elasticsearch, Nifi, Docker.Implementing and maintaining the internal service that calculates the trading book's risk position in the market.Using Python/Pandas, PostgreSQL, and Apache Airflow running on Linux, we are maintaining a system thatprocesses very large-scale customer and market data. These are some of the importantissues/concerns that lasted for years, and I help resolved:● By developing a Python tool to analyze very large logging files that multi-thread information mixedtogether, I helped to locate the pain point of a very long data processing that sometimes runs over13 hours, and developed a brand new thread job dispatcher to reduce the processing time to about 3hours● By refactoring very large Python modules/functions, I help to remove cross dependencies of dataprocessing code, and re-organizing them into small functions with clear interfaces to act each other.This helped the team to add more functions to the module.● Help refactor many modules to make the code base more clean and organized.● With my good understanding of how database runs, I helped to fine-tune the performance of thedatabase operation making quite some of them hundreds of times faster.