Software Engineer
CurrentOver the past two years, my primary focus has been on extracting, transforming, and loading (ETL) customer bills from AWS or GCP.As a prominent agent for both GCP and AWS, responsibilities include billing customers on their behalf.Main areas of concentration:Development of the backend system using Python and FastAPI.Utilization of Cloud SQL for data storage with a focus on designing optimal schemas and selecting suitable indices to enhance SQL query efficiency.Emphasis on reducing code time and space complexity for performance optimization.Deployment of services on GCP Cloud Run and GKE, leading to a deep understanding of mechanisms and effective design of service structures, including networking, volumes, and CPU/memory settings.Leveraging BigQuery for storing raw customer bill data, enabling effective grouping and searching at a large scale.Implementation of the MapReduce concept to parallelize the parsing of substantial datasets, utilizing DataFlow as the supporting tool.Adherence to clean architecture principles and writing maintainable code.Conducting stress tests and unit tests to ensure fault tolerance.I have implemented GCP Cloud Task and queue to handle data counting in parallel, which has significantly saved us a considerable amount of time