Developed a resource-aware multi-node task dispatcher and load balancer that monitors server utilization and dispatches task metadata for asynchronous processing. Employed a hybrid queuing system using Redis and Go channels, with prefetching to boost efficiency.Created a real-time status update mechanism utilizing Server-Sent Events (SSE) and Redis Streams,incorporating rate limiting to transfer up to x (custom) events per second per user to the frontend. Implemented reconnection… Show more Developed a resource-aware multi-node task dispatcher and load balancer that monitors server utilization and dispatches task metadata for asynchronous processing. Employed a hybrid queuing system using Redis and Go channels, with prefetching to boost efficiency.Created a real-time status update mechanism utilizing Server-Sent Events (SSE) and Redis Streams,incorporating rate limiting to transfer up to x (custom) events per second per user to the frontend. Implemented reconnection logic to resume updates from the last read event, preventing data duplication.Engineered a distributed monitoring solution for calculating P95 CPU and memory utilization of multiple servers, storing metrics in Redis for consumption by the task dispatcher.Constructed a worker pool system to process and reconstruct arrays of task metadata, execute endpoint requests, and deliver responses via Pub/Sub for status updates.Developed an email subsystem with template support and LRU caching. Show less