Mukesh Kumar Email & Phone Number

I have 9 + years of development experience is the fields Crawling and Extraction, Large-scale Data Management, and Distributed Algorithms.Specialities: Crawling and Extraction, Data Pipeline Architecture, Data AnalysisDistributed Crawler​ Infrastructure​ : At DataWeave, we crawl 2000+ websites round the clock. As part of the crawler infrastructure, these are the major challenges we address: (1) managing scale, (2) managing crawling policies and priorities, and, (3) managing frequent changes in website structures. We manage scale by distributing tasks to many lightweight workers. This is handled using Celery. We queue URLs to be crawled based on crawling policies and priorities. Each worker takes a list of URLs from RabbitMQ and makes network request to get data and persist data content for the URL into our cache system so that extraction process can be run on the top of these data to get actionable data points. Since domains do not follow any standard structure, their configuration changes from site to site. In order to handle boundary cases, and website specific deviations, we also integrated config pattern for each website. Crawler workers pick up these configurations from the DB and crawl based on those. In addition, website structures change frequently ­­ new web pages are added or existing URLs are changed. A monitoring system alerts the crawler of these changes.

Listed skills include Ruby, Shell Scripting, Git, Linux, and 3 others.