Senior Software Developer
CurrentRoles and Responsibilities :-• Providing data from various sources including databases, websites, social media platforms, PDFs, etc., through implemented data pipelines to a centralized data warehouse of nearly 1500 websites.• Scrapy implementation on spiders to proceed asynchrones and scalability and to automate the repeated process.• Supervised the team to automate deployment and scaling of data pipelines.• Handling the web scrapping team internally and getting the works done• Built and audited application architecture to handle thousands of spiders with effective scheduling..• Responsible for designing databases and project structure, deploying applications on client sites.• Trained, Created and maintained documentation for data pipelines, architecture, and best practices to ensure knowledge sharing and onboarding of 7 new team members.
