Data Engineer
Current-Create and maintain distinct web scrapers daily for government sites by utilizing Python modules and libraries such as Requests, BeautifulSoup, Selenium, and Scrapy; ultimately capturing the bid and contact data of 3000+ agencies within the last 6 months.-Updated integration strategy for capturing multiple agency websites with similar formats by using a base scraper and utilizing it to capture the data of numerous different agencies-Collaborate with a 3+ person team to manipulate and perform ongoing data cleansing on company databases to ensure clean and accurate agency data for 100,000+ agencies for company and client utilization. -Developed a 2 month training curriculum for current employees and led weekly training sessions for off-shore and in-state employees to facilitate appropriate web scraping strategies and increased web scraper production -Managed integration of workplace computer systems through installation of Ubuntu and ultimately configure the computer environment for web scraping