Data Engineer
Current• Developed automated web crawling scripts, increasing data scraping efficiency by 33%• Enhanced internal web crawling tool by adding new features, enabling data collection from new sources• Mitigated bot blocking by implementing proxy cycling, state variable management, and cookie capture• Implemented ETL methodologies to scrape, validate, and store data from websites and documents• Conducted data analysis to identify variance and errors and wrote scripts to generate statistics on data sets