Data Engineer
Current- Developed automated web crawling scripts, increasing data scraping efficiency by 33%
- Enhanced internal web crawling tool by adding new features, enabling data collection from new sources
- Mitigated bot blocking by implementing proxy cycling, state variable management, and cookie capture
- Implemented ETL methodologies to scrape, validate, and store data from websites and documents
- Conducted data analysis to identify variance and errors and wrote scripts to generate statistics on data sets