Data Automation Engineer
Current- Collaborated closely with the CTO to identify data sourcing requirements and deliver high-quality data in the specified formats.- Developed and optimized automation scripts using Node.js and Python to extract, process, and transform data from diverse web sources, improving data accuracy and reducing manual workloads.- Designed and implemented custom Node.js endpoints for efficient data uploads and management, ensuring smooth integration between different systems and automating key workflows.- Engineered and maintained a Linux-based service that automatically launches on system boot, utilizing Ngrok and Tailscale for secure SSH tunneling, and set up automated email notifications for real-time system updates (e.g., SSH tunnel IPs).- Integrated Google Cloud services (including Google Sheets) into data pipelines, enabling seamless data storage, real-time updates, and effective data management.- Leveraged Playwright and Puppeteer to automate web interactions and streamline data extraction tasks, improving the speed, reliability, and scalability of data collection processes.- Utilized Spotify API and Facebook API for data retrieval and analysis, enhancing project capabilities and providing richer insights.