Amit Sangwan Email & Phone Number

Noida, Uttar Pradesh, India

India

>> Web Scraping ETL Pipeline• Tech Stack: Python, Selenium, Beautiful Soup, Requests, Pandas, PostgreSQL, GCS, AWS S3, FastAPI, Airflow, Docker, Kafka, Elastic Search• Prepared Python scripts for Web Scraping using Selenium, Requests, BeautifulSoup and Python Pandas library to pull data from over 50+ websites.• Build and maintained a robust, scalable ETL data pipeline for Web Scraping from 1000+ websites.• Implemented a suite of RestFul APIs using FastAPI to perform various ETL related operations and various other backend tasks.>> Data Backup and Restoration Pipeline• Tech Stack: Python, GCS, AWS S3, Elastic Search, Shell Script, Crontab, Slack• Designed and implemented a comprehensive end‑to‑end data pipeline solution to automate backups, effectively mitigating data corruption and seamlessly facilitating disaster management and recovery processes, ensuring swift and reliable restoration of critical data assets in the event of a disaster.>> Data Migration Pipeline• Tech Stack: Python, GCS, AWS S3, Open Search, Airflow, Slack• Build a robust data migration and loading pipeline, leveraging AWS S3, Python scripting, OpenSearch, Apache Airflow, and Slack integration. Automated the seamless transfer of data from Cloud storage to OpenSearch index, ensuring efficient data ingestion, indexing, and real‑time monitoring via Airflow DAGs and proactive notifications through Slack channels, optimizing data management and accessibility.>> Improvements and Features• Implemented optimisation techniques in Web Scraping ETL pipeline to decrease the data loading time by 20%.• Improved data quality by 15% in Web Scraping ETL pipeline by building custom parser functions to clean data and Python Classes for data validation to catch faulty data.• Developed 15+ data cleaning scripts to transform raw data in desired output format to load in Elastic Search.