Key Responsibilities:- Developed Python web scraping scripts to collect datasets from dynamic URLs on vendor sites, generating Bash scripts with download commands and dynamic paths to automate downloads and storage.- Implemented ETL pipelines for data ingestion and historical backfilling from databases, APIs, and files. Automated validation and monitoring to ensure dataset integrity.- Conducted data analysis and preprocessing using Python libraries like NumPy and Pandas. Created scripts for integrating data into various sources like databases and file formats.- Designed data pipelines using the Snowflake connector with Python, creating schema tables, configuring settings, and developing external stages. Optimized data transfer from Wasabi to Snowflake.- Proficient in SQL and experienced with databases such as MySQL, PostgreSQL, and Snowflake.- Set up PostgreSQL for time-series data ingestion with Python connectors. Debugged and enhanced production databases by replicating and testing locally before deployment.- Developed Python scripts for transforming data, including flattening nested JSON and XML structures. Performed data operations like pivoting, melting, filtering, merging, and splitting large datasets using Pandas.- Managed large dataset processing using Parquet, HDF5, and Arrow IPC formats. Optimized data processing with asynchronous I/O and Python multiprocessing.- Orchestrated workflows with Prefect and Airflow, migrating tasks for improved scheduling. Containerized dataset scripts with Docker, integrating tools for S3-compatible bucket access, including Goofys.- Developed automation scripts for monitoring dashboards and alerting on data quality issues. Optimized Python scripts for complex data handling, enhancing performance.- Worked extensively with Linux VMs, Linux commands, and terminal-based tasks. Monitored system resources such as RAM and memory during processes.- Demonstrated expertise in Bash scripting for automation on Linux systems.