Data Engineer
Current- Executed efficient big data transfers using PySpark to Amazon S3, optimizing data handling and storage solutions.- Implemented transformation ETL processes from Oracle SQL to PySpark, enhancing data processing efficiency and scalability.- Created and managed platforms (Spark, Airflow) using Docker and OpenShift.- Developed and deployed web services with Python and Flask.- Created DevOps/Data pipelines for ML projects on Azure DevOps.- Built an ML development platform using Docker and OpenShift for data scientists.- Developed web scraping tools for automation with Python and Selenium.