Senior Data Engineer
CurrentPromoted to Senior Data Engineer (April 2024) due to my high performance. Responsible for data onboarding and development and maintenance of the infrastructure of RavenPack’s Data science team.My day-to-day involves (but is not limited to):Adding responsibilities to my previous role of deploying ML models in a docker-based environment and Sagemaker (MLOps), AI agent deployment, and designing and building an AWS Batch Jobs orchestrator. Worked mainly with AWS SageMaker, AWS DynamoDB, AWS CloudFormation, Python, and Shell scripting.Data engineering responsibilities:1. Data Onboarding: I am responsible for the end-to-end process of collecting, processing, and making large volumes of data available to our data science team.2. Datalake Maintenance: In AWS, I implemented, deployed, and maintained our datalake infrastructure, which utilized services such as Athena, S3 for storage, and AWS Batch jobs3. Development Environment: I designed, deployed, and maintained our development environment. I designed and built it using Docker containers running Jupyter Lab and Python. I streamlined the deployment process using shell scripts, ensuring our data scientists had a productive and consistent working environment.4. Shell Script and Python Development: I developed shell and Python scripts to enhance automation and streamline data onboarding processes. These scripts were crucial in automating repetitive tasks, allowing faster data integration and more efficient workflows.5. Providing specialist support to the Data Science. I provide Level 1 HelpDesk to the Data Science team regarding our development environment and datalake processes:SQL optimizationPython codingScript deployment best practicesSkills: Cross-functional Team Communication · Docker · AWS Batch Jobs · AWS SageMaker · AWS DynamoDB · Python · Schell scripting · SQL · Team work · Communication · Agile