Lead Software Engineer
Current- Lead Software Engineer at AquaRobotics playing a crucial role in developing innovative solutions and driving project success through effective collaboration and data workflow optimization.- Design and implement end-to-end system architecture for efficient data pipeline and processing, utilizing technologies such as Python, Apache Airflow, and Apache Spark.- Deploy PyTorch-based deep learning models for image prediction and fungi growth analysis.- Schedule data pipelines for automated fish cage image prediction, leveraging Apache Airflow for orchestration.- Set up and manage cloud infrastructure for data processing on AWS, including AWS EKS (Elastic Kubernetes Service) and AWS EMR (Elastic MapReduce), using Terraform for infrastructure as code.- Architect and implement serverless APIs for seamless integration with AWS Lambda and AWS API Gateway.- Lead project implementation, delivery, and maintenance, demonstrating strong leadership and project management skills.- Collaborate with team members to build and fine-tune deep learning models for image inference, enhancing image prediction capabilities.- Optimize data workflows for efficiency and performance, incorporating Docker for containerization and Kubernetes for container orchestration with Karpenter.- Develop an end-to-end data pipeline for processing robot logs from AquaRobotics' robots, handling zip files containing multiple CSV files using Apache Airflow and PySpark for data processing. Cleaned data is stored in AWS Timestream, and Grafana integration is implemented for data visualization.