1. Worked as a Data Engineer in the Data Science team at Concirrus.2. Built APIs required for ETL process of data pipelines from different sources of data for both real-time and batch streaming processes, using Python, Spark, and AWS technologies.3. Design Architecture: Designed and developed LLDs and HLDs for developing optimal data pipeline architecture.4. Performed auditing of AWS S3 buckets, EMR clusters, and EC2 instances and assisted in implementing different methods using Lambda, removing Spark clusters and migrating of scripts to EMR in order to save cost 5. Worked with stakeholders and product team, to assist them in data-related technical issues and regarding ad-hoc data requests.6. Real-Time Data Stream processing: Refactored existing data pipelines to handle large amounts of streaming JSON data, integrate error handling, and automate it using Apache Airflow.7. Worked on creating and developing a complete end-to-end process to extract and transform data and make SQL tables for analytics, using AWS Athena.8. Feature Development: Developed an automated end-to-end data pipeline to build a feature that gives insight into a vessel’s history, showing specific events over the course of the lifetime of the vessel from scratch to meet the product requirements.9. Integrated automated Alerting System for Job Monitoring.

Data Analytics