Software Developer
CurrentAs a Data Engineer at ArabyAds, I am at the forefront of architecting and managing data pipelines, ensuring seamless data ingestion, processing, and analysis. Leveraging a robust skill set and cutting-edge technologies, I contribute to the development of scalable and efficient data solutions.Responsibilities:π οΈ Data Pipeline Architecture: Design and implement end-to-end pipelines for ingesting, transforming, and loading data from diverse sources.π Data Processing: Use Kafka, Spark, and Hadoop for processing and analyzing data at scale, extracting valuable insights.π Integration and Orchestration: Seamlessly integrate data using Kafka Connect, and orchestrate workflows with tools like Oozie and YARN.π Big Data Ecosystem Management: Manage and optimize Hadoop ecosystem components like HDFS, YARN, and MapReduce.π Data Security and Compliance: Implement data security measures and ensure compliance with regulations.Achievements:π Scalable Pipelines: Reduced processing time.π Data Quality Improvement: Decreased data errors, through enhanced quality checks.π Innovative Solutions: Introduced real-time processing, enhancing data capabilities.π Performance Optimization: Improved processing speed, through algorithm optimization.π Streamlined Integration: Automated ingestion processes, increasing efficiency.Skills:π οΈ Big Data Technologies: Hadoop, Spark, Kafka.π Data Integration: Kafka Connect, Oozie.π Data Security: Confidentiality, integrity, availability.π§ͺ Testing and Optimization: Troubleshooting, performance optimization.π Database: MariaDB, Greenplum.π Problem-solving: Complex challenges resolution.π Innovation: Innovative solutions introduction.