 A seasoned Senior Data Engineer with over 5 years of experience specializing in designing, implementing, and optimizing data infrastructure and pipelines to support business needs and decision-making processes. Utilized Apache Hadoop for distributed storage and processing of large datasets, Apache Spark for real-time data processing, interactive analytics, and machine learning tasks, Apache Kafka for building real-time data pipelines. Designed and managed data warehousing solutions using Amazon Redshift, enabling high-performance querying and analysis of large datasets for business intelligence and reporting purposes. Specialization in Apache Kafka and its technologies such as Brokers, Schema Registry, Zookeeper, Producer, Consumers, Shared Clusters, Cluster linking, Mirror-Making for real-time data streaming and designing scalable data pipelines, optimizing performance, and driving innovation for collaborating cross-functionally to deliver impactful data solutions. Managed and optimized relational databases like PostgreSQL and MySQL for efficient data storage, retrieval, and manipulation, ensuring data consistency and integrity. Utilized Python and SQL for data manipulation, analysis, and transformation tasks, leveraging libraries like Pandas, NumPy, and SciPy for advanced data processing and statistical analysis. Implemented containerization using Docker for packaging applications and dependencies into standardized units, ensuring consistency across development, testing, and production environments. Orchestrated containerized workloads using Kubernetes for automated deployment, scaling, and management of containerized applications, improving deployment efficiency and resource utilization. Designed and implemented Extract, Transform, Load (ETL) pipelines using Apache NiFi and Apache Airflow for orchestrating data workflows, enabling automated data ingestion, transformation, and loading into target data stores. Leveraged AWS for building scalable and cost-effective data solutions, utilizing services like AWS S3, EC2 for object storage, Lambda for computing, API Gateway for serverless, AWS Glue for ETL, Amazon Redshift for data warehousing. Managed code changes and collaborated with team members using Git, GitHub and GitLab for version control, ensuring code integrity. Implemented infrastructure as code (IaC) using Terraform and AWS CloudFormation for automating the provisioning and management of cloud resources. Proficient in Agile methodologies including Jira and Trello for Scrum and Asana for Waterfall.