Data Engineer
CurrentProject: Real-time Data Ingestion and Processing Pipeline on Uber ride-hailing platform• Collaborate frequently with senior team members to implement data transformation tasks such as filtering, and aggregating. Specify windowing strategies for handling streaming data using Apache Beam for stream processing.• Assists in configuring Google Cloud Platform (GCP) services, including setting up Pub/Sub for event streaming involving creating topics to publish messages and subscriptions to receive messages for processing in the data pipeline.• Conduct data quality checks, monitor pipeline logs, and assist in troubleshooting data pipeline issues to ensure smooth data flow. List of Data Engineering concept I am comfortable with: - Filtering- Aggregating- Sorting- Flatten transform- ParDo and DoFn - Combiner- CoGroupByKey- Side Inputs and Outputsetc.