Senior Data Engineer
Current- Developing and design ELT/ETL processes and data pipelines and analyzing data at scale on a big data ecosystem, Integrating 3rd party API’s, and Designing data lake and Lakehouse modeling.
- Languages: Python, Pyspark, SQL.
- Orchestration: Airflow.
- Processing: Spark, Kafka.
- Cloud provider: AWS services – Databricks, S3, Athena, Lambda, RDS, Glue, Secret manager etc.
- CI/CD: CircleCI.