Data Engineer
CurrentDevelopment of ETLs and ELTs using Selenium, Airflow, Spark and DBT to acquire and process data from various sources, with logs processed and displayed on PowerBI dashboards with various metrics and KPIs. Automation of manual routines, with CI/CD pipelines integrated into Azure, orchestrating with Prefect, and integrating with AWS services for monitoring and cataloging. Development and maintenance of Delta Lakes with Iceberg, and construction of Data Warehouses and Data Lakes.Python(pandas, pyspark, polars, scikit-learn, h3, pytest, selenium, beautifulsoup), SQL, AWS (S3, Glue, Athena, RDS, Lambda, StepFunctions, RedShift), Azure (Functions, Events, Geoserver, Blob Storage), Terraform, Docker, DBT, Airflow, Prefect, ETL, ELT, APIs, RabbitMQ, Data Warehouse, Data Lake, Delta Lake, Data Modeling, Snowflake, Vertica, Neo4J, ElasticSearch, SQL Server, Salesforce, Data Profiling, Data Quality, Great Expectations, Data Literacy, Data Analysis, PowerBI, DAX, Postman etc.