At Zepz, led migration of BigQuery datawarehouse to Databricks. Next step was Migrating off Composer managed airflow over to Astronomer.Also led the two POCs on dbt with Databricks as well as Astronomer migration of AWS MWAA.Tested and implemented full CICD and development GIT cycle with CircleCI.Managed and designed architecture for data pipelines by setting up over 5 separate repositories for different teams.Ingested one main transactional DB and one customer DB via HVR as a means for ODS operation.Ingestion of sources into databricks using delta live tables.Created and provisioned metastore objects and access via Terraform.Streamlined RBAC policy for data engineering toolings to better audit requirements and safeguarding.Wrote a spike about CDC and semi real time streaming using various procedures in databricks.Migrated a few CircleCI projects over to Github Action.

At Extra Card, I've been designing data pipelines, datawarehouse and setting up infrastructure necessary to ingest, procure and transform internal and external data to support critical downstream applications,Customer Data Platform, product and data analytics. Some of key contributions since my start in Data Engineering and DevOps are as follow:• Deployed serverless application using lambda functions in container images with AWS SAM, ECR, Cloudformation and DynamoDB for storage• Developing , deploying and automating application using python Rest API to ingest internal and external apps e.g Customer.io , Everflow and Intercom.• Designing dbt projects and structure and setting up workflows, source freshness configurations, data quality checks and logging• Assisted in building CDP with Segment and Amplitude and working on migrating to RudderStack• Leading CICD practices for Data team setting up AWS ECS on Fargate and EC2 with Prefect.• Scheduling and alerting with SNS, Eventbridge, Cloudwatch• Security best practices on AWS with Secret Manager and KMS encryption• Continuous ETL and modeling on AWS Redshift and AWS S3 • Putting in place CI/CD best practices and flow diagram on dbt cloud • Setting up Fivetran to ingest raw data and transform/normalize in Snowflake • Configuring Github Action as default CD process along with dbt for PR• Setting up Airbyte docker container on AWS EC2 ingesting PostgresQL into Snowflake• Documenting best practices on Git and Github with branching and merging use cases• Testing orchestration use cases using docker and Kubernetes • Configuring external and Internal stage, snowpipe and integration between AWS S3 and Snowflake• Setting up monitoring, alerting dashboards and logs in DataDog across all data pipelines• Managing over 500 dbt models and macros to assist data science and BI analytics dashboards• Created Metabase metrics ,eg customer churn/conversion rate, customer acquisition cost..etc.

At Customer Data Platform, I'm working on designing and building the single source of truth providing FCA with key consumer insights and help be more effective with business's marketing strategies and campaigns by relying on data science and machine learning approaches to tackle market share challenge in automotive space.First and foremost, working with Data Scientists developing ML models such as :• Migrated Gradient Boosted Tree model from on-premise Data Lake to Cloud CDP• Assisted in feature engineering pertaining to recommending Next Best Action• Inactive or active customer data segmentation as well as data unification using enrichment of demographic, weather and adobe external data sources• Built robust and reliable pipelines to ingest, transform and orchestrate using DigDagAdditionally, designing, testing and creating multiple analytical pipelines in CDP such as:• Designing and deploying use cases in DEV,QA and Prod using GIT and Github• Managing CI/CD best practices and deployment across all environments• Architected an end to end data pipeline to load, transform and unify three major brands of Ram, Dodge and JEEP customer incentive offers data out of their respective Web APIs using python built in libraries and docker• De-serializing JSON messages in Snowflake and normalization of data on the cloud end• Transforming and parsing semi-structured connected vehicle app data in Snowflake• Testing and scheduling Spark Standalone via Docker containers• Performance Tuning on PostgreSQL Greenplum, Hive and Presto• Creating, designing and scheduling Data Warehouse workflows with proper logging and reporting (emails,notifications,..etc) using DigDag, Cron and UDF

Projects: Food Safety, Fraud Analytics and Cloud • Testing third party vendors’ data to assess Food Safety Audits for global stores • Setting a system to accept or reject audits processes based on business requirements• Testing a solution to write Kafka Store sales messages into AWS cloud and analyze them using AWS Glue, Athena and Redshift• Analyzing Social Media data such as Facebook, Yelp and Google reviews for better marketing and better targeting coupons and promotions• Performing ETL & ELT tasks on external sources and supporting data warehouse using Talend Studio, SQL SERVER Management Studio(SSMS) and custom shell scripting• Version control integration with Bitbucket and Jira

Most recent project: IOT analytics- - Real Time StreamingDesigned global real time data streams to aid identify manufacturing damages such as scratch and chip on new vehicles.This resulted in significant cost saving due to reduce warranty claims. Initial pipeline: Semi-Real time data processing: • Using Spark streaming or NiFi to receive, extract and load JSON messages into HDFS • Using Hive SerDe or Alteryx, to parse and transform the JSON to tabular format Final pipeline: Real Time processing• Connecting Kafka to Spark and flowing thousands of messages per minute to HBase • Query and visualize the data using NoSQL and reporting on the fly with PySark/Scala• GIT and Docker implementation in HPC ProdProject 2: EOL (End of Line) Vehicle TestingThis project aimed at providing insights about tested vehicles prior final delivery to dealerships. This resulted in multi-million dollars in saving plus an easy user interface for manufacturing plants users across EU,SA and NA. Phase 1: Created a Quality Application reporting Had led analytics teams to gather information and execute a solution that met quality concerns. • Utilized spark, hive and SQL queries over millions/billions of records • Automated and designed workflow using Oozie in production cluster Phase 2: Replicating MAHLE Aftermarket,Inc API• Optimized, pivoted and tuned multiple queries in Hive and PySpark using Parquet and ORCProject 1: Access and Security Control DashboardImproved Data Governance strategy planning and helped better data management in Hadoop

VZ-SIM in a Security information event management project where extensive amount of log data is gathered and ingested by Flume/Sqoop stored to HDFS and processed using Pig Latin and Hive queries and reported and visualized by Tableau. The log agents, Flume sitting on the Application servers, Routers, Switches and Firewalls push the log files to Web API and land the data into HDFS or NoSQL HBase.• Analyzing log data using Hadoop processing tools such as Hive, Impala , Pig and Spark• Involved in data ingestion into HDFS using Apache Sqoop, Flume and Kafka• Used HQL and Pig queries t and custom UDFs o analyze, encrypt and transform data• Developed and build pipeline processes to land and extract the data cubes for Tableau

. Managed teams of safety engineers and coordinating dispatch to over 10 oil field sites . Led meetings with testing, drilling, production and logistics managers to review safe practices and presenting ways to improve working conditions.. Reviewed and documented workforce best practices according to OSHA and safety protocols to ensure minimal personnel injuries and property damage.. Put in place card system designed for all staff to report and document hazards or any potential risk at the company’s premises.. Held weekly safety and operations workshops for junior staffs to increase awareness and to ensure rules and regulations are to be followed.

. Managed teams of 10 field operators and foremen on three oil rigs platform and overseeing day/night shifts planning. Provided Companyman with tools measurement, testing and status updates of operation. Performed monthly blow-out, fire and H2S drills on-site and in corporate office.