• Migration from GCP to AWS• Creation of a datalake using AWS S3• Utilize HashiCorp Terraform as infrastructure as code• Creating AWS resources using Terraform• Creating new AWS Roles, Users, Group using AWS IAM• Worked with AWS Secret Manager to save credentials and retrieve them• Utilise AWS Glue as ETL• Worked with PySpark to do transformation, clean the data and save them in Redshift• Used AWS Glue crawler to infer the schema• Used AWS Glue catalog for the metadata• Worked with EMR Spark (pyspark) to clean and transform the data• Used AWS Redshift as a data warehouse• Created external tables from S3 in AWS Redshift• AWS Athena to query the data and do the data validation between GCP and AWS• Used AWS Cloud9 as IDE.

Designed, maintained, and implemented data pipeline from remote IoT sensors using REST API, AWS S3, AWS Firehose, AWS Lambda, AWS Kinesis Analy&cs, and RedshiKDesigned and implemented an S3 data lake for storage of raw dataDesigned and implemented data lake, data warehouse, and datamart design on AWS platformMigrated custom models are written by data scientists into S3 based production workflow.Monitored daily production runs running on AWS cloudTested and validated REST API endpoints.Implemented custom ETL, ELT pipelines at the request of other departments

Implemented and maintained Google GCP data lake running on Google Cloud Storage using TerraformDebugged and maintained SQL queries from data scientists running on Google GCP BigQueryUsed Cloud Dataproc for scaling and running Spark jobsWrote, debugged, and maintained Google Airflow/Composer scripts for workflow managementUsed Pub Sub for streaming remote data via REST API endpointsMaintained existing batch data ingestion pipelines written in Google Data Flow/Apache spark

Designed, developed, and maintained IoT sensor pipeline using Apache Kafka, Apache Spark, and Apache Hive.Developed a data pipeline for extracting social media information from online sources.Used Python for web scraping of relevant articles in order to gauge consumer sentiment.Used Python to make requests from news sources Web API’s as well as social media API’s such as Facebook and TwitterWorked with Apache Spark for large data processing integrated with functional programming language Scala for fast, distributed processingWorked with both batch and real-time processing with Spark frameworks. Created a Kafka producer to connect to different external sources and push data to a Kafka broker.Stored unprocessed JSON and HTML files in HDFS data lake Worked with analytics team to provide querying insights and helped develop methods to map informative text more efficiently Optimized HIVE scripts were relevant by replacing them with Spark queries.Optimized tables and schema for use in machine learning models for a more value-added analytic modelSupported analytics team with HDFS data lake using HIVE combined with Spark for data query.Retrieved structured and unstructured data from HDFS using HIVE and Spark in addition to MapReduce jobsImplemented advanced procedures such as text analytics and processing using in-memory computing capability methods through Scala scripts running on Apache SparkUsed Spark Context and Spark Session to process text files by flat mapping, mapping to RDD, and reducing RDDs by key to identifying text containing key metrics