Data Engineer
Current• Migrate data and applications to AWS platform implementing such services as RDS, EMR, Kinesis, Glue, Redshift, Data Pipeline, S3, etc. for cloud computation, storage and transfer.• Data Ingestion and ETL in AWS Glue from datastore such as RDS and S3 and third-party industrial data from IRI and Numerator, modified ETL script in PySpark or Scala, triggering the process using Lambda function.• Crawled and collected real-time data from multiple sources, for example, app data from mobile devices, website user data and third-party/partner (Adobe Analytics) web analytics data, then pushed into AWS Kinesis Stream.• Analyzed data from Kinesis Stream in Kinesis Analytics using SQL, some would be directly passed in EMR Hadoop and Spark cluster for further processing.• Implemented Kinesis Firehose to transfer (e.g.: to ORC and Parquet) and sink data into RedShift and AWS RDS in S3.• Conducted programming using different Spark APIs such as Scala, RDD, PySpark and SQL to transform data for further processing on generating predictive analyses.• Collaborated with Application Development team to build and develop customized dashboard connecting with the data from S3, Kinesis, DynamoDB and data warehouses such as Redshift.