Design, develop and maintain non-production and production transformations in AWS environment and create the data pipelines using PySpark Programming. Wrote, compiled, and executed programs as necessary using Apache Spark in Scala to perform ETL jobs with ingested data. Worked on Big data on AWS cloud services i.e., EC2, S3, EMR and DynamoDB• Implemented End to End solution for hosting the web application on AWS cloud with integration to S3 buckets. Used Spark Streaming to divide streaming data into batches as an input to Spark engine for batch processing. Involved in designing and deploying multi-tier applications using all the AWS services like (EC2, Route53, S3, RDS, Dynamo DB, SNS, SQS, IAM) focusing on high-availability, fault tolerance, and auto-scaling in AWS Cloud Formation• Wrote Spark applications for data validation, cleansing, transformation, and custom aggregation and used Spark engine, Spark SQL for data analysis and provided to the data scientists for further analysis

Utilized AWS CLI to automate backups of ephemeral data-stores to $3 buckets, EBS and create nightly AMIs for mission critical production servers as backups.Maintained and developed Docker images for a tech stack including Cassandra, Kafka, Apache, and several in house written Java services running in Google Cloud Platform (GCP) on Kubernetes. Extensively used Google stack driver for monitoring the logs of both GKE and GCP instances and configured alerts from Stack driver for some scenarios. Involved in Setup and Configuration of Puppet Configuration Management.

Utilized Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine Learning use cases under Spark ML and Mllib.Identified areas of improvement in existing business by unearthing insights by analyzing vast amount of data using machine learning techniques. Interpret problems and provides solutions to business problems using data analysis, data mining, optimization tools, and machine learning techniques and statistics. Designed and developed NLP models for sentiment analysis

Responsible for building scalable distributed data solutions using Hadoop, Involved in collecting, aggregating and moving log data from servers to HDFS using Flume. Imported and Exported Data from Different Relational Data Sources like DB2, SQL Server, Teradata to HDFS using Sqoop

Coordinated with business customers to gather business requirements and interact with other technical peers to derive technical requirements and delivered the BRD and Technical requirements documents