➢ Designed, maintained, and implemented healthcare data pipeline using Teradata, Airflow, HIVE, Spark, and bash scripHng combined with Python ML (machine learning) library.➢ Coordinated with SCRUM target deadlines as well as BitBucket repositories.➢ Designed, maintained, and debugged Airflow scripts.➢ Debugged data issues stemming from primary source data residing in Teradata.➢ ProducHonalized and operaHonalized custom models wriRen by data scienHsts intoproducHon.➢ Monitored periodic producHon model data flow.➢ Resolved code, data, and combined code/data issues stemming from upward data change.➢ Experienced with both ETL and ELT pipeline design and implementaHon.

➢ Maintained AWS EMR Spark using PySpark and uHlized DataFrames and SparkSQL API for faster processing of data➢ Registered datasets to AWS Glue through Rest API➢ Used AWS API Gateway to Trigger Lambda funcHons➢ Queried with Athena on data residing in AWS S3 bucket➢ Wrote AWS Step funcHon used to run a data pipeline➢ Monitored and managed services with AWS CloudWatch➢ Performed transformaHons using Apache SparkSQL➢ Wrote Spark applicaHons for data validaHon, cleansing, transformaHon, and customaggregaHon.➢ Developed Spark code using Python and Spark-SQL for faster tesHng and data processing.➢ OpHmized Spark scripts to reduce latency➢ Monitored and managed services with AWS CloudWatch➢ Configured ODBC Driver, Presto Driver with Okera and RapidSQL

➢ Worked with Hadoop ecosystem on servers including Hadoop, HDFS, Spark, PySpark, Ka\a, HortonWorks, Hive, Cassandra➢ Used Python to make requests from news sources and social media (Facebook, TwiRer, etc.) via REST API➢ Stored unprocessed JSON and HTML files in HDFS data lake➢ Retrieved structured and unstructured data from HDFS and MySQL to Spark to performMapReduce jobs➢ Used Spark Context and Spark Session to process text files by flat mapping, mapping to RDD, and reducing RDD’s by key to idenHfy sentences containing key metric➢ Worked in tandem with analyHcs team to provide querying insights and helped develop new algorithms to map text strings more efficiently➢ Adjusted tables and schema to provide more informaHve data to be used in machine learning models➢ Worked with Apache Spark in Scala for faster large data processing.➢ Created a Ka\a producer to connect to different external sources and bring the data to aKa\a consumer.➢ Maintained and fixed issues related to schema changes in data stream due to upstreamchanges.➢ Created a Ka\a topic for structured streaming to get structured data by schema via CLI.➢ Performed Hive parHHoning, buckeHng, and joins on Hive tables.➢ Performed transformaHons and analysis using Hive

➢ Installed and configured Hadoop HDFS developed mulHple jobs in java for data cleaning and preprocessing.➢ Developed Map/Reduce jobs using PySpark and Python for data transformaHons.➢ Performed various Hadoop operaHons such as Map Reduce, and Hive.➢ Used Sqoop to extract the data back to relaHonal database for business reporHng.➢ Involved in creaHng Hive tables, and loading data and wriHng hive queries➢ Involved in Hadoop Cluster environment administraHon that includes adding and removing cluster nodes, cluster capacity planning, performance tuning, cluster Monitoring.➢ Developed Hive queries and UDFS to analyze/transform the data in HDFS.➢ Designed and Implemented ParHHoning (StaHc, Dynamic), Buckets in HIVE.➢ Used Sqoop to efficiently transfer data between databases and HDFS➢ Debugging and idenHfying issues reported by QA with the Hadoop jobs by configuring tolocal file system.➢ Implemented Flume to import streaming data logs and aggregaHng the data to HDFS.➢ Experienced in running Hadoop streaming jobs to process terabytes data.➢ Involved in evaluaHon and analysis of Hadoop cluster and different big data analyHc toolsincluding HBase database and Sqoop.

➢ Developed a data pipeline used for extracHng historic flood informaHon from online sources via internal API➢ Used Python for web scraping to obtain relevant data regarding most recent floods➢ Stored unprocessed JSON and HTML files in HDFS data lake➢ Retrieved structured and unstructured data from HDFS and MySQL to Spark to preformMapReduce jobs➢ Implemented advanced procedures like text analyHcs and processing using in memorycompuHng capability methods via Apache Spark in Scala➢ Used Spark Context and Spark Session to process text files by flat mapping, mapping to RDD, and reducing RDD’s by key to idenHfy sentences containing valuable informaHon➢ Worked with analyHcs team to provide querying insights and helped develop methods to map informaHve sentences more efficiently➢ Adjusted tables and schema to provide more informaHve data to be used in machine learning models➢ Worked with Apache Spark which provides fast and general engine for large data processing integrated with funcHonal programming language Scala.➢ Created a Ka\a producer to connect to different external sources and bring the data to a Ka\a broker.➢ Handled schema changes in data stream.➢ Created a Ka\a topics for structured streaming to get structured data by schema via CLI.➢ Hive parHHoning, buckeHng, performing joins on Hive tables.➢ Performed transformaHons and analysis using Hive