Developed complete Big Data processing data pipelines using Spark Scala Designed, developed, implemented, testing, and maintenance of data ingestion and integration of ETL pipelinesApplied Git for version control.

Developed Spark code using Scala and Spark-SQL for faster processing and testing.Utilized Spark SQL with DataFrame API to provide efficiently structured data processing.Implemented Spark RDD transformations to Map business analysis and apply actions on top of transformations.Stored the click stream data placed in HDFS and Spark for batch processing.Responsible to store batch processed data into Redshift.Performed storing and querying user session information in HDFS as Hive external tables from batch processing.Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.Designed, developed, and maintained data ETL pipelines by Spark SQL and Scala to ingest million rows of raw data from different sources to AWS Redshift and S3.Deploy services on AWS and utilized Lambda function to trigger the data pipelines.

• Migrated difference formats of data from HDFS to Spark DataFrame API by ETL pipelines.• Applied Spark SQL with DataFrame API to provide efficiently structured data processing.• Used Spark to count the number of transactions from files containing Bitcoin Blockchain data. • Converted raw data with sequence data format, such as ORC and Parquet to reduce data processing time and increase data transferring efficiency through the network.• Applied Hive to analyze historical Bitcoin and Altcoin Blockchain data.• Scheduled and managed the whole workflow of data pipelines by AWS Data Pipeline.

Imported and exported data between RDBMS and HDFS by Sqoop script. Defined gross operation income as a fact metric to measure business performance on real estate market and build the database.Load the data as Spark DataFrame format and analyzed by Scala and Spark SQL for scalable storage and fast query. Created and normalized a star-schema database by data pipelines based on the defined metric, and store structured data for business analyze. Present results in layman term by Tableau dashboards based on ad-hoc data studies.

Acquired and manipulated financial statement data, ratios, bond credit rating and CDS for target industry companies.Created, inserted, and cleaned data into MySQL database for backend operation. Applied Sqoop commands to import, transfer, and store MYSQL database to HDFS. Applied MapReduce in Hadoop to split word into vectors and counted the frequency of the word in financial news, and built sentimental model using NLP NLTK.

Involved ETL processes including data processing and data storage.Applied Spark using Scala to do the data batch processing and store the output in HBase for scalable storage and fast query.Designed and created of Hive tables and worked on various performance optimizations like Partition, Bucketing in Hive.Implemented Hive and analyzed large data sets by running HiveQL to achieve comprehensive data analysis.Migrated of MapReduce jobs and Hive queries into Spark transformations and actions to improve the performance.Utilized Sqoop to import and output data between Oracle database and HDFS.Configure the Sqoop incremental import job for importing the updated input data.Implemented request, beautiful soup to scrape real time data through parsing yahoo finance HTML.