Muhammad . Email & Phone Number

New York, United States

Programming using Python, Scala along with Hadoop framework utilizing Cloudera Hadoop Ecosystem projects (HDFS, Spark, Sqoop, Hive, HBase, Oozie, Impala, and Zookeeper etc.)Involved in developing spark applications using Scala, Python for Data transformations, cleansing as well as validation using Spark API.Written Scala and python script notebooks for Azure Data bricks transformation task.Involved in converting Hive queries into Spark transformations using Spark RDDs and Scala.Analyzed the SQL scripts and designed the solution to implement using Scala.Good Understanding of Data ingestion, Airflow Operators for Data Orchestration and other related python libraries.Worked with dbt to define and execute data transformations and models in SQL.Monitor and manage data replication jobs through the Airbyte interface.Worked with the Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLib, Data Frame, Pair RDD's, Spark YARNUsed Spark Streaming APIs to perform transformations and actions on the fly for building common learner data model which gets the data from Kafka in near real time and persist it to Cassandra.