Abdul Rahman Mohammed Email & Phone Number

Chicago, Illinois, Us

• Implemented Spark Scripts using Scala, Spark-SQL to access hive tables into spark for faster processing of data. • Great hands-on experience with Pyspark for using Spark libraries by using python scripting for data analysis.• Work with Avro and Parquet files and convert the data from either format Parsed Semi Structured JSON data and converted to Parquet using Data Frames in PySpark.• Develop Scala scripts, UDFs using both Data frames/SQL and Data sets in Spark for Data Aggregation. • Work on MongoDB database concepts such as locking, transactions, indexes, replication, and schema design.• Develop workflows using Oozie to automate the tasks of loading the data into HDFS & pre-processing with Python.• Use Spark SQL on data frames to access hive tables into spark for faster processing of data.• Develop applications in Spark with Python to transform data.• Migrate Spark applications to Snowpark using Python• Create task, streams, stages, stored-procedures in Snowflake to ingest data• Create Airflow DAGs to monitor and trigger alerts for invalid data• Create ETLs in PySpark to ingest data from heterogenous sources

Detroit, Michigan, Us

• Implemented Spark Scripts using Python, Spark-SQL to access hive tables into spark for faster processing of data. • Performed ad-hoc queries on structured data using Hive QL and used Partition, bucketing techniques and joins with Hive for faster data access.• Develop Hive queries on external tables in order to perform various analysis.• Used HUE for running Hive queries. Created partitions according to data using Hive to improve performance. • Developed Spark Applications by using Pyspark and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.• Great hands-on experience with Pyspark for using Spark libraries by using python scripting for data analysis.• Designed and developed jobs to validate the data post migration such as reporting fields from source and designation systems using Spark SQL RDDs and Data Frames/Datasets.• Used Spark Data Frames Operations to perform required Validations in the data and to perform analytics on the Hive data in AWS EMR.• Used Different Spark Modules like Spark core, Spark SQL, Spark Streaming, Spark Data sets and Data frames to process high volumes of data.• Worked on Spark Streaming and Structured Spark streaming using Apache Kafka for real time data processing. • Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications.• Experienced in running query using Impala and used BI tools to run ad-hoc queries directly on Hadoop.• Involved in reading uncompressed data formats like Gzip, Avro, Parquet and compressed the same according to the business logic by writing generic code.• Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS. • Involved in Agile methodologies, daily Scrum meetings.

Stamford, Connecticut, Us

• Worked with Hortonworks distribution of Hadoop for setting up the cluster and monitored it using Ambari.• Created ODBC connection through Sqoop between Hortonworks and My-SQL .• Implemented Spark Scripts using Scala, Spark-SQL to access hive tables into spark for faster processing of data. • Great hands on experience with Pyspark for using Spark libraries by using python scripting for data analysis.• Extensively worked with Avro and Parquet files and converted the data from either format Parsed Semi Structured JSON data and converted to Parquet using Data Frames in PySpark.• Extensively worked on Hive, Pig, Map Reduce, Sqoop, Oozie in an optimized way of distributed processing.• Worked extensively on Hive to create, alter and drop tables and involved in writing hive queries. • Developed Scala scripts, UDFs using both Data frames/SQL and Data sets in Spark for Data Aggregation. • Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive.• Injecting large volumes of data from heterogenous sources into the Data Lake.

Foster City, California, Us

• Performed ad-hoc queries on structured data using Hive QL and used Partition, bucketing techniques and joins with Hive for faster data access.• Importing and exporting data into HDFS and HIVE using Sqoop. • Developed Spark Applications by using Scala and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.• Used Spark SQL on data frames to access hive tables into spark for faster processing of data• Designed and developed jobs to validate the data post migration such as reporting fields from source and designation systems using Spark SQL RDDs and Data Frames/Datasets.• Used Spark Data Frame API to process Structured and Semi Structured files and load them into S3 Bucket.• Used Spark Data Frames Operations to perform required Validations in the data and to perform analytics on the Hive data.• Used Different Spark Modules like Spark core, Spark SQL, Spark Streaming, Spark Data sets and Data frames.• Worked on Spark Streaming and Structured Spark streaming using Apache Kafka for real time data processing. • Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications.• Experience in creating Impala views on hive tables for fast access to data.• Experienced in running query using Impala and used BI tools to run ad-hoc queries directly on Hadoop.• Involved in reading uncompressed data formats like Gzip, Avro, Parquet and compressed the same according to the business logic by writing generic code.• Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS. • Developed workflows using Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.• Used the JSON and XML SerDe's for serialization and de-serialization to load JSON and XML data into HIVE tables.

Seattle, Wa, Us

• Follow complete Agile methodology starting from the Product backlog, Sprint Backlog, Sprint planning, User stories etc.• Participate in gathering the requirements with S&P business users and converting them into User Stories. • Export analysed data to relational databases using Sqoop • Use Teradata Fast Export and Parallel Transporter utilities and Sqoop to extract data and load to Hadoop• Write complex SQL queries including inline queries and sub queries for faster data retrieval from multiple tables• Develop automated tasks to perform aggregations and manipulations on the data• Extract, transform and load data into SQL Server tables from Teradata• Rebuild Indexes and Tables as part of Performance Tuning Exercise• Use SSIS and T-SQL stored procedures to transfer data from OLTP databases to staging area and finally transfer into data warehouse• Provide on-call Production support for any issues• Created ETL packages using Heterogeneous data sources (DB2, ORACLE and Flat Files etc.) and then loaded the data into destination tables by performing different kinds of transformations• Load data in to AWS S3 bucket using ETLs• Perform tuning and optimization of queries for reports that take longer execution time using SQLProfiler, index tuning wizard and SQL Query Analyzer• Develop multiple MapReduce jobs for data cleaning and pre-processing• Read data from S3 and process it using SparkSQL.• Supported MapReduce Programs those are running on the cluster

New York, Ny, Us

• Provided immediate response to Production System calls and bug fixing with immediate effect• Setup of the planning application and all its components• Responsible for dimension building and outline maintenance using planning web interface and reflecting changes in Essbase• Loaded metadata through HAL jobs• Created web forms according to the requirement• Created filters, groups and users• Provided access on the applications to the business users according to their business in shared services• Checked the drill through reports on daily basis• Resolved the Data-Reconciliation issues and other issues (mostly security) on UAT phase• Developed all the required detailed project documentation of all the tasks performed and the issues faced which will be used for future reference• Assigned (dimension and data form) Access rights to users and user groups• Developed calc. scripts for final administration tasks such as Full aggregations, full currency conversion, and data copy from one scenario to another• Developed Business Rules with Run Time prompts for end user calculations by Entity for driver calculations, currency conversion at their entity, aggregation for their entity etc• Developed Essbase components Load rules for actual and other data loads, setting up substitution variables• Used various types of Calculations such as Dynamic calc. Two-Pass Calculations, Intelligent calculations for Outline Calculations• Performed optimization for various data load rules and calc. scripts• Trained business analysts on retrieving data from Essbase cubes by using Excel spreadsheet add-in and some of the planning features such as alias table and Smart view offline process• Provided 24/7 technical support to all business units, offered suggestions and proposed solutions to resolve issues• Monitored the automated data-load and aggregations using Appworx

Master'S Degree, Computer Science

Education record

Icse, Mbipc