Mohan G Email & Phone Number

New York, New York, Us

• Developed applications using spark to implement various aggregation and transformation functions of Spark RDD and Spark SQL. • Worked on DB2 for SQL connection to Spark Scala code to Select, Insert, and Update data into DB. • Used Broadcast Join in SPARK for making smaller datasets to large datasets without shuffling of data across nodes. • Used Oozie Scheduler systems to automate the pipeline workflow and orchestrate the Spark jobs. • Created Spark Streaming jobs using Python to read messages from Kafka & download JSON files from AWS S3 buckets • Used Spark Streaming to receive real-time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra• Implemented Spark in EMR for processing Big Data across our One Lake in AWS System • Developed AWS strategy, planning, and configuration of S3, Security groups, IAM, EC2, EMR and Redshift • Developed Spark/Scala, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources. • Data sources are extracted, transformed and loaded to generate CSV data files with Python programming and SQL queries. • Developed data processing applications in Scala using SparkRDD as well as Dataframes using SparkSQL APIs. Environment: Spark, Scala, AWS, Python, Spark SQL, Redshift, PgSQL, Data bricks, Jupiter, Kafka

New York, Ny, Us

• Work on requirements gathering, analysis and designing of the systems.• Developed Spark programs using Scala to compare the performance of Spark with Hive andSparkSQL.• Used Spark API over Hadoop YARN as an execution engine for data analytics using Hive.• Exported the analyzed data to the relational databases using Sqoop to further visualize andgenerate reports for the BI team.• Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's• Used AWS services like EC2 and S3 for small data sets processing and storage• Implemented Nifi flow topologies to perform cleansing operations before moving data intoHDFS.• Worked on different file formats (ORCFILE, Parquet, Avro) and different Compression Codecs(GZIP, SNAPPY, LZO).• Created applications using Kafka, which monitors consumer lag within Apache Kafka clusters.• Worked on importing and exporting data into HDFS and Hive using Sqoop, built analytics onHive tables using Hive Context in spark Jobs.• Developed workflow in Oozie to automate the tasks of loading the data into HDFS.• Worked in an Agile environment using Scrum methodology.• Developed spark streaming application to consume JSON messages from Kafka and performtransformations.• Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.• Implemented Spark using Scala and SparkSql for faster testing and processing of data.• Involved in developing a MapReduce framework that filters bad and unnecessary records.Environment: Hadoop, Hive, MapReduce, Sqoop, Kafka, Spark, Yarn, Pig, PySpark, Cassandra,Oozie, Nifi, Solr, Shell Scripting, Hbase, Scala, AWS, Maven, Java, JUnit, agile methodologies, Hortonworks, Soap, Python, Teradata, MySQL.

• Work on requirements gathering, analysis and designing of the systems. • Actively involved in designing Hadoop ecosystem pipeline. • Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data. • Involved in designing Kafka for multi data center cluster and monitoring it. • Responsible for importing real time data to pull the data from sources to Kafka clusters.• Work on requirements gathering, analysis and designing of the systems. • Actively involved in designing Hadoop ecosystem pipeline. • Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data. • Involved in designing Kafka for multi data center cluster and monitoring it. • Responsible for importing real time data to pull the data from sources to Kafka clusters. • Implemented Spark RDD transformations to Map business analysis and apply actions on top of transformations. • Involved in using Spark API over Hadoop YARN as execution engine for data analytics using Hive and submitted the data to BI team for generating reports, after the processing and analyzing of data in Spark SQL. • Performed SQL Joins among Hive tables to get input for Spark batch process. • Used Sqoop to import functionality for loading Historical data present in RDBMS to HDFS • Implemented ELK (Elastic Search, Log stash, Kibana) stack to collect and analyze the logs produced by the spark cluster. • Developed Python script for start a job and end a job smoothly for a UC4 workflow • Developed shell scripts to periodically perform incremental import of data from third party API to Amazon AWS • Worked extensively with importing metadata into Hive using Scala and migrated existing tables and applications to work on Hive and AWS cloud. Environment: Hadoop, HDFS, Hive, Python, Hbase, Nifi, Spark, MYSQL, Oracle 12c, Linux, Hortonworks, Oozie, MapReduce, Sqoop, Shell Scripting, Apache Kafka, Scala, AWS.

Hyderabad, Telangana, In

• Analyzing Functional Specifications Based on Project Requirement. • Ingested data from various data sources into Hadoop HDFS/Hive Tables using SQOOP, Flume, Kafka. • Extended Hive core functionality by writing custom UDFs using Java. • Developing Hive Queries for the user requirement. • Worked on multiple POCs in Implementing Data Lake for Multiple Data Sources ranging from TeamCenter, SAP, Workday, Machine logs. • Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data. • Worked on MS Sql Server PDW migration for MSBI warehouse. • Planning, scheduling and implementing Oracle to MS SQL server migrations for AMAT in house applications and tools. • Worked on Solr Search Engine to index incident reports data and developed dash boards in Banana Reporting tool. • Integrated Tableau with Hadoop data source for building dashboard to provide various insights on sales of the organization. • Worked on Spark in building BI reports using Tableau. Tableau was integrated with Spark using Spark-SQL. • Developed Spark jobs using Scala and Python on top of Yarn/MRv2 for interactive and Batch Analysis. • Created multi-node Hadoop and Spark clusters in AWS instances to generate terabytes of data and stored it in AWS HDFS. • Developed workflows in Live Compare to Analyze SAP Data and Reporting. • Worked on Java development and support and tools support for in-house applications. • Participated in daily scrum meetings and iterative development. • Search functionality for searching through millions of files of logistics groups.

Bangalore, Karnataka, In

• Gathered requirements from Business and documented for project development.• Coordinated design reviews, ETL code reviews with teammates.• Developed mappings using Informatica to load data from sources such as Relational tables, Sequential files into the target system.• Extensively worked with Informatica transformations.• Created datamaps in Informatica to extract data from Sequential files.• Extensively worked on UNIX Shell Scripting for file transfer and error logging.• Scheduled processes in ESP Job Scheduler.• Performed Unit, Integration and System testing of various jobs.Environment: Informatica Power Center 8.6, Oracle 10g, SQL Server 2005, UNIX Shell Scripting, ESP job scheduler