James Dull Email & Phone Number

Philadelphia, Pa, Us

• Understanding the business functionality & Analysis of business logic for the ETL process• Developed Spark Streaming job to consume the data from the Kafka topic of different source systems and push the data into HDFS locations• Responsible for migrating data from legacy system to AWS Cloud which was running on XMBI Data Lake• Responsible for creating data warehouse and managing entire ETL Data Pipeline from planning to deployment Using Big Data Technologies (Hive, Sqoop, HDFS, Impala, Hue)• Hands on experience of creating tables in Hive with performance optimization using bucketing & partitioning • Creating ETL job’s and extracting data from Data Lake and loaded data into Datawarehouse as per the Business requirement.• Wrote Kafka producers to stream the data from external rest APIs to Kafka topics from legacy system to downstream the data.• Used Kafka and Kinesis for Real time streaming data from Amdocs and ingested into Data Lake.• Built real time data pipelines by developing Kafka producers and Spark streaming applications for consuming the source data as part of Xfinity Mobile• Created lambda function by using python and Downstream the data by streaming from AMDOCS.• Using CloudWatch metrics to monitor the logs for the job’s success and failure.• Each consumer will be consuming the data from Data Lake which was loaded by XMBI.• Involved in extracting the data from different data sources like FLAT files, ORC, PARQUET, CSV files • As part of development has done the cloud POC on VERTICA and ingested files from SFTP server to Data Lake of Hive External Tables.

Mclean, Va, Us

• Worked on Real Time Streaming data of various source system (EROMINUS, IVR, FINANCIAL,COMPLIANTS,FINANCE) OF VARIOUS STREAMS AND PROCESSING THE DATA FROM SOURCE AWS S3 TO TARGET AWS SNOWFALKE• Developed lambda function for data migration services for data to load from AWS S3 TO SNOWFALKE• Created lambda function for both Historical and Delta load • Designed Kafka producer client using Confluent Kafka and produced events into Kafka topic.• Subscribing the Kafka topic with Kafka consumer client and process the events in real time using spark.• Closely working with Machine learning team and Developed Algorithms using Python for data migration by using lambda functions.• Used Kafka producer and consumer and established data into snowflake database.• Post this step to sdp stream compatible avro message is constructed using sdp_sdk and published to SDP.• Successfully Generated consumer group lags from Kafka using their API• Ability to design high availability applications on AWS across availability zones and availability regions.• Involved in performance tuning on Kinesis Auto scaling by SCALE-IN & SCALE-OUT based on the input records by adding shards to the streams.

Indianapolis, Indiana, Us

• Done data ingestion by using Sqoop from various source like Oracle, SQL Server and BDF data (big data fabric)• Created Hive external Tables and loaded data in the where hive is the staging area in scope of the project.• Build some complex hive queries on top of the business logic and applied partitions based of load_ingstn_id Created a new table and dump the data into the external table for member details.• Very Strong hands on experience wring shell scripting for data ingestion and creating tables and loading data• Good hands on experience on Scheduling jobs through control-m on initial stage• Developed Spark scripts by using python as per the requirement. Ensuring defect less delivery by delivering the code as per client requirement within timelines. • Very Strong Hands on experience on writing CDC(Change Data Capture) in spark by using Scala Programing for incremental loads/Delta Records• Where CDC will capture from Hive Staging for newly updated and Inserted records and drop duplicated records and load the new record into Mongo Collections for all 22 Tables• And Created View’s in Mongo as per the business logic for New Jersey member summary Market by using hive table.