Bigdata-Hadoop , Spark Developer

Clients:-Standard Chartered Bank Malaysia - Bigdata - Hortonworks - Hadoop, HBase, Spark-Streaming, CassandraToyota - Bigdata - Cloudera - Hadoop, SparkRoles and Responsibilities:-• Designed Kafka properties , Spark Streaming design, HBase and Cassandra table schemas.• Used Spark Streaming for consuming data from Kafka topics and populate the raw data in HDFS / HBase, HBase rowkey Design.• Implemented Zero data loss using reading/updating offsets in… Show more Clients:-Standard Chartered Bank Malaysia - Bigdata - Hortonworks - Hadoop, HBase, Spark-Streaming, CassandraToyota - Bigdata - Cloudera - Hadoop, SparkRoles and Responsibilities:-• Designed Kafka properties , Spark Streaming design, HBase and Cassandra table schemas.• Used Spark Streaming for consuming data from Kafka topics and populate the raw data in HDFS / HBase, HBase rowkey Design.• Implemented Zero data loss using reading/updating offsets in Zookeeper with the combination of Direct Streaming approach.• The system is running on production clusters 24x7 and processing almost around 4 Million of data every day.• Implemented DQ validation for streaming data and persist invalid records in HDFS location• Read the data from HBase using HortonWorks HBase connector and applied validation, business rules and transformations on top of the raw data and populated the validated data in Cassandra.• Used Spark SQL DataFrame API and Aggregate functions extensively for transforming the raw data to meaningful data for Visualization.• Implemented various optimizations in spark application to improve the performance – Tuning memory, executors and cores.• Scheduled spark action through Oozie• writing shell script for maintain Yarn jobs Show less

DataDotz, based at Chennai, started with a small team of five vibrant professionals, is now on its lead providing comprehensive Big Data technology solutions to leverage and transform your traditional business to analytics. Established in 2013 in a challenging environment, DataDotz has been making significant impact in this industry. Our capabilities in Big Data technologies such as Apache Hadoop, Apache Spark , Cassandra comprise solid solutions with powerful frameworks that address every… Show more DataDotz, based at Chennai, started with a small team of five vibrant professionals, is now on its lead providing comprehensive Big Data technology solutions to leverage and transform your traditional business to analytics. Established in 2013 in a challenging environment, DataDotz has been making significant impact in this industry. Our capabilities in Big Data technologies such as Apache Hadoop, Apache Spark , Cassandra comprise solid solutions with powerful frameworks that address every functional component, including Data management, Data Provisioning, Data consumption.With the constant growth in our Big Data industry, DataDotz is now the trusted partner to our increasing number of clients across the globe. We work towards providing superior experience to our clients through the combination of intellectual curiosity, friendly work style, business insight and strong partnership with our clients. Our in depth technical expertise in Cloud based Technologies migrate risk involved and reduces your overall investment costs.Our Team embraces most of the open sources technologies related to Data such Apache Hadoop, Apache Spark, Apache Storm and other related Big Data Technologies. We also leverage most of the cloud technologies for our work such as Amazon Web Services, Google Compute Engine. Our members are regular contributors to Chennai Hadoop Users Group. We are also moving towards Data Science Specialization solving the unanswered questions of the business Show less