Spark Engineer with Kafka and Hadoop Expertise for a Top Ten E-commerce Retailer

Project Description

We are one of the top 10 e-commerce retailers in the world and are looking for a Spark Engineer with Kafka and Hadoop experience.  Here is our system topology.

• Real time application events (Web requests metrics - CPU, Memory, Response Times, Transaction Count) are published to Kafka messaging queue.
• Storm acts as a consumer of those real time events from Kafka queue.
• The Storm topology processes the data and writes it to HBase.
• REST API(Jersey).Service Layer running on a separate JBoss VM is being used to query raw data from HBase and render it to the user interface.
• Hortonworks distribution – HDP 2.4, Spark 1.6
• Strong understanding of HBase architecutre, capacity sizing

We need to have a real time / dynamic aggregation on the raw data available in HBase.

We are looking for an expert who has experience developing Spark data streaming in near realtime (< 5 seconds) and have a good understanding of Hadoop architecture and its components. Please respond with your previous experience developing realtime systems using Spark, Kafka and Hadoop.

This is a two-week project, involves working remotely and would need to start as soon as possible.