Make everything as simple as possible, but not simpler. -Albert Einstein 

Home Apache Ignite Apache Ignite: Build Cloud Ready Applications Today!
formats

Apache Ignite: Build Cloud Ready Applications Today!

cloud

Summary:  Apache Ignite is a distributed in-memory platform designed for computing and transacting on large-scale data sets. Apache Ignite’s unique zero deployment feature makes it a viable platform for building high performance cloud ready applications. This software middle-ware was originally developed by GridGain Systems, Inc. under the name “GridGain In-Memory Data Fabric”.  In 2014,  GridGain Systems, Inc. contributed over 90% of the core features (ie:  compute grid, data grid, and streaming engine) to the Apache Ignite code base.

In this article, we demonstrate a fictitious consumer survey algorithm that is ‘cloud ready’.  The algorithm will perform a simple score calculation for a sample of 1000 surveys.  This application is extremely scalable, as it demonstrates Apache Ignite’s distributive features.

Customer Satisfaction Survey:

NOTE: The following survey has a score ‘8’. (ie: Q1 + Q2 + Q3 + Q4= survey score)

survey

What does ‘cloud ready’ mean?

All applications fundamentally are comprised of computing instructions and data the instructions utilize to solve a problem.  These applications are high performant when computing instructions and data  are  distributed among available computing resources.

A ‘cloud ready’ application should be able to:

  •       Massively parallelize compute instructions.
  •       Massively parallelize data
  •       Scale automatically as hardware resources are introduced into the network.

All of these features and more are provided by Apache Ignite’s in memory computing platform.

Apache Ignite In Memory Computing Platform Features:

Image by: Apache Ignite: https://ignite.apache.org/index.html

Compute Grid –  Allows  an application to split computing instructions  into multiple parts, and execute them on different computing resources in parallel.

Data Grid – Allows  an application to distribute data across available computing resources.

Advanced Clustering- A cluster is an arrangement of computing resources.  In Apache Ignite, computing resources can dynamically discover each other when new compute nodes are added to the network  without restart.

In this post, we will show how our consumer survey algorithm leverages Apache Ignite middle-ware to exhibit Compute Grid and Advanced Clustering characteristics. Data Grid and other features will be discussed in future posts.

Prerequisites: If you would like to obtain this article’s complete sample, it may be obtained from our GitHub repository.  All samples are Maven based java projects.  In addition, this article will require that you have Apache Ignite installed. Our sample will utilize version 1.9.  Apache Ignite requires that Java 7 SE or later.

Installation Steps:

  1. Download Apache Ignite .zip archive from https://ignite.apache.org/
  2. Unzip ZIP archive into the installation folder in your system.
  3. Create an IGNITE_HOME environment variable to point to Apache Ignite installation path.

Lets Get Started!   ComputeTask  and ComputeJob interfaces are two major abstractions within Apache IgniteComputeTask represents a major unit of work, while  ComputeJob represents a sub task.  Also,  ComputeTask is  responsible for dividing the unit of work into ComputeJobs, mapping the ComputeJobs on to available compute nodes, and aggregating results from ComputeJobs.    Our survey algorithm will make use of ComputeTask and ComputeJobs  to calculate survey scores to return a final average score.   Apache Ignite implements a well known algorithm called MapReduce for parallelizing computations.  Apache Ignite abstracts a compute node with the ClusterNode interface.

The MapReduce algorithm within Apache Ignite can be summarized in the following steps:

Steps:

  1. A task (ComputeTask)  is split into sub-tasks  called ComputeJobs.
  2. Next, ComputeJobs are mapped and shipped to various compute nodes (ClusterNode) for parallel processing.
  3. Upon completion, results from ComputeJobs are returned.
  4. All results from ComputeJobs are aggregated by ComputeTask into a final result.

Play the video below for description of the MapReduce algorithm :

 

Calculating Survey Score: Given  a random population of 1000 Surveys, we use Apache Ignite to calculate an average score.     In order to implement our use case, we will first generate a sample of 1000 Surveys.  The  CalculateSurveyTask and CalculateSurveyJob are central to our algorithm.

Please refer to Listing 1: CalculateSurveyTask or the following explanation:

In  line  40,  Our  CalculateSurveyTask extends ComputeTaskAdapter.

In line  47,  The map() method  accepts a list of  ClusterNodes (ie: eligible JVMs/machines) and random Surveys .

In lines  61-62,   For each Survey, a CalculateSurveyJob is created and assigned to a ClusterNode. Upon completion of the map() method,  Apache Ignite will ship CalculateSurveyJobs to respective ClusterNodes for further processing.

NOTE: In our example, we will select available ClusterNodes randomly.  However, Apache Ignite provides various Load Balancing API’s that can be customized based on use case.

In  line  72,  Apache Ignite will invoke the reduce()  method once all Survey results are calculated by CalculateSurveyJobs. In this routine,  the results are totaled and divided by number of Surveys to compute an average score.

Listing 1: CalculateSurveyTask.java
package techbysample.ignite.sample1;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
import org.apache.ignite.IgniteException;
import org.apache.ignite.cluster.ClusterNode;
import org.apache.ignite.compute.ComputeJob;
import org.apache.ignite.compute.ComputeJobResult;
import org.apache.ignite.compute.ComputeTaskAdapter;
import techbysample.ignite.sample1.model.Survey;


/**
 * 
 * @author TechBySample.com
 *  
 *  
 * Provides an example to demonstrate Apache Ignite's distributive features.
 * 
 * using CalculateSurveyTask and CalculateSurveyJob.
 * 
 * List of surveys are passed as task argument and distributed with CalculateSurveyJob among cluster nodes.
 * 
 * CalculateSurveyJob is responsible for totaling score for individual Surveys.
 *  
 * Each CalculateSurveyJob calculates an individual Survey and returns result to master node where average 
 * survey score is calculated in reduce method.
 *
 * NOTE:
 * 
 * Remote nodes should always be started with special configuration file which
 * enables Peer to Peer class loading.
 * 
 * 
 *  IE: 'ignite.{sh|bat} ../ignite-sample1/config/sample1-ignite.xml'.
 *
 */
public class CalculateSurveyTask extends ComputeTaskAdapter<List<Survey>, Double>  {


	/**
	 *  Map creates CalculateSurveyJobs and randomly assigns to available nodes
	 *  
	 */
	public Map<? extends ComputeJob, ClusterNode> map(List<ClusterNode> nodes, List<Survey> list)
			throws IgniteException {
		
		Map<ComputeJob, ClusterNode> map = new HashMap<>();
		for (Survey survey: list)
		{
			int min=0;
			int max = nodes.size()-1;
			
			//Randomly select a node fron curent list of nodes.
			Random r = new Random();
			
			int nodeToSelect =r.nextInt((max - min) + 1) + min;
			
			CalculateSurveyJob job = new CalculateSurveyJob(survey);
			map.put(job, nodes.get(nodeToSelect));
		}
        return map; 
	}
	
	/**
	 * 
	 *  Reduce sums individual results from ALL surveys to calculate average survey score
	 *  
	 */
	public Double reduce(List<ComputeJobResult> list) throws IgniteException {
		
	   double sum = 0;
	   
	   for (ComputeJobResult result : list)
	   {
		   double avalue = result.getData();
		   sum = sum + avalue;
	   }
	   double result = sum /list.size();
	   return result;
	}
    
	

}

Unit  Testing: A JUnit testcase (CalculateSurveyTest)  is utilized to demonstrate Apache Ignite’s distributive computing behavior.

Please refer to Listing 2: CalculateSurveyTest.java  for the following explanation:

In line 32, the initialize() method is used to start the Apache Ignite runtime.  The Ignition class is a factory creating an Ignite instance.  The Ignite class is the main entry point for accessing all functionality.  An Ignite instance is created by invoking Ignition.start(/config/sample1-config.xml) method.  The ‘/config/sample1-config.xml’ is a Spring based file that allows you to configure Apache Ignite runtime.

In line 33, An IgniteCompute instance is obtained.  The IgniteCompute class will be used to invoke our CalculateSurveyTask.

In lines 46, 50, the testCalculateSurveys() method is used to generate a random population of  Surveys and execute our CalculateSurveyTask.

 

Listing 2: CalculateSurveyTest.java 

package techbysample.ignite.sample1;

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCompute;
import org.apache.ignite.Ignition;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import techbysample.ignite.sample1.model.Question;
import techbysample.ignite.sample1.model.Survey;

/**
 * 
 * @author TechBySample.com
 *
 */

public class CalculateSurveyTest 
{
    private Ignite ignite= null;
    private IgniteCompute compute = null;
    
    @Before
    public void initialize()
    {    
        try {
              ignite = Ignition.start("config/sample1-config.xml");
              compute = ignite.compute();
            }
        catch (Exception e)
        {
            
            System.out.println(e);
        }
        
    }
    
    @Test
     public void testCalculateSurveys()
     {
        List<Survey> list = generateSample();
        
        // Execute task on the cluster and wait for its completion.
        // List of Surveys are passed to CalculateSurveyTask
        double avgScore = compute.execute(CalculateSurveyTask.class,list);

        System.out.println("\n\n");
        System.out.println("############################################################################################################################");
        System.out.println("Number of Surveys is : " + list.size());
        System.out.println("Survey average score is : " + avgScore);
        System.out.println("############################################################################################################################");
        
     }
    
    /**
     * Utility method to generate sample surveys
     * @return
     */
    private List<Survey> generateSample()
    {
      List<Survey> surveySample = new ArrayList();
      
      for (int i=0; i<1000; i++)
      {
          Survey survey = new Survey();
          
          for (Question question: survey.getQuestions())
          {
              Random rn = new Random();
              int answer = rn.nextInt(3);
              question.setResponse(answer);
          }
          
          surveySample.add(survey);
      }
                
      return surveySample;
    }

    /**
     *  Clean up resources
     */
    @After
    public void tearDown()
    {
        ignite=null;
    }
    
    
}

Running  CalculateSurveyTest:

Prior to running the JUnit test, we will start 2 standalone compute nodes (ClusterNodes) to be available for processing our ComputeJobs

NOTE: Its worth mentioning, that these nodes are ‘barebone’ nodes with only the Apache Ignite runtime.  Our classes are NOT pre-installed on each JVM node.   Its NOT necessary as  Apache Ignite takes care of ‘dynamicallly‘ shipping required  classes to remote nodes for processing.

Follow these steps:

1.  Navigate to your <Apache Ignite installation path>/bin  folder  and type  the startup script corresponding to your OS:

     ignite.{sh|bat}   <TechBySample Insatallation path>/techbysample/ignite-sample1/config/sample1-ignite.xml

2.  You should see a display similar to the following:

Node 1: (ie: ClusterNode)

[08:19:08]    __________  ________________
[08:19:08]   /  _/ ___/ |/ /  _/_  __/ __/
[08:19:08]  _/ // (7 7    // /  / / / _/
[08:19:08] /___/\___/_/|_/___/ /_/ /___/
[08:19:08]
[08:19:08] ver. 1.9.0#20170302-sha1:a8169d0a
[08:19:08] 2017 Copyright(C) Apache Software Foundation
[08:19:08]
[08:19:08] Ignite documentation: http://ignite.apache.org
[08:19:08]
[08:19:08] Quiet mode.
:
[08:19:25] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[08:19:25]
[08:19:25] Ignite node started OK (id=d25c048d)
[08:19:25] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=1.0GB]

At this time, the console output on Node 1 depicts that we currently have a single ClusterNode in our cluster.

3.  Repeat step 1 in a another console window.

Once Node 2 is started, both Node 1 and Node 2 will recognize each other and form a cluster.

You should now see the following:

Node 2: (ie: ClusterNode)

[08:26:16]    __________  ________________
[08:26:16]   /  _/ ___/ |/ /  _/_  __/ __/
[08:26:16]  _/ // (7 7    // /  / / / _/
[08:26:16] /___/\___/_/|_/___/ /_/ /___/
[08:26:16]
[08:26:16] ver. 1.9.0#20170302-sha1:a8169d0a
[08:26:16] 2017 Copyright(C) Apache Software Foundation
[08:26:16]
[08:26:16] Ignite documentation: http://ignite.apache.org
[08:26:16]
[08:26:16] Quiet mode.
:
[08:26:34]
[08:26:34] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[08:26:34]
[08:26:34] Ignite node started OK (id=803351fe)
[08:26:34] Topology snapshot [ver=2, servers=2, clients=0, CPUs=4, heap=2.0GB]

4.  From the ‘ignite-sample1′ project directory, type:

mvn -Dtest=CalculateSurveyTest test

On Node 1 ,Node 2, and  JUnit Node (ie: ; where testcase was executed ), you should see  several intermediate results from various CalculateSurveyJobs.  We now have a total of 3 cluster nodes.

You should see the following similar output on all 3 cluster nodes:

Node 1/ Node 2/JUnit Node:

Survey Id: 344 has score of:  7.0
Survey Id: 170 has score of:  5.0
Survey Id: 555 has score of:  6.0
Survey Id: 505 has score of:  5.0
Survey Id: 751 has score of:  4.0
Survey Id: 418 has score of:  5.0
Survey Id: 368 has score of:  3.0
Survey Id: 328 has score of:  5.0

5. Finally,  when CalculateSurveyTest completes, on JUnit Node  you should see a final result:

JUnit Node:

Survey Id: 814 has score of:  1.0
Survey Id: 888 has score of:  3.0
Survey Id: 447 has score of:  4.0
Survey Id: 952 has score of:  6.0
Survey Id: 77 has score of:  6.0
Survey Id: 603 has score of:  3.0
Survey Id: 792 has score of:  4.0
Survey Id: 494 has score of:  6.0
############################################################################################################################
Number of Surveys is : 1000
Survey average score is : 4.112
############################################################################################################################
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.391 sec

Resources:

Official Gridgain website

Apache Ignite website

 
 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn
No Comments  comments 

Leave a Reply

Your email address will not be published. Required fields are marked *


*

© Techbysample.com, all rights reserved.