OP @ (v0.3)

Distribute code on a Native Big Data network is easy with the op language and the @ functions. The general API is:

@( server, code, callback code )

Where:

• server is :
  • A string: the remote server where execute the code.
• code is:
  • The code to be executed remotely.
• callback is optional and must be the code for mixing the remote scope with the local one. This code will be executed in a monothread safe environment to prevent race problems.

The @ function will return the value returned by the remote execution.

Maybe the API description is hard to be understood without examples. At this point we will study how to distribute algorithms with Native Big Data with an example.

Supose we want a function than calculates a sample of 1000 random values with a normal distribution normal(0,x) where x is a parameter. We want the values and the sum of the values.

local version

(x) => {
	var s = [];
	var t =0;
	for (var i=0;i<100;i++)
	{
		var k = RND.normal(0,x);
		s[]=k;
		t+= k
	}; [t,s]
}

remote version without concurrency

The following script will execute our code in other server.

(x) => {
	@( "urp:server:/s1", 
		var s = [];
		var t = 0;
		for (var i=0;i<100;i++)
		{

			var k = RND.gauss(0,x);
			s[]=k;
			t+= k
		}; [t,s]
	)
}

The @ function works in this case with two parameters. The first one is one string with the urp of the server than must execute the code. The secons parameter is the code to be executed. The @ command do the following actions:

• Send the scope and the code of the script to the remote server.
• The remote server executes the code on a cloned scope.
• The remote server returns the result of the op and the modified scope.
• The local server updates the scope and returns the result.

remote version with concurrency with @ function

In this case we want to execute our code in several servers:

(x) => {
	var result = [ 0, [] ];
	var servers = ["urp:server:/s1","urp:server:/s2"];
	pfor(val s=0;s<2;s++)
	{
		@(servers[s],
			var s = [];
			var t = 0;			
			for (var i=0;i<50;i++)
			{
				var k = RND.gauss(0,x);
				s[]=k;
				t+= k
			}; [t,s]
			,
			result[0] += _[0];
			result[1].append(_[1]);
		)
	}
	[t,s]
}

In this case the @ function has 3 parameters.

• The first one is the server where the code must be executed.
• The second one is the code to be executed.
• The 3st is a callback function than says how to mix the remote scope with the local one. This function has as parameter the result of the script.

In this case, the @ command do the following actions:

• Send the scope and the code of the script to the remote server.
• The remote server executes the code on a cloned scope.
• The remote server returns the result of the code and the modified scope.
• The local server updates the result in the callback function.

We can do the same using closures:

(x) => {
	var result = [ 0, [] ];
	var servers = ["urp:server:/s1","urp:server:/s2"];
	pfor(val s=0;s<2;s++)
	{
		@(servers[s],
			for (var i=0;i<50;i++)
			{
				var k = RND.gauss(0,x);
				result[0]   +=k;
				result[1][] = k;
			}; null
			,
			result[0] += _scope.result[0];
			result[1].append(_scope.result[1]);
		)
	}
	[t,s]
}

In this case each server modifies its cloned scope, (the result variable). In the callback code, the local var result must be updated using the remote var result. The remote var is accesible in the local server because the "_scope" variable has a remote modified scope. Accessing variables in the remote scope is as easy as reading the variables from the scope parameter like reading from a hash.

remote version with concurrency with @for loop

The easiest way to launch an op code in several servers is with a @for loop.

The @for loop has this API:

@for( init code ; conditional code ; incremental code ; servers , callback code ) { loop_code }

where:

• init code, conditional code and incremental code have the same meaning as a standard for loop.
• servers is an array of servers where distribute the code.
• callback code is a code similar to the callback function in the @() function. The callback code has two "special" variables:
  • _  : The result of the remote code executed.
  • _scope: The remote scope.

Example:

(x) => {
	var result = [ 0, [] ];
	var servers = ["urp:server:/s1","urp:server:/s2"];
	@for(val s=0;s<2;s++;servers, (result[0] += _[0]; result[1].append(_[1])) )
	{
		var r2 = [ 0, [] ];
		for (var i=0;i<50;i++)
		{
			var k = RND.gauss(0,x);
			r2[0]   +=k;
			r2[1][] = k;
		}; r2;
	}
}

Pay atention that the callback code must be between '(' and ')'. Thsi is because the @for loop has ';' as parameter separator and the callback code uses ';' as sequence operator.

And, if we want to return the value in the closure:

(x) => {
	var result = [ 0, [] ];
	var servers = ["urp:server:/s1","urp:server:/s2"];
	@for(val s=0;s<2;s++;servers; (result[0] += _scope.result[0]; result[1].append(_scope.result[1]) ) )
	{
		var r2 = [ 0, [] ];
		for (var i=0;i<50;i++)
		{
			var k = RND.gauss(0,x);
			r2[0]   +=k;
			r2[1][] = k;
		}; r2;
	}
}