Recently I was working on a project for the Raspberry Pi that grew a little too big for a single Pi to handle. Instead of moving the app to a system that contains more resources, I thought I would divide up the work load and have individual tasks ran on multiple Raspberry Pi’s instead of one. Since I already have a RPi cluster, I thought this would be a great opportunity to put it to work. Even though there are plenty of different ways to do distributed computing, I chose to go the RPC route. Since I had a really good grasp of what needed to be done and what could be offloaded to other Pi’s for processing, I chose not to use any of the available 3rd party frameworks that are out there on the web. Instead, I chose to roll my own module using the XML RPC libraries that come packaged with Python. Since my project doesn’t require any extra dependencies, it makes it easier to get up & running on the different Pi’s in my cluster. Because I went thru the (easy) work of writing the RPC app, I thought I would share a simpler version of it with all of you in case you find yourself needing to do something similar.
The first thing you are going to need is a worker. This is a simple app that sets up a SimpleXMLRPCServer that listens on a specific address and port. For security purposes (and cleanliness), you can also tell the server which path to listen on as well. In the example below, I have setup a SimpleXMLRPCServer that listens on all addresses on port 8000. It is also setup to listen only for requests to the “/some_path” path. You can change the path to match whatever you want. You’ll also need to remember this path when calling it from the host which we will look at in a moment.
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
rpc_paths = (‘/some_path’,)
server = SimpleXMLRPCServer((”, 8000), requestHandler=RequestHandler)
Now that you have your server setup, you will need to define a class that contains all of the functions you want to expose to your host. These are the functions that will execute the work that is sent to the worker. There are also other ways of registering functions, but I prefer to encapsulate all of my functions in a single class. This makes it nice if I have a lot of functions that I need to expose. I can just move the class into a file of its own which I can reference from the server. For this example, I have created a class called “Functions” that contains the “say_hello” and “say_goodbye” functions. You can name your class and functions anything you want. You can also make these functions do whatever you want. For this example, both of my functions take in a name as an argument and return either “Hello [name]” or “Goodbye [name]“. In the project that inspired this article, I have a lot of mathematical calculations that take place inside these functions. Instead of passing in a name, I pass in a Numpy array. For now, here is what my example class looks like:
def say_hello(self, name):
return ‘Hello %s’ % name
def say_goodbye(self, name):
return ‘Goodbye %s’ % name
Once you have your class and functions defined, you will need to make your server aware of the class and tell the server to startup and listen for connections.
Here is the entirety of the worker app:
from SimpleXMLRPCServer import SimpleXMLRPCServer from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler class RequestHandler(SimpleXMLRPCRequestHandler): rpc_paths = ('/some_path',) server = SimpleXMLRPCServer(('', 8000), requestHandler=RequestHandler) server.register_introspection_functions() class Functions: def say_hello(self, name): return 'Hello %s' % name def say_goodbye(self, name): return 'Goodbye %s' % name server.register_instance(Functions()) server.serve_forever()
Alrighty. Now that you have your worker built, you can fire it up on a second machine (or on the same machine for testing & debugging purposes) where it will listen for requests on port 8000 (unless you specified differently). Since the worker is expected to run on a separate machine, it is also believed that the separate machine will be headless, meaning it will not have a monitor attached. So, I opted to leave out any print, echo, or log statements. Instead, any work that happens on the worker will be returned to the host which is where we will see our results. So, let’s now take a look at the host.
In a separate file, you will need to import the xmlrpclib and build a ServerProxy from it that points to the IP address, port, and path you specified in your worker file. Once you have your server configured, you are free to make requests to the worker. If you want to know what functions are available on the worker for you to use, you can call the “system.listMethods()” function. If you run this using the worker from above, you should see a list of functions that include the “say_hello” and “say_goodbye” functions you specified earlier. You should also see other functions for “system.listMethods” (which we just called), “system.methodHelp“, and “system.methodSignature“. If you want to use the custom functions you created in your worker, you can call those just like you would if you had the code running locally. For the “say_hello” and “say_goodbye” functions, just pass in a name and it will return “Hello [name]” and “Goodbye [name]” respectively.
Here is what the host code looks like if you are running the worker on the same machine as the host. If you want to run the worker somewhere else, just change “localhost” to either the computer name or the IP address of the machine that will be running the worker. Also, make sure that you have the worker running before you run the host. If you want to run the worker on more machines, just duplicate the code below (not receommended) or (recommended) create a list of all the addresses where your worker will be running and iterate thru that list passing each IP address to the server proxy. Keep in mind that you can have different functions for each worker in your network. Workers are not even required to contain the same functions. By calling the “system.listMethods()” function as mentioned above, you can easily identify which worker is capable of performing which tasks.
import xmlrpclib s = xmlrpclib.ServerProxy('http://localhost:8000/some_path') print s.system.listMethods() print s.say_hello('Lucus') print s.say_goodbye('Lucus')
When you run the host app, the functions you have specified will be executed on the machine that is running the worker app. After the work has been completed, any output will be returned to the host which you can do with as you please. In my case, I am passing “Lucus” to both functions which return and print out “Hello Lucus” and “Goodbye Lucus“. For the project that I’m working on that inspired this article, I am crunching a bunch of numbers in Numpy arrays on the worker machines which are returned to my host where they get merged using numpy.vstack(). As the data changes on the host (coming from the user), the Numpy array is divided equally among all of the RPi’s in my cluster using numpy.split(). Each piece of the split array is then sent to a separate RPi where the data is crunched and sent back to the host where the process repeats after every new data received.
In the examples shown here, I have specifically defined the IP addresses of the worker machines. But, I am using DHCP instead of static IP addresses in my RPi cluster. Because of that, I have integrated a UDP broadcast server & client into the mix so that whenever a Raspberry Pi worker is turned on and activated, it will broadcast its IP address onto my network where my host will receive that broadcast and register the broadcasted IP address in a list. The host then uses the size of this list to determine how many parts to breakup the Numpy array into and which IP addresses to send each piece to. This is nice because I can easily add new workers into the network without having to modify any code. Instead, I can just flash the image that already contains my code onto a new SD card, plug it into a new Raspberry Pi, and crank it up. Once it is powered up, it will tell the host that it is online and ready for work. I will show you how to do that in a future article. For now, you have everything you need to do distributed computing using Python and RPC.
PayPal will open in a new tab.