Moving all ports in a buildout up an offset amount

Use z3c.recipe.filetemplate with its interpreted-option setting. This will give you the opportunity to do arithmetic on the port numbers for use both inside the buildout and in rendered files.

I have a relatively complex buildout that builds a combination of servers written in Python, C, C++ and Node.js, and then runs all of them with supervisord.

The servers need to know about each other's host names, ports, user names and passwords. Buildout works well for this, just define all the pertinent data inside a buildout config file and supply the config data as command line options in supervisor for those servers who can parse that,  and write out configuration files for the servers that prefer to get their settings from config files. In this way all settings are inside of buildout and by changing one value, i.e. the host name of the PostgreSQL server, all servers now know about this change and everything continues to work.

But a new challenge has occurred: I need to run more than one instance of this buildout on the same machine. I now need a port offset, that moves all the half-dozen or so ports up by a specific amount, so that the second buildout doesn't try to put its servers on the same ports as the first buildout.

In order to do this I switched out the templating system I was using to z3c.recipe.filetemplate . z3c.recipe.filetemplate can do calculations on settings, if you put the settings you need to calculate under a special "interpreted-options" category. Well actually it is python code. It looks something like this:

[buildout]
parts = config


[config]
recipe=z3c.recipe.filetemplate
source-directory = etc/templates

files = etc/config/test.txt
port_offset = 100
myserver_port_base = 1024
interpreted-options =
                     myserver_port
port_offset = 10
myserver_port = str(int(options['myserver_port_base']) + int(options['port_offset']))

In the above example, port_base and port_offset are defined in the usual buildout way. However myserver_port is declared to be an interpreted option, and that means that when it gets defined, python expressions are valid inside of its definition. options refers to the name space of the buildout part.

myserver_port is now a defined variable in the buildout, reachable as ${myserver_port} in the templates but also available as ${config:myserver_port} from anywhere else in the buildout (such as in a supervisord section).

source-directory above refers to where the template for test.txt is to be found. If test.txt is to be written to etc/config/test.txt, then the template should be:

etc/templates/etc/config/test.txt.in

Note the repetition of the path etc/config inside of the source directory.

z3c.recipe.filetemplate can handle multiple input and output files in one fell swoop:

[buildout]
parts = config


[config]
recipe=z3c.recipe.filetemplate
source-directory = etc/templates

files = 
        etc/config/test.txt
        conf/another_config.yaml
        etc/config/and_one_more.cfg
        mymodules/apythonmodule.py
port_base = 1065
port_offset = 100
myserver_port_base = 1024
interpreted-options =
                     myserver_port
port_offset = 10
myserver_port = str(int(options['myserver_port_base']) + int(options['port_offset']))

 

In the above example 4 files are defined an hence we get 4 files created. Their respective templates must be under etc/templates with their respective paths and have the suffix ".in".