Multi-Agent Distributed System for monitoring and control of industrial processes

IMPORTANT: Look at https://mads-net.github.io for in-depth information, help, and guides.

The architecture is based on a distributed network of processes, called agents, connected via ZeroMQ protocol as publishers-subscribers, using a central broker for network discovery and settings sharing.

Agents can be implemented as monolithic processes, i.e. a single executable that performs a specific task, or as plugins, i.e. shared libraries that can be loaded at runtime by a general purpose executable.

Plugins can be of three different types:

• Sources: they grab data from the field and publish them to the broker
• Filters: they receive data from other sources or filters, process them, and publish the results to the broker
• Sinks: they receive data from the broker and process them locally (visualizing, logging, bridging to other networks, etc.)

There are three corresponding general purpose, plugin-based agents, named source, filter, and sink.

Some specific monolithic agents are also available for common tasks, such as logger, which saves incoming data to a MongoDB database.

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
flowchart LR
    broker["Broker"]
    source["Source"]
    filter["Filter"]
    sink["Sink"]
    monolithic["Mon. source"]
    source_plugin["Source Plugin"]
    filter_plugin["Filter Plugin"]
    sink_plugin["Sink Plugin"]
    source_plugin <--> source
    filter_plugin <--> filter
    sink_plugin <--> sink
    source --> broker
    filter --> broker
    broker --> sink
    monolithic --> broker
    broker --> filter
    sink --> field
    broker --> logger
    logger --> mongodb[(MongoDB)]

The settings are stored in an INI file according to the TOML format. By defaut, the setting file is read by the broker (which must be the first process to start), and then passed to the other processes via a ZeroMQ REQ/REP socket. This way, all agents share the same settings even if they run on different filesystems.

The settings file is divided into sections, one for each agent. The section name is the name of the executable file, without the path and the extension. For example, the settings for the logger agent are stored in a section named [logger].

Agents that are plugin-based (i.e. source, filter, and sink) have their settings stored in a section named after the plugin file name, without the path and the extension. For example, the settings for the source agent loading the plugin my_plugin.plugin are stored in a section named [my_plugin].

NOTE: it is possible to have multiple plugin agents with different settings, but sharing the same plugin file. This is useful when the same plugin is used in different contexts, that require different settings. To do that, the mads-source, mads-filter, and mads-sink executables have the command line option -n|--name, which sets the name of the plugin to be loaded. The name is used to select the proper settings section in the INI file. FOr example, the agent loaded as mads source clock.plugin -n clock_slow will load settings from the section [clock_slow] in the INI file. If that option is missing, the name of the plugin is assumed to be the same as the plugin file name.

An [agents] section is also present, which contains the settings that are meant to be shared among all agents.

The logger agent can log all the messages to a plain file or to a MongoDB database (or both). MongoDB is the preferred route: file output is just for debugging purposes.

The suggested way to install MongoDB is via Docker. The following command will install the latest version of MongoDB in a Docker container, and will expose the default port (27017) to the host machine.

docker run --name mads-mongo --restart unless-stopped -v ${PWD}/db:/data/db -p27017:27017 -d mongo

The logger agent connects to a MongoDB instance. The URI of the database (possibly on a different machine) is specified in the mads.ini file, as well as the name of the database to be used.

For each and every message received by the logger agent, it saves a new document on a table named as the topic of the message. The document contains the following fields:

• _id: a unique identifier for the document, automatically generated by MongoDB
• timestamp: the timestamp of the message
• message: the message itself, as a JSON object
• error: error message when the JSON is not valid

• We use C++17 standard.
• Classes and namespaces shall be named in CamelCase, variables and functions in snake_case, member variables shall have a leading underscore.
• Globals are discouraged, but if used, they shall be named in __snake_case__ (two leading and trailing underscore).
• Preprocessor macros shall be named in SNAKE_CASE.
• Header files shall have .hpp extension, source files .cpp extension.
• All classes and library files shall go in src dir. These are compiled as a single static library, to be included from executables
• All source files with main() function shall go in src/main dir.
• clang-format shall be used to enforce a common coding style using LLVM style.

To implement a monolithic agent two files have to be created:

• src/new_agent.hpp. This is a class that inherits from Mads::Agent and should:
  • override the method Mads::Agent::load_settings(), which has to load the customized settings from the INI file
  • implement custom methods for its specific funxtionality
• src/main/new_agent.cpp: this is the executable code.

The base Agent class provides all the low level functionality to exchange information with the other agent via broker.

In the main function for the new agent, the steps to follow are:

1. Instantiate the new agent: auto agent = MyNewAgent(argv[0], "settings.ini"). This way, the agent reads the settings in the ini file from the section named after the last element in argv[0], i.e. the executable file name
2. If the agent is a source, call agent.connect_sub(), which sets up the connection to the broker XSUB port
3. If the agent is a filter or a sink, also call agent.connect_pub(), which sets up the connection to the broker XPUB port
4. Optionally, print configuration input with agent.info()
5. Within the main loop, you typically have to:
   1. If the agent is a source, read data from the field and pack them as JSON using nlohmann::json
   2. If the agent is filter or sink, read incoming topics with agent.receive(). This receives inbound messages and stores them in a status hash (one key per topic), retrivable with agent.status(). The very last message is available as a tuple topic,content from agent.last_message(). Messages are encoded as JSON, so decoding is needed with nlohmann::json class
   3. Operate on inbound or field data to build the new outbound payload
   4. Publish the new payload with agent.publish(). It will use the topic specified in the settings file.
6. That's it.

In order to keep the code more readable and organized, the algorithms implemented in step 5.3 are preferably implemented in the class MyNewAgent, rather than in the main function or in the executable file.

For plugin agents, see the dedicated section below.

The project is designed to be extensible via plugins. A plugin is a shared library that can be loaded at runtime by the main executable, and that can be used to extend its functionality.

The plugin system is based on the dlopen and dlsym functions, which are available on Unix systems. On Windows, the equivalent functions are LoadLibrary and GetProcAddress.

The plugin system is based on the C++ pugg library, which simplifies and abstracts the loading and unloading of plugins.

The advantages of the pluging systems are:

• each plugin can be developed in separate repos and compiled separately from the main executables, reducing the number of dependencies of this repository
• the main executables can be compiled without the plugins, and the plugins can be added later, without recompiling the main executables
• in the future, it would be possible to have the broker distribute the plugins to the agents, so that the agents can load the plugins at runtime together with the settings file
• Respect to have completely separate projects and repos for whole agents, the plugins are supposed to be smaller and more focused, and much easier to develop: they can act as simple blocks that takes an input in the form of a JSON object and provide a similar output. No knowledge about the Miroscic agent network is required

To develop a plugin follow these steps:

1. use the command mads plugin that creates a stub project with template files
2. look at the files within src/plugin dir
3. customize the CMakeLists.txt file to compile the plugin. Add any external library needed by the plugin. If the libraries are static, you won't need to install them in the target system, but the resulting plugin file would be bigger (possibly an issue if distributing plugins via broker)
4. in developing the plugin you must also add a main() function, which is used to test the plugin. This function is not used in the final plugin, but it is useful to test the plugin in isolation
5. once the plugin is ready, compile it and copy it on the target system where the Mads agent is supposed to run
6. the Mads agent that loads the plugis are source, filter, or sink: they take as argument the name of the plugin as a key for loading the proper settings section and as a publishing topic. Settings are passed to the plugin as a JSON object on loading, the name of the section being the name of the plugin file (no extension).

The project is distributed under CC BY-SA license.

Main author: Paolo Bosetti (University of Trento)

Contributors: Anna-Carla Araujo (INSA Toulouse), Guillaume Cohen (INSA Toulouse)