High-level Operations

The Bayware Configuration Management (CM) protocol underpins the following high-level operations (borrowed from the Wikipedia’ article on TR-069):

  • initial configuration of node
  • node initial configuration recovery
  • node interface provisioning
  • verification of node status and functionality
  • manual node reconfiguration
  • firmware upgrade/downgrade
  • configuration backup/restore
  • throughput and connectivity diagnostics
  • parameter value retrieval
  • log file retrieval

Configuration Management Protocol

Configuration Commands

The configuration management protocol relies on controller’s Southbound RESTful API. The protocol utilizes HTTP response messages in a specific way. Each response message contains a command that can trigger a next RESTful API request. Using such a command, controller can order node to:

  • update configuration
  • provide configuration
  • provide statistics
  • provide logs

When controller places a command in the response, it uses the code HTTP 202 indicating that the request has been accepted for processing, but the processing has not been completed. A controller’s response with the code HTTP 200 indicates there are no more commands.

The commands are usually implemented by placing special values of parameters in controller’s response. For instance, the controller passes 1 as a value of the parameters cfg_refresh and net_refresh to get blocks of configuration from the node. An example of such response on a keep-alive update (HTTP 202) is shown below.

    “host_id”: “123456789abc”,
    “keepalive_period”: 600,
    “cfg_refresh”: 1,
    “net_refresh”: 1

Master/Slave Replication of Configuration Blocks

Configuration Management protocol provides master/slave replication of configuration blocks between node and controller. All node configuration data is organized in a form of blocks. As described later, some node configuration blocks can be updated by controller only (controller is Master), while others by node only (node is Master). Any time the controller discovers that the node’s copy of controller-owned data differs from its own, the controller pushes its copy of data to the node. When the controller realizes that its copy of node-owned data differs from the node’s, it pulls data from the node.

HTTP Persistent Connection

Multiple HTTP requests/responses reuse the same HTTP connection. The node does not close the connection until it receives the HTTP response with no command (HTTP 200) from the controller.

Configuration Stages

Triggering Algorithm Execution

Events that can trigger configuration algorithm execution are:

  • a scheduled keep-alive
  • an unscheduled keep-alive initiated by the node (e.g. on a boot/reboot, configuration update, etc.)
  • for switches only, an unscheduled keep-alive initiated by the controller (e.g. to push a new configuration, retrieve parameter value, etc.)
A Scheduled Keep-alive

Each node sends keep-alive message every given interval of time.

An Unscheduled Keep-alive Initiated by the Node

When a node needs to immediately get configuration data from the controller, not waiting for the next scheduled keep-alive, the node dispatches an unscheduled keep-alive message.

An Unscheduled Keep-alive Initiated by the Controller

When a controller needs to immediately transfer an order to the switch, not waiting for the next scheduled keep-alive, the controller sends a special Connection Request Message (CRM) to the switch. The message triggers a dispatching mechanism of the switch that sends the unscheduled keep-alive message to the controller.


The execution of the configuration algorithm always begins with a keep-alive request sent to the controller from the node. Upon receiving the message, the controller checks whether any commands are available for this node. The controller can respond either with or without commands in the body of the HTTP response.

The controller dispatches the response message using the following codes:

  • HTTP 200 - no command present in the response body
  • HTTP 202 - command present in the response body

Command Execution

On receiving the keep-alive response (HTTP 202) containing command from the controller, the node replies with the HTTP message carrying out the command. One or more request/response exchanges may happen between the controller and the node. When no more commands are left, the controller sends an HTTP response (HTTP 200) without new command. On receiving such response, the node terminates its session with the controller.

Node Configuration Data Model

A simplified model of OF-Config 1.2 is used as a configuration data model.

Data Ownership

Locked Configuration

Some configuration data is not accessible via Configuration Management protocol. For instance, node and controller certificates are passed between the node and controller in the course of node registration process only.


Node is a master for the following blocks of configuration data:

  • Node Capabilities, including node interfaces
  • Node Connections
  • Node Endpoints


Controller is a master for the following blocks of configuration data:

  • Node Settings
  • ECDSA Certificates
  • Revoked Flow Profiles
  • Topic Configuration

Data Blocks

All configuration data is organized in the form of blocks. Each block has its own MD5-hash. The list of blocks and the names of corresponding hashes are shown below.

Configuration Block Owner Hash NodeType  
      Host Switch
Node Capabilities Node cfg_hash
Node Connections Node net_hash
Node Endpoints Node srv_hash
Node Settings Controller set_hash
ECDSA Certificates Controller crt_hash
Revoked Flow Profiles Controller crl_hash
Topic Configuration Controller tpc_hash

Interface Provisioning

Interface Types

As of current implementation, Icebreaker Overlay supports the following types of interfaces:

  • untagged Ethernet interface
  • tagged Ethernet interface
  • GRE layer 3 interface
  • GRE layer 2 interface

Interface Auto-configuration Procedure

The node interface provisioning is part of the “8-Step Service Auto-configuration” value proposition. As a new Bayware-enabled host has been spun up, the agent registers the host at the controller. Next, the tunnel has to be established between the host and a Bayware switch. When the tunnel is ready, the host and switch discover each other and establish a link. From this point, any application inside the host can initiate network service endpoint establishment and begin exchanging data via the endpoint.

Steps 2-4 of this process involve network provisioning. Provisioning network interfaces is a fully automated procedure.

The interface auto-configuration procedure comprises three steps:

  • retrieving data from the legacy IP-fabric management system
  • adding interface on the host
  • adding interface on the switch

In future, different types of layer 2 interfaces might be used. The document describes auto-configuration procedure for L2GRE tunnel endpoints used in the current implementation.


In the context of the GRE tunnel endpoints, execution of the port auto-configuration algorithm always starts with adding an interface to the host and finishes with adding an interface to the switch.

Retrieving Network Configuration from Cloud Manager

Adding Interface on Host

When an agent decides to start execution of the host interface auto-configuration procedure (e.g., there are no ib-gre port settings in the configuration file on the agent), the agent requests the ib-gre interface settings from the controller SB API Gateway. The agent specifies in the request such parameters as:

  • host IPv4 address

At the agent request, the controller chooses a switch layer 2 interface to serve the host. It can be either:

  • a VLAN interface – e.g. the OVS br-provider’s VLAN of the tenant to which the host belongs
  • an untagged Ethernet interface (physical or virtual)

As well, the controller decides if an existing or new port on the switch will be used. For example, if a port with the host’s IPv4 address already exists on the switch’s given VLAN, the controller does not create a new port. In response, the controller provides the agent with such parameters:

  • switch IPv4 address

After receiving the switch IPv4 address, the agent creates a new port inside the host using the script shown below.

add ib-gre interface

ip link add ib-gre type gretap remote ${GRE_REMOTE} local ${GRE_LOCAL}
ip link set ib-gre up

As soon as the port is up, the agent runs the link establishment and registration procedure on the interface.

Adding Interface on Switch

If a new switch interface is to be created, the controller starts the switch interface auto-configuration procedure. Otherwise, this step is skipped.