FrameProcessor#

Commandline Interface#

The following options and arguments can be given on the commandline:

$ frameProcessor --version
frameProcessor version 1.7.0

$ frameProcessor --help
Usage: frameProcessor [options]


Generic options:
  -h [ --help ]                         Print this help message
  -v [ --version ]                      Print program version string

Configuration options:
  -d [ --debug-level ] arg (=0)         Set the debug level
  -l [ --logconfig ] arg                Set the log4cxx logging configuration
                                        file
  --iothreads arg (=1)                  Set number of IPC channel IO threads
  --ctrl arg (=tcp://0.0.0.0:5004)      Set the control endpoint
  --ready arg (=tcp://127.0.0.1:5001)   Ready ZMQ endpoint from frameReceiver
  --release arg (=tcp://127.0.0.1:5002) Release frame ZMQ endpoint from
                                        frameReceiver
  --meta arg (=tcp://*:5558)            ZMQ meta data channel publish stream
  -j [ --json_file ] arg                Path to a JSON configuration file to
                                        submit to the application

Limitations#

Currently the application has a number of limitations (which will be addressed):

  • No internal buffering: the application currently depend on the buffering available in the frameReceiver. If the frameProcessor cannot keep up the pace it will not release frames back to the frameReceiver in time, potentially causing the frameReceiver to run out of buffering and drop frames. Fortunately missing frames are obvious in the output files (as empty gaps).

  • Metadata like the “info” field and original frame ID number is not recorded in the file.

  • Sensor data is only stored in raw 16bit format for both gain and reset frames. The fine/coarse ADC and gain data can be generated in the file by a post-processing step (see python script decode_raw_frames_hdf5.py)

Runtime Configuration#

Further configuration of the application is done by json configuration messages from a client, but for convenience configuration can be passed as a JSON file on startup to set them up without the need for a control connection. All configuration messages that are supported from the control interface are also supported through the configuration file.

To supply a JSON configuration file from the command line use the -j or --json_file options and supply the full path and filename of the configuration file. The file must be correctly formatted JSON otherwise the parsing will fail.

File Structure#

The file must specify a list of dictionaries, with each dictionary representing a single configuration message that is to be submitted through the control interface. When the configuration file is parsed each dictionary is processed in turn and submitted as a single control message, which allows for multiple instances of the same control message to be submitted in a specific order (something that is required when loading plugins into a FrameProcessor application.

The example below demonstrates setting up a FrameProcessor with two plugins. There are five control messages defined here, with two sets of duplicated messages (for loading and connecting plugins).

  • The first message will setup the interface from the FrameProcessor to the FrameReceiver application

  • The second message loads the DummyProcessPlugin into the FrameProcessor application

  • The third message loads the FileWriterPlugin into the FrameProcessor application

  • The final two messages connect the plugins together - in this example, the FileWriterPlugin is connected to the DummyProcessPlugin, which is connected to frame_receiver

Note

Note: frame_receiver is a special key that connects the plugin to the SharedMemoryController. This is the FrameProcessor interface to the shared memory made available by the FrameReceiver.

Configuration Options#

The IPC control channels operate via IPCMessages. There are a few main message types:

  • cmd

  • ack

  • nack

  • notify

cmd is used by clients to monitor and configure the applications, notify is passed between the FrameReceiver and FrameProcessor to managed shared memory buffers and ack and nack are used to report success and failure from other messages.

These are the message values that can be used by a client in a “cmd” message:

  • shutdown - Cleanly shutdown application

  • reset_statistics - Reset any parameters storing long running statistics

  • request_version - Request version parameters

  • status - Request current read-only status parameters

  • request_configuration - Request read-write configuration parameters

  • configure - Set the value of a read/write parameter

FrameProcessorController#

Below are some common configurations to send to a frameProcessor application. In these examples, the given json is the value of params in the following complete control message:

{
  "msg_type": "cmd",
  "id": 1,
  "msg_val": "configure",
  "params": {},
  "timestamp": "2022-03-20T18:47:58.440432"
}

Initialise FrameReceiver Interface#

FrameReceiver Interface
{
  "fr_setup": {
    "fr_ready_cnxn": "tcp://127.0.0.1:5001",
    "fr_release_cnxn": "tcp://127.0.0.1:5002"
  }
}

Load Plugin#

Load an instance of a plugin into the application. This can be be done multiple times per class as long as the index is unique.

Load Plugin
{
  "plugin": {
    "load": {
      "index": "sum",
      "name": "SumPlugin",
      "library": "prefix/lib/"
    }
  }
}

Note

index is the label for referencing the plugin in further commands, such as connecting it to another plugin

Connect Plugins#

Connect one plugin to another. Frames pushed by the plugin given as connection will be added to the queue of the plugin given by index.

Connect Plugin
{
  "plugin": {
    "connect": {
      "index": "hdf",
      "connection": "sum"
    }
  }
}

Disconnect Plugins#

Disconnect the plugin given as index from the plugin given as connection.

Disconnect Plugin
{
  "plugin": {
    "disconnect": {
      "index": "hdf",
      "connection": "sum"
    }
  }
}

Disconnect All Plugins#

A quick way to remove the connections between all plugins. This is useful as part of a stored stored config for reconnecting plugins in a different mode.

Disconnect All Plugins
{
  "plugin": {
    "disconnect": "all"
  }
}

Plugin Configuration#

See specific plugins for what configurations are accepted. As an example, here is a message to configure the FileWriterPlugin to start writing, i.e. the key is the index of the plugin to configure and the value is a json dictionary that will be passed to the plugin as configuration.

Configure Specific Plugin
{
  "hdf": {
    "write": true
  }
}

Set Debug Level#

Set the verbosity of debug level log messages.

Set Debug Level
{
  "debug": 1
}

Clear Errors#

Clear any error state.

Set Debug Level
{
  "clear_errors": true
}

Note

The value is ignored. Any valid json with the key clear_errors is sufficient.

Store Config#

Store a series of configuration messages to be applied with the given index. This is useful for reconnecting the plugins in a different design.

Store Plugin Connections With Sum
{
  "store": {
    "index": "sum",
    "value": [
      {
        "plugin": {
          "disconnect": "all"
        }
      },
      {
        "plugin": {
          "connect": {
            "index": "dummy",
            "connection": "frame_receiver"
          }
        }
      },
      {
        "plugin": {
          "connect": {
            "index": "sum",
            "connection": "dummy"
          }
        }
      },
      {
        "plugin": {
          "connect": {
            "index": "hdf",
            "connection": "sum"
          }
        }
      }
    ]
  }
}
Store Plugin Connections Without Sum
{
  "store": {
    "index": "no_sum",
    "value": [
      {
        "plugin": {
          "disconnect": "all"
        }
      },
      {
        "plugin": {
          "connect": {
            "index": "dummy",
            "connection": "frame_receiver"
          }
        }
      },
      {
        "plugin": {
          "connect": {
            "index": "hdf",
            "connection": "dummy"
          }
        }
      }
    ]
  }
}

Execute Config#

Reconfigure a previously saved plugin connection config given by index.

Execute Stored Config
{
  "execute": {
    "index": "sum"
  }
}

FileWriterPlugin#

Process#

Configure the rank of the node and the total number of nodes in the deployment.

Process
{
  "process": {
    "number": 4,
    "rank": 0
  }
}

File#

Configure the output for the file.

File
{
  "file": {
    "name": "test",
    "path": "/tmp",
    "extension": "h5"
  }
}

Note

The full file path will be {path}{name}{number}{extension} where number is generated by the process based on its rank, the total number of processes and how many files it has written.

Create a Dataset#

Create a dataset to be written to the file.

Create Dataset
{
  "dataset": "data"
}

Note

Dataset <DATASET NAME> will be created the first time any dataset config is sent with it included. It does not need this specific message before sending further configuration.

Dataset Config#

Configure a dataset in the file.

Configure Dataset
{
  "dataset": {
    "data": {
       "datatype": "uint64",
       "dims": [515, 2069],
       "chunks": [1, 515, 2069],
       "compression": "none"
    }
  }
}
Data Types

uint8, uint16, uint32, uint64, float

Compression Modes

none, LZ4, BSLZ4, blosc

Dataset Blosc Config#

Configure the blosc compression of the dataset.

Configure Dataset Blosc Compression
{
  "dataset": {
    "data": {
       "blosc_compressor": "blosclz",
       "blosc_level": 8,
       "blosc_shuffle": 2
    }
  }
}
Blosc Compressors

0-5 -> blosclz, lz4, lz4hc, snappy, zlib, zstd

Blosc Levels

1-8

Blosc Shuffle Modes

0-2 -> none, byteshuffle, bitshuffle

Start/Stop Writing#

Start and stop file writing.

Start file writing
{
  "write": true
}
Stop file writing
{
  "write": false
}