Binary Parser

Binary Parser is a function for Soracom Air devices that parses binary data with a fixed format and converts the data into easy-to-use JSON data, which can then be passed to Soracom Beam, Funnel, Funk, and Harvest Data. Binary Parser is available for Air for Cellular, Air for Sigfox, and Air for LoRaWAN.

When using Binary Parser, you must select either a Predefined Format or define a Custom Format.


How Data is Parsed

With the Binary Parser option enabled, the binary payload sent from a device to Beam, Funnel, Funk, Harvest Data, or Unified Endpoint is automatically parsed into a JSON object, and the resulting JSON data is sent to the configured destination. There is no need to modify the Beam, Funnel, Funk, or Harvest Data destination configuration.

Binary Parser Mechanism

When binary data is converted to JSON, a binaryParserEnabled: true attribute will be added to the JSON data.


Configuration

The Binary Parser option is found within Soracom Air Group settings.

Binary Parser settings are applied to all device types in the same group. Therefore, configuring Binary Parser for cellular devices in a group will also apply the same Binary Parser configuration for Sigfox and LoRa devices in the same group. If you require multiple Binary Parser settings, please create separate groups.

  1. Login to the User Console. From the Menu, open the Groups screen.

  2. From the list of groups, click the name of the group you want to configure to open its settings page.

  3. From the Basic Settings tab, click the SORACOM Air for Cellular, SORACOM Air for Sigfox, or SORACOM Air for LoRaWAN panel to expand its settings.

    https://console.soracom.io

    Binary Parser

  4. Enable the Binary Parser option by switching the option to ON.

  5. Enter the Binary Parser Format to use.

  6. Click Save at the bottom of the panel.

Options


Predefined Format

Predefined formats are available for select devices and libraries. When using a supported device or library, simply enter the Format string when configuring Binary Parser settings.

Supported devices:

Device Format Notes
Soracom LTE-M Button @soracom.button Supports Soracom LTE-M Button
Sigfox Sens'it v2 @sensit Supports Sens'it v2
Sigfox Sens'it v3 (Discovery) @sensit.v3 Supports Sens'it v3
Sigfox Shield for Arduino @unashield Supports data sent using the unabiz-arduino library; other data formats not supported
(any) @msgpack Supports data that is compressed using MessagePack ; data must be sent to Unified Endpoint in order for Binary Parser to decompress
(any) @json Supports data that is in JSON format
(any) @cbor Supports data that is in CBOR format, as specified in RFC 8949 Concise Binary Object Representation ; data containing CBOR Tags extensions used for representing datetime types and large integers (Big Integer) cannot be processed and will return an error

Refer to the Soracom blog for an example of using Binary Parser with a supported device.


Custom Format

When defining a custom format, individual values can be parsed from the payload using the following definition:

[name]:[byte index]:[type]:[options]

Multiple values can be parsed by creating multiple definitions, each separated with a space.

For example, the following Format definition will produce a JSON object that looks like this:

flag:0:bool:7 temp:1:int:13:/10 humid:3:uint:8:/100 lat::float:32 long::float:32
{
  "flag": true,
  "temp": 18.0,
  "humid": 80.0,
  "lat": 39.8283,
  "long": 98.5795
}

Name

The JSON object key where the parsed value should be stored. Available characters: letters, numbers, -, _, #, and $.

Note that the following words cannot be used, as they are reserved for use in Sigfox and LoRaWAN paylods:

Byte Index

Defines which byte of the payload should be read in order to parse the value, with 0 corresponding to the first byte of the payload.

Type

Defines the type to use for the value when parsing data. The following value types are available:

Options

Available options vary depending on the defined type. For types that support multiple options, each option should be separated using a colon :.

bool

The bool type will read one bit within the specified byte, and return true if the bit is 1, or false if the bit is 0.

Options

  • offset (required, min: 0, max: 7) - indicates which bit should be used to parse the boolean value, where:

    • 7 will read the left-most bit of the byte
    • 0 will read the right-most bit of the byte

    For example, with a byte value of 0x9A or 10011010b, the Binary Parser format flag:0:bool:7 will:

    • read the first byte of the payload (byte index: 0),
    • read the left-most bit of the byte (offset: 7),
    • convert the bit value to a boolean (type: bool),
    • assign the value to a key named flag (name: flag)

    Binary Parser Format Offset

int, uint

The int and uint types will read a specified number of bits and convert the signed or unsigned binary value to an integer. When specifying int, the first (left-most) bit of the bit sequence is used as the sign of the integer value, with 0 returning a positive number, and 1 returning a negative number.

Options

  • length (required, min: 1, max: 32) - the number of bits to read when parsing the integer value. If the number of bits (including any offset) exceeds the number of bits in the byte, the bit sequence will continue to the next byte.
  • offset (optional, min: 0, max: 7, default: 7) - indicates which bit should be used as the start of the bit sequence, where:

    • 7 will begin reading the bit sequence from the left-most bit of the byte
    • 0 will begin reading the bit sequence from the right-most bit of the byte

    Examples:

    • Read the last four bits of a byte by specifying a length of 4 with an offset of 3: temp:0:uint:4:3.
    • Read the last ten bits of a 2-byte sequence by specifying a length of 10 with an offset of 1: temp:0:int:10:1.
  • operations (optional, limit: 5 operations) - a simple arithematic expression to evaluate on the parsed integer value, supporting:

    • + to add a number to the previous value
    • - to subtract a number from the previous value
    • * to multiply the previous value by a number
    • / to divide the previous value by a number
    • (any number) to use in the operation

    Expressions are evaluated left-to-right, and order of operations (such as paranthetical evaluation) is not supported. For example, an expression +10*2 will first parse a binary value to an integer value, then add 10 to the parsed value, then multiply the result by 2.

    When dividing, the resulting value is converted to a decimal value.

float

The float type will read a specified number of bits and convert the single- or double-precision float value to a decimal using the corresponding IEEE 754 standard.

Options

  • length (required, 32 or 64) - the number of bits to read when parsing the float value
  • offset (optional, min: 0, max: 7, default: 7) - same as the int and uint offset parameter
  • operations (optional, limit: 5 operations) - same as the int and uint operations parameter
char

The char type will convert a bit value to the corresponding ASCII value.

Options

  • length (required, min: 1, max: 8) - the number of bits to read when parsing the ASCII value

Advanced Formatting

Endianness

By default, Binary Parser will parse numerical values using big-endian format, with most-significant bit first and least-significant bit last. You can override this behavior to parse the bit sequence using little-endian by appending :little-endian to the end of the format definition.

Example: value:0:float:64:little-endian

Byte Index Shorthand

Normally, the byte index defines which byte in the payload contains the value to be parsed. However, the byte index can be left blank using ::, which will cause the Binary Parser to use the next byte as the byte index, following the previously-used byte index.

For example, with 2-byte payload and a definition value1:0:int:6 value2::int:8:

However, when using the byte index shorthand with multiple consecutive definitions that are type bool, Binary Parser will re-use the previously-used byte index. Any non-bool definition after a sequence of bool definitions will resume the normal byte index shorthand behavior of using the next byte index.

This allows you to read each of the individual bits of a byte as separate boolean values, using a definition such as:

flagA:0:bool:7 flagB::bool:6 flagC::bool:5 value:0:int:5:4

This format will parse the following values:

Note that when using the byte index shorthand with multiple type bool definitions, Binary Parser will not keep track of bit offsets, and will continue to re-use the same byte index for all consecutive type bool definitions. For example, to parse the first ten bits of a 2-byte sequence as boolean values, you can use the byte index shorthand for the 2nd-8th definitions. However, you must re-specify the byte index in the 9th definition in order to move the byte index forward. Afterwards, you can use the byte index shorthand again for the 10th definition.


Limitations


Error Handling

Any parse errors are stored in an errors key within the JSON object.

Soracom does not provide any tools for inspecting or debugging binary data.


Advanced Configuration

Binary Parser settings can also be configured through the Soracom API or CLI by using the SoracomAir namespace.

Binary Parser settings are shared among all device types (cellular, Sigfox, and LoRaWAN) within the same group. If you require different Binary Parser settings, please create separate groups for each device type.

Configuration Structure

  "binaryParserEnabled": true|false,
  "binaryParserFormat": "",

Parameters

Enable or disable Binary Parser:

Modify the Binary Parser format:

Example

The following configuration will set and enable Binary Parser:

[
  {
    "key": "binaryParserEnabled",
    "value": true
  },
  {
    "key": "binaryParserFormat",
    "value": "flag:0:bool:7 temp:1:int:13:/10 humid:3:uint:8:/100 lat::float:32 long::float:32"
  }
]

Programmatic Usage

You can use the advanced configuration structure to configure the Binary Parser option programmatically using the Soracom API and Soracom CLI.

Soracom API

To access the Soracom API, first use the auth API to obtain an API Key and Token. Refer to the API Reference Guide for instructions on how to use the API Key and Token in API requests.

Then, use the putConfigurationParameters API with the SoracomAir namespace to set the Binary Parser option:

curl -X PUT \
>  -H 'X-Soracom-API-Key: <MY-API-KEY>' \
>  -H 'X-Soracom-Token: <MY-TOKEN>' \
>  -H 'Content-Type: application/json' \
>  -d '[
>        {
>          "key": "binaryParserEnabled",
>          "value": true
>        },
>        {
>          "key": "binaryParserFormat",
>          "value": "flag:0:bool:7 temp:1:int:13:/10 humid:3:uint:8:/100 lat::float:32 long::float:32"
>        }
>      ]' \
>  https://g.api.soracom.io/v1/groups/<GROUP-ID>/configuration/SoracomAir

Soracom CLI

To use the Soracom CLI, you must first configure it to authenticate with your account information, authorization key, or SAM user credentials.

Then, run the following command to set the Binary Parser option:

soracom groups put-config --group-id "<GROUP-ID>" --namespace "SoracomAir" --body "<CONFIG>" --coverage-type g

Here, the JSON array from the API example above should be passed in to the --body parameter.