Soracom Harvest Data

Harvest Data Options

Harvest Data provides two options:

Batch Write is always enabled alongside Overwrite Timestamp with Payload Data.

On the other hand, Overwrite Timestamp with Payload Data can be used on its own.

Batch Write

Send multiple data points to Harvest Data in a single command by enabling Batch Write. This approach can help reduce power consumption on the cellular module and extend the device's battery life.

How to Enable Batch Write

  1. Login to the User Console. From the Menu, expand Soracom Air for Cellular and select Groups.

  2. Select a group.

    See Managing Groups for further information on creating and managing groups.

  3. Expand Soracom Harvest Data and toggle the switch to on.

  4. Toggle Batch write to on.

    This will automatically enable Overwrite timestamp with payload data.

    JSON Pointer to Data Array: Specify the JSON Pointer that identifies the data array in the payload (e.g., /data).

  5. Click Save.

    A confirmation dialog will be displayed:

    Screenshot of the confirmation dialog.

JSON Pointer to Data Array

JSON Pointer is a standardized syntax for identifying a specific value within a JSON document (RFC 6901).

For example, set the JSON Pointer to /data if your device sends data in the following format:

{
    "data": [
        {"time": "2025-03-17T00:00:00Z", "temperature": 15.5}, 
        {"time": "2025-03-18T00:00:00Z", "temperature": 15.4}
    ]
}

All data in the array must include a timestamp value that can be parsed by enabling Batch Write and Overwrite Timestamp with Payload Data.

When you want to send the entire data array without any modifications, enable Batch Write and leave the JSON Pointer to data array field blank to reference the entire JSON object.

Limitations

Overwrite Timestamp with Payload Data

Normally, the timestamp of data in Harvest Data is the time when the data is saved in Harvest Data. This default behavior can be overridden by using the timestamp included in the transmitted payload data.

JSON Pointer to Timestamp

JSON Pointer is a standardized syntax for identifying a specific value within a JSON document (RFC 6901).

For example, set JSON pointer as /time if the device sends data as shown below.

{
  "temperature": 12.3,
  "humidity": 45.6,
  "time": "2025-03-24T12:34:56.789Z"
}

When used together with Batch Write, the JSON Pointer should be relative to each data item, not to the root of the JSON document. For example, set JSON Pointer as /time if the device sends data as shown below. JSON Pointers such as /data/0/time and /data/1/time are not valid.

{
  "data": [
    {"time": "2025-03-17T00:00:00Z", "temperature": 15.5}, 
    {"time": "2025-03-18T00:00:00Z", "temperature": 15.4}
  ]
}

Date and time cannot be used as a timestamp if the values are split into multiple properties such as illustrated in the example below.

[
  {
    "temperature": 12.3,
    "humidity": 45.6,
    "date": "2025-03-20",
    "time": "12:34:56.789"
  }
]

Format

There are three formats available to parse the time values in the JSON document.

*1 When the milliseconds are not provided, Harvest Data automatically adds .000 milliseconds.

When ISO-8601 is set, the timezone identifier such as Z for UTC+00:00 and -08:00 for non-UTC timezones is necessary.

Strings without the timezone identifier like 2025-03-24T12:34:56 are not valid and fail to save returning an error.

Limitations

Example Data Sent First Data Sent Later Stored Data
1 {"temperature": 20} {"humidity": 30} {"temperature": 20, "humidity": 30}
(humidity is added)
2 {"temperature": 25, "humidity": 30} {"temperature": 28} {"temperature": 28, "humidity": 30}
(temperature is overwritten, humidity remains the same)

Send Data to Harvest Data

Devices using IoT SIMs that are assigned to the group where these settings are configured send their data to Harvest Data.

The example below illustrates group settings and explains the response returned when sending data to Harvest Data using Unified Endpoint as the endpoint.

  1. Enable Batch Write.

    Overwrite Timestamp with Payload Data is automatically enabled when Batch Write is enabled.

  2. Configure the following for Batch Write:

    • JSON Pointer to data array: Enter /data.

  3. Configure the following for Overwrite timestamp with payload data:

    • Payload format: Select JSON.
    • JSON Pointer to timestamp: Enter /time.
    • Format: Select Date/Time (ISO-8601).

  4. Click Save.

  5. Send data from the device to the Unified Endpoint.

    curl -v http://uni.soracom.io \
-H 'content-type: application/json' \
-d '{
  "data": [
    {"time": "2025-03-17T09:00:00+09:00Z", "temperature": 20.5},
    {"time": "2025-03-18T09:00:00+09:00Z", "temperature": 20.4}
  ]
}'
    * Host uni.soracom.io:80 was resolved.
* IPv6: (none)
* IPv4: 100.127.69.42
*   Trying 100.127.69.42:80...
* Connected to uni.soracom.io (100.127.69.42) port 80
> POST / HTTP/1.1
> Host: uni.soracom.io
> User-Agent: curl/8.5.0
> Accept: */*
> content-type: application/json
> Content-Length: 147
> 
< HTTP/1.1 201 Created
< Date: Fri, 28 Mar 2025 08:55:22 GMT
< Content-Length: 51
< Connection: keep-alive
< Keep-Alive: timeout=65
< 
* Connection #0 to host uni.soracom.io left intact
{"total":2,"status":{"successCount":2,"errors":[]}}

The data is saved in Harvest Data and the processing result is returned in the response. The meaning of each property in the response is as follows:

Properties Details
total The number of data entries recognized by Harvest Data.
status.successCount The number of data entries saved to Harvest Data.
status.errors[].index The number of data entries that could not be saved to Harvest Data.
status.errors[].code Error code.
status.errors[].message Error message.

Screenshot of Harvest Data: Time, Content Type, Content, Parsed data, and Chart data items shown.