Configuration

Soracom Harvest settings are found in Soracom Air or Soracom Inventory group settings. To send data to Harvest, refer to the Harvest Data section below. To upload files to Harvest, refer to the Harvest Files section below.

Harvest Data

Enabling Harvest Data will incur fees based on the number of devices where the service is enabled. Refer to the Pricing & Fee Schedule for more information.

Harvest Data settings are shared among all device types within the same group. That means that enabling Harvest for one device type (cellular, Sigfox, LoRaWAN, or Inventory) will also enable Harvest for other device types within that group.

To configure Harvest, open the settings page of the group that you want to configure.

Underneath the Basic Settings tab, click the SORACOM Harvest Data panel to expand its settings.

Harvest Data configuration

Enable Harvest Data by switching the option to ON.

Harvest Files

Enabling Harvest Files will incur fees based on the amount of data uploaded or downloaded. Refer to the Pricing & Fee Schedule for more information.

Harvest Files settings are only available for Air for Cellular devices.

To configure Harvest Files, open the settings page of the group that you want to configure.

Underneath the Basic Settings tab, click the SORACOM Harvest Files panel to expand its settings.

Harvest Files configuration

Enable Harvest Files by switching the option to ON.

Parameters

Harvest Files allows you to configure options to control where files are saved, and how uploaded files can be accessed by devices.

In addition, the Log file uploading event to Harvest Data option allows you to create a data entry in Harvest Data whenever a file is uploaded, consisting of the file path and upload timestamp. This option can be combined with the Dynamic Image Panel in Lagoon in order to dynamically show new images in a Lagoon dashboard as they are uploaded from a device.

Harvest Data does not need to be enabled in order to use the Log file uploading event to Harvest Data option. In addition, file upload events that are logged to Harvest Data do not incur any Harvest Data fees service fees.

The Log file uploading event to Harvest Data option will only log file upload events for files that are uploaded using a cellular connection (http://harvest-files.soracom.io entry point). Files uploaded through the https://api.soracom.io/v1/files/private API entry point will not be logged to Harvest Data.


Extended Data Retention

By default, Harvest data retention period is 40 days, once the retention period has passed, data older than 40 days will be deleted. Deleted data cannot be recovered. However, users can subscribe to the Harvest Data Extended Retention Option, which will extend data retention to 731 days. Please refer to our Pricing & Fee Schedule for the cost of enabling this option.

To enable Extended Data Retention, log into the Soracom User Console, select Soracom Harvest Data. The alert at the top of the page will indicate that the default retention period of 40 days is enabled. Click the link within the alert to subscribe

  • Data export fees (fees for reading from Harvest Data) will be charged when the Extended Data Retention option is activated. Please refer to SORACOM Harvest Data Pricing Plan for more details.
  • The Harvest Data data export charge is displayed as soracomHarvestDataExportCharge in the Billing Details CSV.
  • Data export charges may be high depending on the number of times data is read and the volume of reads. Data export charges are incurred for any of the following operations. Both include the use of SORACOM API and SORACOM CLI for the following operations.
    • Confirming data stored in SORACOM Harvest Data
    • When changing the data displayed on the SORACOM Harvest Data screen when asking AI (ChatGPT API) a question using SORACOM Harvest Data Intelligence
    • When setting data stored in Harvest Data as Query in the Dashboard of Lagoon 3. In particular, if you enable Auto refresh Dashboard, or if you enable Public dashboard reflecting the latest data to any third party may result in high data export fees.
    • If data stored in Harvest Data is set to Query in the Alert rule of Lagoon 3. Data export fees can be expensive, especially if the Alert rule evaluation interval is shortened.

Advanced Configuration

Harvest Data and Harvest Files can also be configured through the Soracom API or CLI by using the SoracomHarvest and SoracomHarvesetFiles namespaces, respectively.

Configuration Structure

For Harvest Data, the group configuration looks like this:

"SoracomHarvest": {
  "enabled": true|false
}

For Harvest Files, the group configuration looks like this:

"SoracomHarvestFiles": {
  "enabled": true|false,
  "defaultPath": "/path/to/file",
  "assumedRoleId": "myRole",
  "logFileUploadingEventToHarvestData": {
    "enabled": true,
    "pathPatterns": [
      "/lagoon/.*"
    ]
  }
}

Parameters

For the SoracomHarvest namespace:

Enable or disable Harvest Data:

For the SoracomHarvestFiles namespace:

Enable or disable Harvest Files:

Set the default path for upload files:

Control the device read and write permissions:

Configure Log file uploading event to Harvest Data option:

Sample

Enable Harvest Data:

[
  {
    "key": "enabled",
    "value": true
  }
]

Enable Harvest Files, while specifying that each file uploaded should be given a file path (filename) according to the IMSI of the subscriber, and the timestamp when the file is uploaded:

[
  {
    "key": "enabled",
    "value": true
  },
  {
    "key": "defaultPath",
    "value": "/:imsi/:time"
  }
]