Soracom Harvest Files

Downloading Files

Files that have been uploaded to Harvest Files can be downloaded by devices, as well as directly from the User Console.

Downloading from a device

To download files from Harvest to your device, first make sure that your device is connected to Soracom using Soracom Air for Cellular or Soracom Arc, then configure your device to download files using Harvest Files' HTTP entry point.

Your device should be configured to download data from: http://harvest-files.soracom.io, appending the file path and filename to the request URL.

curl -O http://harvest-files.soracom.io/images/a.png

Using ETags

In some situations, you may want to download a file again only if it has been modified since a previous download. This can be accomplished using ETags (or entity tags), which serves as an identifier for a specific version of a resource. The ETag of a file stored in Harvest Files will be shown in the HTTP response header.

The following examples illustrate ETag usage with the curl command, however ETags can be used with other HTTP clients.

To simply check the ETag of a file:

curl -I http://harvest-files.soracom.io/images/a.png

When downloading a file for the first time, you can save the ETag value to an adjacent file using the --etag-save option:

curl -O \
|  --etag-save a_png_etag.txt \
|  http://harvest-files.soracom.io/images/a.png

Then on subsequent downloads, you can use the --etag-compare option to first check if the ETag has changed. This will ensure that the file is downloaded only if the ETag has changed:

curl -O \
|  --etag-save a_png_etag.txt \
|  --etag-compare a_png_etag.txt \
|  http://harvest-files.soracom.io/images/a.png

The --etag-save and --etag-compare options are available in curl version 7.68.0 or higher.

Note that while using ETags will allow your device to avoid re-downloading the same file multiple times, a small amount of data will still be used to request the HTTP headers in order to perform the ETag comparison.

Downloading from the User Console

To download files that have been uploaded to Harvest Files:

Login to the User Console. From the Menu, expand Data storage & visualization and select Soracom Harvest Files.

https://console.soracom.io

Harvest Files browser

You can then browse through the files and click the button to save the file locally.

Programmatic Usage

You can also use the Soracom API or Soracom CLI to download files.

Soracom API

To access the Soracom API, first use the auth API to obtain an API Key and Token. Refer to the API Usage Guide for instructions on how to use the API Key and Token in API requests.

Then, use the listFiles API to get a list of files stored by Harvest Files:

curl -X GET \
|  -H "X-Soracom-API-Key: <MY-API-KEY>" \
|  -H "X-Soracom-Token: <MY-TOKEN>" \
|  -H "Accept: application/json" \
|  https://g.api.soracom.io/v1/files/private/<PATH>/

A JSON object containing a list of files will be returned. By default, this API will return the first 10 files. To increase the number of files returned in the list, specify a limit using the limit query parameter in the request URL, such as https://g.api.soracom.io/v1/files/private/?limit=100. The maximum amount of files that can be retrieve per request is 100. If you have more than 100 files stored in Harvest Files, the API response will also return a value link in the response header indicating the request URL to use to retrieve the next set of files.

Then, use the getFile API to download a file to your local machine:

curl -X GET -O -J \
|  -H "X-Soracom-API-Key: <MY-API-KEY>" \
|  -H "X-Soracom-Token: <MY-TOKEN>" \
|  https://g.api.soracom.io/v1/files/private/<PATH>/<FILE>

You can repeat the getFile API call while looping through the list returned from the listFiles API in order to download multiple files at once.

Files can also be downloaded in bulk. See our Soracom Blog for more information on this topic

Soracom CLI

To use the Soracom CLI, you must first configure it to authenticate with your account information, authorization key, or SAM user credentials.

Then, run the following command to get a list of files stored in Harvest Files:

soracom files list --coverage-type g

By default, the CLI will return the first 10 files. To increase the number of files returned in the list, use the --limit flag to specify a new limit. Compared to the API usage, you can use the --fetch-all flag to automatically perform pagination and return a list of all files at once.

Then, run the following command to download a specific file:

soracom files get --path "<PATH>/<FILE>" --coverage-type g

Similar to the API, you can repeat the soracom files get command while looping through the list returned from the soracom files list command in order to download multiple files at once.