Using the SORACOM Funnel Kinesis Firehose adapter to aggregate data on the cloud

This guide explains how to use the Amazon Kinesis Firehose adapter on SORACOM Funnel (hereafter, "Funnel") to aggregate data sent from a device on Amazon S3.

Funnel makes use of the SORACOM User Console for its settings.

Preface

Funnel is a cloud resource adapter that allows you to directly forward data from a device to a specific cloud service.

Simply indicating a resource on a cloud service allows you to directly aggregate data from your device. The major difference between Funnel and SORACOM Beam is that, while the forwarded destination is not a generic protocol, you can expend a minimum of time and effort to rapidly deploy use of cloud services.

Funnel explained

Amazon Kinesis Firehose setup

If you have selected the Tokyo region, you will see the screen below. Select your region; here, we select US (Oregon).

Funnel

Begin by setting up Amazon Kinesis Firehose. Select Kinesis from the AWS (Amazon Web Services) console.

Funnel

Click Go to Kinesis Firehose.

Funnel

From the Amazon Kinesis Firehose administration screen, select Create Stream.

Funnel

Next, select Amazon S3 as the Destination and my-kinesis-firehose01 as the Delivery stream name. Then select S3 bucket.

Funnel

Select the IAM role used by the Kinesis Firehose service. Here we select Firehose delivery IAM role.

Funnel

If the screen below appears, allow access to the IAM role.

Funnel

Confirm the settings and click Create Delivery Stream to create a Delivery Stream.

Funnel

Next, create the IAM users that will access Kinesis Firehose. On the AWS IAM Management Console, select an existing or create a new user and attach the policy. We attach Full Access.

This concludes setup of the Amazon Kinesis Firehose cloud service.

SORACOM Funnel setup

SORACOM Funnel setup consists of two steps:

Air SIM group creation and setup
Register credentials
Specify cloud service

Air SIM group creation and setup

Funnel requires that you configure Air SIMs on a group basis. You must first assign those Air SIMs to some sort of group. In this example, we create a dedicated group used for testing the Funnel functionality and assign SIMs to it. To create a group and assign SIMs, take the following steps. First open the Air SIM administration page and select the Air SIM you want to communicate with Funnel with. Click the Execute button, then select Change Assigned Group.

Funnel

A dialog asking you to select the group with which the Air SIM is associated will appear. From the [New Group] dropdown, select [Create...]

Funnel

A group creation dialog like that seen below will appear. Enter the group name and click the Create Group button. In this example, we crete a group with the name "Kinesis_firehose"

Funnel

Pressing the Create Group button will take you back to the Change SIM Group dialog. A dropbox indicating New Group will appear; confirm that this is set to the newly-created "hello beam" group and click the Change Group button.

Funnel

Register credentials

Begin by registering the credential data to access AWS. Log in to the User Console .

From the dropdown menu, click Security. Funnel

The screen below will appear. From the menu at left, select Store Authentication Data and then click Register Authentication Data.

Funnel

You will now have registered authentication data.

The Authentication ID is the authentication ID specified by the Funnel service.
As needed, enter a summary here.
Select AWS Authentication Data for the type.
For the AWS Access Key ID and AWS Secret Access Key, use the IAM user data previously set.

Funnel

This completes registration of the my-aws-credentials ID.

Funnel

Cloud service setup

Next, we specify a cloud service. The Funnel settings (cloud service set) are applied to the Air SIM group.

From the menu at the top of the screen, click Groups, then select the group you created. A screen like that seen below will be displayed.

Funnel

Open the SORACOM Funnel Settings tab and specify a cloud service as below.

Select the resource type. Here we select Amazon Kinesis Firehose.
Enter the resource URL.
- For the US (Oregon) region, the resource URL is as follows:
```
https://firehose.us-west-2.amazonaws.com/<Delivery stream name>
```
For the authentication data, select the AWS authentication data previously created.
Select JSON for the data format.

Funnel

This concludes the setup.

Using the Kinesis Firehose adapter to aggregate data on the cloud

We now send some data. Remember to assign the SORACOM Air SIM card to the group you previously created.

Funnel

Send a request to the Funnel endpoint. The endpoints are as follows:

funnel.soracom.io:23080
funnel.soracom.io:80

Send over TCP

Transfer command example

$ nc funnel.soracom.io 23080
{"message":"Hello SORACOM Funnel via TCP!"} [Enter]
200
[Ctrl+C]

Send over UDP

Transfer command example

$ nc -u funnel.soracom.io 23080
{"message":"Hello SORACOM Funnel via UDP!"} [Enter]
200
[Ctrl+C]

Send over HTTP

Transfer command example

curl -vX POST \
>  -H 'Content-Type: application/json' \
>  -d '{
>        "message": "Hello SORACOM Funnel via HTTP!"
>      }' \
>  http://funnel.soracom.io
>
>* Rebuilt URL to: http://funnel.soracom.io/
>*   Trying 100.127.65.43...
>* Connected to funnel.soracom.io (100.127.65.43) port 80 (#0)
>> POST / HTTP/1.1
>> Host: funnel.soracom.io
>> User-Agent: curl/7.43.0
>> Accept: */*
>> Content-Type:application/json
>> Content-Length: 44
>>
>* upload completely sent off: 44 out of 44 bytes
>< HTTP/1.1 204 No Content
>< Date: Thu, 14 Jan 2016 17:18:46 GMT
>< Connection: keep-alive
><
>* Connection #0 to host funnel.soracom.io left intact

Check aggregated data

Check the S3 Bucket previously specified as the Kinesis Firehose Destination. Check that a file has been created as below.

Funnel

{"operatorId": "OP0026965167", "timestamp": 1455672330494, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello SORACOM Funnel via TCP!"}, "sourceProtocol": "tcp", "imsi": "440XXXXXXXXXX"}
{"operatorId": "OP0026965167", "timestamp": 1455672351248, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello SORACOM Funnel via UDP!"}, "sourceProtocol": "udp", "imsi": "440XXXXXXXXXX"}
{"operatorId": "OP0026965167", "timestamp": 1455672363539, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello SORACOM Funnel via HTTP!"}, "sourceProtocol": "http", "imsi": "440XXXXXXXXXX"}

This concludes the explanation of how to use the Kinesis Firehose adapter on SORACOM Funnel to aggregate data (console version).

With Funnel, there is no need to load the SDK for the cloud service on your device. In addition, you do not need to maintain cloud service credentials on your device.

You can now simply specify the cloud service credentials and resource URL to directly send data from your device to the cloud service in question. This allows you to expend a minimum of effort to quickly build a system linking your devices with cloud services.

We plan to continue adding a range of features, so do not hesitate to share your requests with us.