Using the Soracom Funnel Kinesis Firehose adapter to aggregate data on the cloud

This guide explains how to use the Amazon Kinesis Firehose adapter on Soracom Funnel (hereafter, "Funnel") to aggregate data sent from a device on Amazon S3.

Funnel makes use of the Soracom User Console for its settings.

Preface

Funnel is a cloud resource adapter that allows you to directly forward data from a device to a specific cloud service.

Simply indicating a resource on a cloud service allows you to directly aggregate data from your device. The major difference between Funnel and Soracom Beam is that, while the forwarded destination is not a generic protocol, you can expend a minimum of time and effort to rapidly deploy use of cloud services.

Funnel explained

Funnel uses TLS 1.2. Please ensure to configure your cloud service to support TLS 1.2.

Amazon Kinesis Firehose setup

Select your region; here, we select US (Oregon).

Funnel

Begin by setting up Amazon Kinesis Firehose. Select Kinesis from the AWS (Amazon Web Services) console.

Funnel

Click Create Delivery Stream

Funnel

Next, select Amazon S3 as the Destination and my-kinesis-firehose01 as the Delivery stream name. Then select S3 bucket.

Confirm the settings and click Create Delivery Stream to create a Delivery Stream.

Funnel

Next, select the IAM users that will access Kinesis Firehose. On the AWS IAM Management Console, select an existing user and attach the policy.

From the AWS IAM, select Roles

Roles

Create a new role, selecting AWS Account and enable an External ID

ExternalRole

Create a name for the role, select to Create. Open the role and select Create Inline Policy

Addpermissions

Search for Firehose, select and then within the Actions Allows, enter Put

Addactions

Enable both options for Put and Click to add ARN.

SelectPut

Paste in the ARN from the Firehose Delivery Stream

AddARN

Within the details of your IAM role, you can view the ExternalID and ARN which will be necessary for setting credentials within the configuration of Funnel.

This concludes setup of the Amazon Kinesis Firehose cloud service.

Soracom Funnel setup

Soracom Funnel setup consists of two steps:

Air SIM group creation and setup
Register credentials
Specify cloud service

Air SIM group creation and setup

Funnel requires that you configure Air SIMs on a group basis. You must first assign those Air SIMs to some sort of group. In this example, we create a dedicated group used for testing the Funnel functionality and assign SIMs to it. To create a group and assign SIMs, take the following steps. First open the Air SIM administration page and select the Air SIM you want to communicate with Funnel with. Click the Execute button, then select Change Assigned Group.

Funnel

A dialog asking you to select the group with which the Air SIM is associated will appear. From the [New Group] dropdown, select [Create...]

Funnel

A group creation dialog like that seen below will appear. Enter the group name and click the Create Group button. In this example, we crete a group with the name "Kinesis_firehose"

Funnel

Register credentials

Begin by registering the credential set to access AWS. Log in to the User Console .

From the dropdown menu in the top right where your email is displayed, click Security. Funnel

The screen below will appear. From the menu at left, select Credentials and then click Register credentials.

Funnel

You will now have registered credential set.

The Credential Set is what will be specified by the Funnel service.
As needed, enter a description.
Select AWS Authentication Data for the type.
For the Role ARN and External ID, insert the information from creating an IAM Role

Funnel

This completes registration of the my-aws-credentials ID.

Funnel

Cloud service setup

Next, we specify a cloud service. The Funnel settings (cloud service set) are applied to the Air SIM group.

From the menu at the top of the screen, click Groups, then select the group you created. A screen like that seen below will be displayed.

Funnel

Open the Soracom Funnel Settings tab and specify a cloud service as below.

Select the resource type. Here we select Amazon Kinesis Firehose.
Enter the resource URL.
- For the US (Oregon) region, the resource URL is as follows:
```
https://firehose.us-west-2.amazonaws.com/<Delivery stream name>
```
For the Credential Set, select the AWS Credential set previously created.
Select JSON for the data format.

Funnel

This concludes the setup.

Using the Kinesis Firehose adapter to aggregate data on the cloud

We now send some data. Remember to assign the Soracom Air SIM card to the group you previously created.

Funnel

Send a request to the Funnel endpoint. The endpoints are as follows:

funnel.soracom.io:23080
funnel.soracom.io:80

Send over TCP

Transfer command example

$ nc funnel.soracom.io 23080
{"message":"Hello Soracom Funnel via TCP!"} [Enter]
200
[Ctrl+C]

Send over UDP

Transfer command example

$ nc -u funnel.soracom.io 23080
{"message":"Hello Soracom Funnel via UDP!"} [Enter]
200
[Ctrl+C]

Send over HTTP

Transfer command example

curl -vX POST \
>  -H 'Content-Type: application/json' \
>  -d '{
>        "message": "Hello Soracom Funnel via HTTP!"
>      }' \
>  http://funnel.soracom.io
>
>* Rebuilt URL to: http://funnel.soracom.io/
>*   Trying 100.127.65.43...
>* Connected to funnel.soracom.io (100.127.65.43) port 80 (#0)
>> POST / HTTP/1.1
>> Host: funnel.soracom.io
>> User-Agent: curl/7.43.0
>> Accept: */*
>> Content-Type:application/json
>> Content-Length: 44
>>
>* upload completely sent off: 44 out of 44 bytes
>< HTTP/1.1 204 No Content
>< Date: Thu, 14 Jan 2016 17:18:46 GMT
>< Connection: keep-alive
><
>* Connection #0 to host funnel.soracom.io left intact

Check aggregated data

Check the S3 Bucket previously specified as the Kinesis Firehose Destination. Check that a file has been created as below.

Funnel

{"operatorId": "OP0026965167", "timestamp": 1455672330494, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello Soracom Funnel via TCP!"}, "sourceProtocol": "tcp", "imsi": "440XXXXXXXXXX"}
{"operatorId": "OP0026965167", "timestamp": 1455672351248, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello Soracom Funnel via UDP!"}, "sourceProtocol": "udp", "imsi": "440XXXXXXXXXX"}
{"operatorId": "OP0026965167", "timestamp": 1455672363539, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello Soracom Funnel via HTTP!"}, "sourceProtocol": "http", "imsi": "440XXXXXXXXXX"}

This concludes the explanation of how to use the Kinesis Firehose adapter on Soracom Funnel to aggregate data (console version).

With Funnel, there is no need to load the SDK for the cloud service on your device. In addition, you do not need to maintain cloud service credentials on your device.

You can now simply specify the cloud service credentials and resource URL to directly send data from your device to the cloud service in question. This allows you to expend a minimum of effort to quickly build a system linking your devices with cloud services.

We plan to continue adding a range of features, so do not hesitate to share your requests with us.