Using the SORACOM Funnel Kinesis Firehose adapter to aggregate data on the cloud
This guide explains how to use the Amazon Kinesis Firehose adapter on SORACOM Funnel (hereafter, "Funnel") to aggregate data sent from a device on Amazon S3.
Funnel makes use of the SORACOM User Console for its settings.
Preface
Funnel is a cloud resource adapter that allows you to directly forward data from a device to a specific cloud service.
Simply indicating a resource on a cloud service allows you to directly aggregate data from your device. The major difference between Funnel and SORACOM Beam is that, while the forwarded destination is not a generic protocol, you can expend a minimum of time and effort to rapidly deploy use of cloud services.
Amazon Kinesis Firehose setup
If you have selected the Tokyo region, you will see the screen below. Select your region; here, we select US (Oregon).
Begin by setting up Amazon Kinesis Firehose. Select Kinesis from the AWS (Amazon Web Services) console.
Click Go to Kinesis Firehose.
From the Amazon Kinesis Firehose administration screen, select Create Stream.
Next, select Amazon S3 as the Destination and my-kinesis-firehose01 as the Delivery stream name. Then select S3 bucket.
Select the IAM role used by the Kinesis Firehose service. Here we select Firehose delivery IAM role.
If the screen below appears, allow access to the IAM role.
Confirm the settings and click Create Delivery Stream to create a Delivery Stream.
Next, create the IAM users that will access Kinesis Firehose. On the AWS IAM Management Console, select an existing or create a new user and attach the policy. We attach Full Access.
This concludes setup of the Amazon Kinesis Firehose cloud service.
SORACOM Funnel setup
SORACOM Funnel setup consists of two steps:
- Air SIM group creation and setup
- Register credentials
- Specify cloud service
Air SIM group creation and setup
Funnel requires that you configure Air SIMs on a group basis. You must first assign those Air SIMs to some sort of group. In this example, we create a dedicated group used for testing the Funnel functionality and assign SIMs to it. To create a group and assign SIMs, take the following steps. First open the Air SIM administration page and select the Air SIM you want to communicate with Funnel with. Click the Execute button, then select Change Assigned Group.
A dialog asking you to select the group with which the Air SIM is associated will appear. From the [New Group] dropdown, select [Create...]
A group creation dialog like that seen below will appear. Enter the group name and click the Create Group button. In this example, we crete a group with the name "Kinesis_firehose"
Pressing the Create Group button will take you back to the Change SIM Group dialog. A dropbox indicating New Group will appear; confirm that this is set to the newly-created "hello beam" group and click the Change Group button.
Register credentials
Begin by registering the credential data to access AWS. Log in to the User Console .
From the dropdown menu, click Security.
The screen below will appear. From the menu at left, select Store Authentication Data and then click Register Authentication Data.
You will now have registered authentication data.
- The Authentication ID is the authentication ID specified by the Funnel service.
- As needed, enter a summary here.
- Select AWS Authentication Data for the type.
- For the AWS Access Key ID and AWS Secret Access Key, use the IAM user data previously set.
This completes registration of the my-aws-credentials ID.
Cloud service setup
Next, we specify a cloud service. The Funnel settings (cloud service set) are applied to the Air SIM group.
From the menu at the top of the screen, click Groups, then select the group you created. A screen like that seen below will be displayed.
Open the SORACOM Funnel Settings tab and specify a cloud service as below.
- Select the resource type. Here we select Amazon Kinesis Firehose.
-
Enter the resource URL.
-
For the US (Oregon) region, the resource URL is as follows:
https://firehose.us-west-2.amazonaws.com/<Delivery stream name>
-
- For the authentication data, select the AWS authentication data previously created.
- Select JSON for the data format.
This concludes the setup.
Using the Kinesis Firehose adapter to aggregate data on the cloud
We now send some data. Remember to assign the SORACOM Air SIM card to the group you previously created.
Send a request to the Funnel endpoint. The endpoints are as follows:
- funnel.soracom.io:23080
- funnel.soracom.io:80
Send over TCP
- Transfer command example
$ nc funnel.soracom.io 23080
{"message":"Hello SORACOM Funnel via TCP!"} [Enter]
200
[Ctrl+C]
Send over UDP
- Transfer command example
$ nc -u funnel.soracom.io 23080
{"message":"Hello SORACOM Funnel via UDP!"} [Enter]
200
[Ctrl+C]
Send over HTTP
- Transfer command example
curl -vX POST \
> -H 'Content-Type: application/json' \
> -d '{
> "message": "Hello SORACOM Funnel via HTTP!"
> }' \
> http://funnel.soracom.io
>
>* Rebuilt URL to: http://funnel.soracom.io/
>* Trying 100.127.65.43...
>* Connected to funnel.soracom.io (100.127.65.43) port 80 (#0)
>> POST / HTTP/1.1
>> Host: funnel.soracom.io
>> User-Agent: curl/7.43.0
>> Accept: */*
>> Content-Type:application/json
>> Content-Length: 44
>>
>* upload completely sent off: 44 out of 44 bytes
>< HTTP/1.1 204 No Content
>< Date: Thu, 14 Jan 2016 17:18:46 GMT
>< Connection: keep-alive
><
>* Connection #0 to host funnel.soracom.io left intact
Check aggregated data
Check the S3 Bucket previously specified as the Kinesis Firehose Destination. Check that a file has been created as below.
{"operatorId": "OP0026965167", "timestamp": 1455672330494, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello SORACOM Funnel via TCP!"}, "sourceProtocol": "tcp", "imsi": "440XXXXXXXXXX"}
{"operatorId": "OP0026965167", "timestamp": 1455672351248, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello SORACOM Funnel via UDP!"}, "sourceProtocol": "udp", "imsi": "440XXXXXXXXXX"}
{"operatorId": "OP0026965167", "timestamp": 1455672363539, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose01", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello SORACOM Funnel via HTTP!"}, "sourceProtocol": "http", "imsi": "440XXXXXXXXXX"}
This concludes the explanation of how to use the Kinesis Firehose adapter on SORACOM Funnel to aggregate data (console version).
With Funnel, there is no need to load the SDK for the cloud service on your device. In addition, you do not need to maintain cloud service credentials on your device.
You can now simply specify the cloud service credentials and resource URL to directly send data from your device to the cloud service in question. This allows you to expend a minimum of effort to quickly build a system linking your devices with cloud services.
We plan to continue adding a range of features, so do not hesitate to share your requests with us.