Using Soracom Funnel to Store data in S3 through Amazon Kinesis Data Firehose

Introduction

In this guide, we create an Amazon Kinesis Data Firehose instance and an S3 bucket on AWS CloudFormation, then use the Soracom Funnel Amazon Kinesis Firehose Adapter to send data directly from your device to an S3 bucket.

We will be using Soracom Funnel along with the following AWS resources::

Click the links to read about these services in more depth.

Data Flow

  1. The Data sent from your devices will be received by Soracom Funnel.
  2. Funnel uses IAM Role authentication to send your data over to Amazon Kinesis Data Firehose.
  3. Amazon Kinesis Data Firehose relays the data to S3.
  4. When the data transfer to S3 fails, the error log is recorded on Amazon CloudWatch Logs, which will be available for 14 days after the error occurs.

Funnel

Step 1: Set up AWS Resources

You will first use AWS CloudFormation to configure and prepare all the AWS Resources used on this guide.

Prepare AWS Resources using CloudFormation

  1. Click here to Use the AWS CloudFormation Template.

    This template will default to the Tokyo Region. Change your region and use the template URL if you are using a different region.
    https://soracom-files.s3.amazonaws.com/cfn-samples/SoracomAwsCfnSampleFirehoseToS3Stack.yml

    For details on this template, refer to AWS CloudFormation Resource Stack Details in the References section below.

  2. Configure the settings below:

    Parameter Content
    CoverageAWSAccountID This parameter depends on your SIM's coverage:
    • JP Coverage: 762707677580
    • Global Coverage: 950858143650
    DeliveryStreamName Enter any stream name
    IamRoleExternalIdForFunnel Enter any external id, then save it somewhere since it will be used later
    (e.g. external-id-0123456789)
    IamRoleNameForFunnel Enter any IAM Role name
    S3BucketName Enter any S3 Bucket name


    https://aws.amazon.com

    Security

  3. Click the check box next to I acknowledge that AWS CloudFormation..., then click Create stack

    https://aws.amazon.com

    Security

    You have successfully set up AWS Resources.

Retrieve IAM Role ARN

  1. Navigate to AWS IAM Roles

    https://aws.amazon.com

    Security

  2. Search for and click the IAM Role you just created (enter the IAM Role Name you entered when you created the CloudFormation Resource Stack)

    https://aws.amazon.com

    Security

  3. Copy the Role ARN and save it for later

    https://aws.amazon.com

    Security

    At this point you should have an External ID and a Role ARN saved somewhere.

Step 2: Configure Soracom Funnel

Now that you have the AWS Resources set up, you will set up and enable Soracom Funnel.

Register your AWS IAM Role Credentials

  1. Login to your Soracom console, then click the top right corner

    https://console.soracom.io

    Security

  2. Click Security

    https://console.soracom.io

    Credentials

  3. Click the Credentials tab, then click the Register credentials button

    https://console.soracom.io

    Credentials

  4. Enter your IAM credentials as follows, then click Register:

    Parameter Content
    Credentials set ID Any name for this credentials set (i.e. AWS-IAM-Role-Soracom-Funnel-s3)
    Type AWS IAM Role
    Role ARN Role ARN of the IAM Role you created in Step 1　　　　　　　　　　　　　　　　 　
    External ID Your IAM Role's external id from Step 1


    https://console.soracom.io

    Credentials

    For a more detailed guide on this section, go to Creating a Credential Set, while still using the specifications above.

Enable Soracom Funnel

  1. Add all desired SIMs to a group, then click SORACOM Funnel.

    https://console.soracom.io

    Credentials

    Setting up Soracom Funnel is done at the group level. For more information on Groups and how to add your SIMs to a group, check the Groups section in the User-Console Guide.

  2. Turn Soracom Funnel on by clicking the switch

  3. Configure all the below settings for Funnel

    Parameter Content
    Service Amazon Kinesis Firehose
    Destination https://firehose.<Your Region code>.amazonaws.com/<DeliveryStreamName> (e.g. https://firehose.ap-northeast-1.amazonaws.com/funnel-delivery-stream)
    Credentials The credentials you just created
    Content Type JSON


    https://console.soracom.io

    Credentials

  4. Click Save

    https://console.soracom.io

    Credentials

    You have successfully set up and activated Soracom Funnel.

Step 3: Send Data to Soracom Funnel

Now that you have all the set up done, send a request to Funnel's entry point. We have provided the following examples for connecting with TCP, UDP, and HTTP.

nc funnel.soracom.io 23080
> {"temperature":20} [Enter]
>>  200
> [Ctrl+C]

Step 4: Confirm your Data in S3

Now that you've sent data through Soracom Funnel, let's actually confirm that the data arrived in S3.

  1. Open and Login to the AWS S3 Console

  2. Search for and click the S3 Bucket you created in Step 1

    https://aws.amazon.com

    Credentials

  3. Click the latest year, month, day, hour folders

    https://aws.amazon.com

    Credentials

  4. Click the newly created Object

    https://aws.amazon.com

    Credentials

  5. Click Download

    https://console.soracom.io

    Credentials

  6. Open the file and confirm that the data you provided through Soracom Funnel is in the file

    {"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "tcp", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"}
{"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "udp", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"}
{"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "http", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"}

    You have successfully stored data in S3 through Soracom Funnel and Amazon Kinesis Data Firehose.

References

AWS CloudFormation Resource Stack Details

Below are the details of the CloudFormation Resource Stack you created in Step 1.

AWS Resource Resource Name
Kinesis Data Firehose DeliveryStreamName from Step 1
S3 Bucket S3BucketName from Step 1
IAM Role IamRoleNameForFunnel from Step 1
IAM Role <DeliveryStreamName>-role
CloudWatch Logs Log Group <DeliveryStreamName>-log-group
CloudWatch Logs Log Stream <DeliveryStreamName>-log-stream

Terminate usage

When you want to stop using Funnel to store your data in S3, disable the Soracom Funnel service you enabled in Step 2, then simply delete the CloudFormation Resource Stack you made in Step 1.

  1. Navigate to the Amazon CloudFormation Resource Stack you created in Step 1

  2. Click Delete

    https://aws.amazon.com

    Credentials

    The S3 bucket will not terminate even if you delete the Stack.
    Visit the Amazon guide on Deleting a Bucket to delete the S3 Bucket resource.