Soracom Junction
Inspection Mode
Junction's Inspection mode will parse all packets passing through the VPG and send statistical information to a specified cloud service using a service adapter. From the cloud service, the data can then be connected with a tool in order to build visualizations and perform network analysis.
The following cloud service adapters are available:
- Amazon Kinesis Streams
- Amazon Kinesis Firehose
- AWS IoT
- Microsoft Azure Event Hubs
- Google Cloud Pub/Sub
Example Implementation
As each adapter provides access to different cloud services, implementation instructions will vary according to the data analysis tools you use. This guide demonstrates a practical implementation of Junction Inspection with the following architecture:
- Deliver packet statistics to Amazon Kinesis Stream.
- Add an Amazon Lambda function to format the data and transfer it to Elastic Cloud.
- Analyze the data using Elasticsearch.
- Build packet visualization with Kibana.
Requirements
- A Virtual Private Gateway
- An AWS account
- An Elastic Cloud account
Elastic Cloud Setup
Let's first start by setting up Elasticsearch and Kibana on Elastic Cloud. If you don't have an account, you can sign up for a 14-day trial .
-
Login to Elastic Cloud.
-
Click Create Cluster.
-
Enter the following options:
- Cluster Size -
1GBmemory and24GBstorage. - Region - Select a region near you.
Keep the defaults for the remaining options, then create the cluster.
- Cluster Size -
-
Once the cluster is created, a
elasticuser and its password will be displayed. Make a note of this information, as we will need it later. -
Next, from the Kibana section, click the Enable button.
Now we need to test that we can connect to Elasticsearch.
-
Open the cluster Overview page. Then from the Endpoints section, click HTTPS. Use the
elasticusername and password noted earlier to authenticate the connection. If successful, you should see a JSON response similar to the following:
-
We also need to check that we can access Kibana. Follow the same process, using the same
elasticusername and password from earlier.
Last, we need to register a template in Elasticsearch, which will tell Elasticsearch about the format of the statistical data that Junction will be sending.
-
Download the Soracom Realtime VPG Metrics template file to your computer.
-
Register it to Elasticsearch by making an HTTP PUT request to your cluster's HTTPS endpoint, specifying the
elasticusername and password, and using the template file as the HTTP request body:curl -X PUT \ > -H 'Content-Type:application/json' \ > --user elastic:<PASSWORD> \ > -d '@/path/to/soracom-realtime-vpg-metrics-template.json' \ > "https://xxxxxxxxxxxxxxxxxx.ap-northeast-1.aws.found.io:9243/_template/soracom-vpg"Elasticsearch will return a
{"acknowledged":true}response when the template is successfully registered.
Kinesis Stream Setup
Next, we need to set up a Kinesis Stream, which is where Junction will send Inspection data.
-
Login to the AWS Management Console . From the Services menu, open the Kinesis dashboard. Then click the Go to the Streams console button.
-
Click the Create Kinesis stream button.
-
Enter a name for this stream, such as
junction-inspection-escloud. We will keep the default settings for this stream. Then click the Create Kinesis stream button.
After the Kinesis stream has been created, we need to create credentials that Junction can use in order to connect to Kinesis. For security, we will create a new AWS IAM user with programmatic access which only has permission to access Kinesis.
-
From the AWS Services menu, open the IAM dashboard. Then from the Users, click the Add user button.
-
Enter a User name, and enable Programmatic access. Then click the Next: Permissions button.
-
Under the Set permission section, click Attach existing policies directly. Search for the
AmazonKinesisFullAccesspolicy and click its to attach it to the new user.
-
Continue the Add user wizard by clicking the Next buttons, keeping the default values for the remaining sections. Then click Create user to complete the wizard.
Once the new user has been created, it will appear along with its Access key ID and Secret access key. Save these keys in a secure location.
Lambda Setup
Now with Elasticsearch and Kinesis set up, we can create a Lambda function that will take care of sending our Inspection data from Kinesis to Elasticsearch.
In order to let Lambda access Kinesis, we need to set up a Role.
-
In the IAM dashboard, click the Roles section. Then click the Create new role button.
-
In the Select role type screen, select AWS Lambda.
-
In the Attach Policy screen, search for the
AWSLambdaKinesisExecutionRolepolicy and click its to attach it to the new role.
-
Last, enter a Role name. We will use this role when setting up Lambda next. Here, we use the name
lambda_kinesis_execution.
Now we can create the Lambda function.
-
From the AWS Services menu, open the Lambda dashboard. Click the Create a Lambda function button.
-
In the Select blueprint screen, choose Blank Function.
-
In the Configure triggers screen, select Kinesis, then select the
junction-inspection-escloudstream we created earlier. Then click Activate Trigger.
-
Now we can write the Lambda function that will take our data from Kinesis and send it to Elasticsearch. Copy and paste the Soracom Junction Lambda function . For runtime, select Python 3.6.
-
Rather than entering our Elasticsearch cluster endpoint and credentials directly in the function, let's add them as environment variables in the next section.
Environment variable Value ES_BASE_URLYour Elasticsearch cluster HTTPS endpoint, without the trailing /.ES_USERNAMEelasticES_PASSWORDThe password created earlier for the elasticuser.ES_INDEXA prefix added to the data. Set this as soracom, as this is the format the Elasticsearch template expects.ES_TYPEStats -
Then set configure this function to use the
lambda_kinesis_executionrole we created earlier. - Last, set the timeout to 1 minute. Then click Create function.
Configure Junction
So far, we've completed the following configuration:
- Elasticsearch - We registered the packet inspection template, and our cluster is ready to accept data at its endpoint.
- Kinesis stream - Our Kinesis stream is set up, and the IAM user we created to access it is ready.
- Lambda - Our function is ready to send data from our Kinesis stream to Elasticsearch.
Let's configure our VPG to send Inspection data to our Kinesis stream.
Follow the Inspection configuration instructions from the Junction Configuration documentation, providing the following parameters:
- Service - Amazon Kinesis Streams
- Destination - The endpoint of the Kinesis stream we created, in the format of
https://kinesis.<YOUR-AWS-REGION>.amazonaws.com/<DELIVERY-STREAM-NAME>, providing the AWS region where your Kinesis stream was created, and thejunction-inspection-esclouddelivery stream name we set earlier. - Credentials set - The IAM user Access key ID and Secret access key we created during Kinesis stream setup.
Once we save these settings, Junction will immediately begin sending Inspection data to our Kinesis stream. Our Lambda trigger will then send the data to Elasticsearch, and we can start to visualize the VPG traffic.
If any Air SIM subscribers were not previously attached to the VPG, you must disconnect and reconnect their cellular connection in order to connect to the VPG.
Kibana Configuration
Now that everything is hooked up, we just need to configure our Kibana visualization.
We need to define an Index Pattern that will match the data being sent from Lambda function.
-
Login to Kibana and click Management, then click Index Pattern.
-
Enter
soracom-*as the Index name or pattern. Then click the Create button.
Next, let's import a dashboard template that is pre-configured.
-
Download the Soracom Realtime VPG Metrics Dashboard template file.
-
From Kibana, click Management Saved Objects. Then click the Import button, and import the dashboard template.
Our Inspection configuration is now fully configured. As traffic passes through our Soracom VPG, Junction will take care of parsing the packets for statistical data. Kinesis, Lambda, and Elasticsearch will take care of handling the data, and we can now view realtime statistics from Kibana.
From Kibana, click Dashboard Soracom.
The dashboard template will include information about throughput, protocol, packet destination, and so on.
