Connect to private resources from API Gateway with VPC Link

Until recently it was not possible to reach services in your VPC’s private subnets from API  Gateway. This recently changed and API Gateway now supports Endpoints to Private
VPC’s . This means you can reach services in Private VPC’s without using custom headers or Lambda Proxy’s.

In this example we will create an API in API Gateway with some resources . When we access these resources, API Gateway will contact the backend which runs in a private subnet and returns the response back to the client.

The following endpoints will be created:

• / (returns the status of the API)
• /header (returns the host header of the request)
• /foo/* (example to proxy everything under /foo to the backend)

To establish this connection a Network Load Balancer (NLB) will be used. A Network Load Balancer operates at level 4 of the OSI model and therefore only uses the TCP protocol. We will use HTTP to communicate with our service In this example we use Nginx as a proxy to our application.

Network Load Balancer

Open the AWS EC2 console and create a new Load Balancer of the type Network Load Balancer

• Give it a name and choose Internal Facing (we don’t want to expose it publicly).
• Select TCP and Choose port 80 (this is the tcp port where we will receive connections on).
• In Availability Zones select the VPC and subnets where you want to deploy the NLB.

Configure routing

In the next step we need to create a Target Group (or use an existing one). In this example we will create a new target group. In the Health Check configuration we’ll use the same port as the traffic port.

NOTE: If your backend service is listening on another tcp port you’ll need to adjust this so that the health check doesn’t fail.

Register targets

In the next screen select the instances where you want to send traffic to. You can also skip this and add the instances later.

NOTE: Instead of a single instance you could also attach the Target Group to an Auto Scaling Group.

API Gateway

Now that we have our NLB configured we can configure our API Gateway.

Configure the VPC Link in API gateway

Open the API Gateway console and choose VPC Links. Create a new VPC Link and select the Target NLB we created earlier.

Create the API

Now we can create our API. The fasted way is to create a Swagger file and import that.

Example Swagger file:

{
  "swagger": "2.0",
  "info": {
    "version": "2017-11-17T04:40:23Z",
    "title": "MyAppAPI"
  },
  "host": "p3wocvip9a.execute-api.us-west-2.amazonaws.com",
  "basePath": "/",
  "schemes": [
    "https"
  ],
  "paths": {
    "/": {
      "get": {
        "produces": [
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "200 response",
            "schema": {
              "$ref": "#/definitions/Empty"
            }
          }
        },
        "x-amazon-apigateway-integration": {
          "responses": {
            "default": {
              "statusCode": "200"
            }
          },
          "uri": "http://flaskapp-api.vincent.cloudar.be",
          "passthroughBehavior": "when_no_match",
          "connectionType": "VPC_LINK",
          "connectionId": "${stageVariables.vpcLinkId}",
          "httpMethod": "GET",
          "type": "http_proxy"
        }
      }
    },
    "/header": {
      "get": {
        "produces": [
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "200 response",
            "schema": {
              "$ref": "#/definitions/Empty"
            }
          }
        },
        "x-amazon-apigateway-integration": {
          "responses": {
            "default": {
              "statusCode": "200"
            }
          },
          "uri": "http://flaskapp-api.vincent.cloudar.be/header",
          "passthroughBehavior": "when_no_match",
          "connectionType": "VPC_LINK",
          "connectionId": "${stageVariables.vpcLinkId}",
          "httpMethod": "GET",
          "type": "http_proxy"
        }
      }
    },
    "/foo/{proxy+}": {
      "get": {
        "produces": [
          "application/json"
        ],
        "parameters": [
          {
            "name": "proxy",
            "in": "path",
            "required": true,
            "type": "string"
          }
        ],
        "responses": {
          "200": {
            "description": "200 response",
            "schema": {
              "$ref": "#/definitions/Empty"
            }
          }
        },
        "x-amazon-apigateway-integration": {
          "responses": {
            "default": {
              "statusCode": "200"
            }
          },
          "uri": "http://flaskapp-api.vincent.cloudar.be/foo/{proxy}",
          "passthroughBehavior": "when_no_match",
          "connectionType": "VPC_LINK",
          "connectionId": "${stageVariables.vpcLinkId}",
          "httpMethod": "GET",
          "type": "http_proxy",
          "requestParameters": {
            "integration.request.path.proxy": "method.request.path.proxy"
          }
        }
      }
    }
  },
  "definitions": {
    "Empty": {
      "type": "object",
      "title": "Empty Schema"
    }
  }
}

Once created you should see your API Resources

Create VPCLink

To create the actual connection to our private subnets we need to create a VPC Link. In the API Gateway console and choose VPC Links. Create a new VPC Link and select the Target NLB we created earlier.

Deploy API

We can now deploy the API by running APIs => <your api name> => Actions => Deploy API . Choose a name for the stage (in this example we use ‘test’ ).

Once deployed you will see a ‘test’ stage under Stages. Navigate to the Stage Variables tab and add a new variable with the name vpcLinkId  and the value is the ID of the VPCLink we created earlier.

Create Custom Domainname

Because we are using Virtual Hosts in Nginx based on the incoming url we will have to make the API available on that URL. To do this we need to create a Custom Domain Name in the API Gateway Console.
In this example we are using flaskapp-api.vincent.cloudar.be

NOTE: Because Cloudfront is used for this, it can take a while until the Cloudfront Distribution is deployed.

We will also need to create a DNS record in Route 53 that will point to this Cloudfront Distribution. Use the Target Domain Name value.

Testing

The backend application used in this example is a simple Python Flask application which exposes some API endpoints:

Python Flask app:

#!/usr/bin/env python
# coding=utf-8
from flask import Flask, jsonify
from flask import request

app = Flask(__name__)

@app.route('/', methods=['GET'])
def get_health():
    return jsonify({'status': 'OK'})

@app.route('/header', methods=['GET'])
def get_header():
    header = request.headers['host']
    return jsonify({'header': header})

@app.route('/foo/bar', methods=['GET'])
def get_foobar():
    return jsonify({'foo': 'bar'})

@app.route('/foo/baz', methods=['GET'])
def get_foobaz():
    return jsonify({'foo': 'baz'})

if __name__ == '__main__':
    app.run(host='0.0.0.0', debug=True)

The application is proxied by Nginx.

NOTE: The server_name parameter is important as API Gateway will send the Endpoint URL configured in the Integration Request as host header.

Nginx config:

server {
    listen      80;
    server_name flaskapp-api.vincent.cloudar.be;
    charset     utf-8;

    location / {
        include 	   uwsgi_params;
    	uwsgi_pass         127.0.0.1:8080;
    }
}

Now let’s see if everything works.

[~] http <strong>https://flaskapp-api.vincent.cloudar.be/</strong>                                                                                   7:32:02 
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 20
Content-Type: application/json
Date: Mon, 16 Apr 2018 05:32:06 GMT
Via: 1.1 f3a5b0b3f0604afcc0fb3447f930ad0a.cloudfront.net (CloudFront)
X-Amz-Cf-Id: zhRmMp-MCnNoCOy-sP5ry-CIChZvhvo5p81MzBp8_HPf67UoC33Teg==
X-Cache: Miss from cloudfront
x-amz-apigw-id: Fa1FdHapDoEF9mg=
x-amzn-Remapped-Connection: keep-alive
x-amzn-Remapped-Content-Length: 20
x-amzn-Remapped-Date: Mon, 16 Apr 2018 05:32:06 GMT
x-amzn-Remapped-Server: nginx/1.12.1
x-amzn-RequestId: 7ffddb48-4137-11e8-9f89-cd357aeb7d73

{
    "status": "OK"
}

[~] http <strong>https://flaskapp-api.vincent.cloudar.be/header</strong>                                                                             7:32:06 
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 49
Content-Type: application/json
Date: Mon, 16 Apr 2018 05:33:42 GMT
Via: 1.1 777c0716c0ef8010208c3559195306d7.cloudfront.net (CloudFront)
X-Amz-Cf-Id: oIuKAnxpDyRN7h0wmTOfr5iXFFfrbypQRA7IaOR4_86dNtUf9OnxAg==
X-Cache: Miss from cloudfront
x-amz-apigw-id: Fa1UjEfYjoEFuGA=
x-amzn-Remapped-Connection: keep-alive
x-amzn-Remapped-Content-Length: 49
x-amzn-Remapped-Date: Mon, 16 Apr 2018 05:33:42 GMT
x-amzn-Remapped-Server: nginx/1.12.1
x-amzn-RequestId: b990526d-4137-11e8-bbf1-c7ecef1da020

{
    "header": "flaskapp-api.vincent.cloudar.be"
}

[~] http https://flaskapp-api.vincent.cloudar.be/foo/bar                                                                            7:33:42 
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 18
Content-Type: application/json
Date: Mon, 16 Apr 2018 05:34:06 GMT
Via: 1.1 a86da8347e06cd1a49dfa25142e0bbf8.cloudfront.net (CloudFront)
X-Amz-Cf-Id: HEAqrKKSqZnT2SVg7VUF8RINEJQp92sUiliTOyjCcm9JQOxlzoDdIw==
X-Cache: Miss from cloudfront
x-amz-apigw-id: Fa1YSGaojoEFYdA=
x-amzn-Remapped-Connection: keep-alive
x-amzn-Remapped-Content-Length: 18
x-amzn-Remapped-Date: Mon, 16 Apr 2018 05:34:06 GMT
x-amzn-Remapped-Server: nginx/1.12.1
x-amzn-RequestId: c7d93e62-4137-11e8-b8fa-852d561a1e18

{
    "foo": "bar"
}

Conclusion

If you want to use API Gateway but don’t want or can’t expose your backend services publicly VPC Link can be a good solution. However you’ll still need to find a solution to proxy the requests and do TLS/SSL offloading. Of course it will probably be a matter of time before AWS supports this natively.