API Gateway integration with AWS Services.

In this blog post we will discuss on how API gateway can be used for integration with other AWS services such as dynamodb, lambda functions, step functions, SNS etc.

What is AWS API Gateway ?

AWS API Gateway us managed service that is highly available, scalable . API gateway provides easy integration with various AWS services like lambda, dynamodb, step functions and external/internal HTTP endpoints for integration purposes. It acts as entrypoint for your API traffic.
It provides features like authorization, throttling, CORS, caching, transformations, openAPI specs, custom domain names to name few.

API Gateway High Level Architecture

Integration Type

Lambda Function: lambda function can be integrated with API gateway in 2 different ways.

First called as Lambda Proxy and second one Lambda Direct Integration. In lambda proxy integration pattern, when client makes request to api, API Gw wraps body into events and some contextual data in context and pass it both event and context to lambda function. The response from lambda function is passed as-is back to api client which means lambda function will need to prepare response in way that includes status code, http headers, body etc.This is supported by HTTP and Rest api.

Second called Lambda Direct Integration in which case we can use VTL to perform some modifications on request body or headers and pass it lambda and then in response similar modifications can be done to response. VTL is bit complex and hard to understand at first place but its very powerful tool for message transformation and enrichment between request and response.

AWS Services Integration: This pattern allows you to integrate API Gw with various AWS Services directly without using lambda as middleware. This integration lets an API expose AWS service actions. In AWS integration, you will need to configure both the integration request and integration response and set up necessary data mappings from the method request to the integration request, and from the integration response to the method response.

HTTP Endpoints Integrations: In this pattern you place your API GW in front of http endpoints be it internal or external services. This can be either HTTP or HTTP_PROXY mode integration. In HTTP Proxy mode API Gateway passes the incoming request from the client to the HTTP endpoint and passes the outgoing response from the HTTP endpoint to the client.
In HTTP mode, you can use integration or VTL mapping to update request and responses.

Mock IntegrationThis type of integration lets API Gateway return a response without sending the request further to the backend. API response is defined at API gateway and used for mock responses for testing purposes.

Private Integrations: It allows you to integrate with services running in VPC by using VPC private links through network load balancer or to on premises services using AWS direct connect. You can also integrate with application load balancer using VPC private link in this case.

Now we will look at use case on how API gateway be integrated with AWS S3 bucket to list files, download or upload file to specific bucket. We will use CDK(if you want to know more about CDK,read it here https://dev.to/smkhub/aws-cdk--45l0

Complete CDK code in javascript for this use case can be found at GitHub repo https://github.com/SMKHub/cdk-s3-api

Create directory named "cdk-s3-api" and Initialise cdk app using "cdk init app --language=javascript" from cdk-s3-api directory.

In cdk-s3-app-stack.js file under constructor function first create S3 bucket, code below will create S3 bucket named s3-api-bucket with sane default values set by cdk as best practises.

   const s3ApiBucket = new s3.Bucket(this, 's3ApiBucket', {
      bucketName: 's3-api-bucket',
      removalPolicy: cdk.RemovalPolicy.DESTROY
   }
);

Create Rest API gateway with endpoint type as edge. Since we are dealing with s3 objects we will need to set values for "binaryMediaTypes". This is required to handle binary media types and for our API we are setting it to octet-stream and image/png types.

const restApi = new apigw.RestApi(this, 's3Api', {
      restApiName: "s3Api", 
      description: "Rest API for accessing S3 bucket and its object", 
      endpointConfiguration: {
        types:[ apigw.EndpointType.EDGE]
      }, 
      binaryMediaTypes: ["application/octet-stream", "image/png"]
    });

Next define AWS Integration with S3 service. Code below define following properties for integration.

• service : specifies name of AWS service to be integrated in our case it's s3
• path : s3 bucket name
• integrationHttpMethod : specifies integration method type.
• options. passthroughBehavior : Allows pass-through when the integration has NO content types mapped to templates.
• role to be used by API Gateway while making calls to S3 API's
• set integration request path bucket to method path folder from incoming api call.
• options. integrationResponses : Response that API Gateway provides after a method's backend completes processing a request with additional information mentioned below.
• set response headers to match integration for content type, content length and date headers. this is so that when client receives response it understand response content type as it might be xml, jpg image or normal text file stored in s3 bucket.

Note that setting up integration requests parameters and response parameters is necessary for API GW to successfully call S3 apis and send response back to client

const listBucketApiIntegration = new apigw.AwsIntegration({
      service:"s3", 
      region:"us-east-1", 
      path: s3ApiBucket.bucketName, 
      integrationHttpMethod: "GET", 
      options: {
        credentialsRole:this.apiGatewayRole, 
        passthroughBehavior: apigw.PassthroughBehavior.WHEN_NO_TEMPLATES,
        requestParameters: {'integration.request.path.bucket': 'method.request.path.folder'},
        integrationResponses: [
          {
            statusCode: '200', 
            responseParameters: { 
              'method.response.header.Date': 'integration.response.header.Date',
              'method.response.header.Content-Length': 'integration.response.header.Content-Length', 
              'method.response.header.Content-Type': 'integration.response.header.Content-Type'
            }
          }
        ]
      }
    });

Next add method to list bucket objects. Here we are using IAM authorization for making call to this http GET method.

bucketRootResource.addMethod('GET', listBucketApiIntegration, 
      { 
        authorizationType: apigw.AuthorizationType.IAM, 
        requestParameters: {
          'method.request.path.folder': true
        }, 
        methodResponses: [
          {
            statusCode: '200', 
            responseParameters: 
              {
                'method.response.header.Content-Type': true,
                'method.response.header.Date': true, 
                'method.response.header.Content-Length': true
              }
          }
        ]
      }
    );

Now we will add resource under root with path as "/{object}" and then define GET and PUT method separately for downloading object and uploading object to S3 bucket.

const bucketItemResource = bucketRootResource.addResource("{item}");
const getBucketItemApiIntegration = new apigw.AwsIntegration({
      service: "s3", 
      region:'us-east-1', 
      path: s3ApiBucket.bucketName+'/{object}',
      integrationHttpMethod: "GET", 
      options: {
        credentialsRole: this.apiGatewayRole, 
        passthroughBehavior: apigw.PassthroughBehavior.WHEN_NO_TEMPLATES, 
        requestParameters: {
          'integration.request.path.bucket': 'method.request.path.folder', 
          'integration.request.path.object': 'method.request.path.item',
          'integration.request.header.Accept': 'method.request.header.Accept'
        }, 
        integrationResponses: [          
            {
              statusCode: '200', 
              responseParameters: { 
                'method.response.header.Date': 'integration.response.header.Date',
                'method.response.header.Content-Length': 'integration.response.header.Content-Length', 
                'method.response.header.Content-Type': 'integration.response.header.Content-Type'
              }

          }
        ]
      }
    });

bucketItemResource.addMethod('GET', getBucketItemApiIntegration, 
      { 
        authorizationType: apigw.AuthorizationType.IAM, 
        requestParameters: {
          'method.request.path.folder': true,
          'method.request.path.item': true, 
          'method.request.header.Accept': true
        }, 
        methodResponses: [
          {
            statusCode: '200', 
            responseParameters: 
              {
                'method.response.header.Content-Type': true,
                'method.response.header.Date': true, 
                'method.response.header.Content-Length': true
              }
          }
        ]
      }
    );

Run cdk synth to validate if code is fine and then run cdk deploy to create stack resources in AWS account.

Integration response and request parameters that we have mapped in cdk code can be seen in API GW console as below.

Testing of API can be done using postman. Note that you will need to set access key and secret access key as API Gw authentication is set to IAM in CDK construct.

Finally don't forget to delete resources from AWS account using "cdk destroy" command. Note you might need to delete objects from S3 bucket manually before cdk destroy command if you have uploaded objects as part of testing API else stack deletion will fail stating S3 bucket is not empty.