Import/Export data from DynamoDB (small amounts)

I use this CLI command for exporting and then later importing data from/to DynamoDB (small amounts of data – eg. upto a few hundred items).

It does take a while to get data back into DynamoDB, as it’s doing it line-by-line, rather than as a batch … but gets the job done!

Export;

aws dynamodb scan --table-name source-table-name --no-paginate > data.json

Import;

cat data.json | jq -c '.Items[]' | while read -r line; do aws dynamodb put-item --table-name destination-table-name --item "$line"; done

This can be done in one line as well;

aws dynamodb scan --table-name source-table-name --no-paginate | jq -c '.Items[]' | while read -r line; do aws dynamodb put-item --table-name destination-table-name --item "$line"; done

Credit goes to; https://github.com/guillaumesmo

Cleaning up old DynamoDB Auto-Scaling Resources

I’ve found a strange problem with cloud-formation roll-backs which don’t automatically remove any Auto Scaling resources you might have setup.

This means then when you next deploy, CloudFormation starts complaining about resources already existing!

To clean these up, you need the following (run from the CLI, using the AWS CLI);

List resources;

aws application-autoscaling describe-scalable-targets --service-namespace dynamodb

From there, de-register (remove) each of the ones which shouldn’t be there;

aws application-autoscaling deregister-scalable-target --service-namespace dynamodb --resource-id "table/myTableName" --scalable-dimension "dynamodb:table:ReadCapacityUnits"

aws application-autoscaling deregister-scalable-target --service-namespace dynamodb --resource-id "table/myTableName" --scalable-dimension "dynamodb:table:WriteCapacityUnits"

That’s it!

AWS Invoking Lambda functions from CLI

aws lambda invoke \
  --function-name <your function name> \
  --payload '"<your json payload>"' \ 
  --cli-binary-format raw-in-base64-out /dev/stdout

The raw-in-base64-out lets you skip having to base64 encode the payload.

The /dev/stdout bit at the end just shows the output on your screen, rather than outputting it to a file and then having to read that file.

Ref;

  • https://docs.aws.amazon.com/cli/latest/reference/lambda/invoke.html
  • https://acloud.guru/forums/aws-lambda/discussion/-Lys1N6wVQCHE6Ucoxvt/getting-error-as-invalid-base64-for-the-same-data-provided-for-the-kinesis-lectu?answer=-M0gBMHWlAYMm9z1cv6i
  • https://stackoverflow.com/questions/47675032/invoking-aws-lambda-without-output-file

Unit-testing Bref lambda handlers

Hopefully this helps someone out there unit-testing Bref lambda consumers (eg. AWS lambda handlers for SNS / EventBridge / SQS, etc) with PHPUnit;

Essentially this includes the consumer (which is essentially a PHP function), and calls the function with array of event-data (in the same format AWS would normally give it).

The function (handler) would then return a response (hopefully with no thrown errors), and any unit-testing on the result would be done.

public function testConsumeUpdatePerson() {
        $handler = include(__DIR__ . '/../bin/consume');

        $data = json_encode([
            'action' => 'update-person',
            'id' => 1234
        ]);
        $overallJson = '{
  "Records": [
    {
      "EventVersion": "1.0",
      "EventSubscriptionArn": "arn:aws:sns:us-east-2:123456678:sns-lambda:abc-123",
      "EventSource": "aws:sns",
      "Sns": {
        "SignatureVersion": "1",
        "Timestamp": "2019-01-02T12:45:07.000Z",
        "Signature": "aaaabbbb/ccccdddd/111111==",
        "SigningCertUrl": "https://sns.us-east-2.amazonaws.com/SimpleNotificationService-aaaabbbb.pem",
        "MessageId": "aaaabbbbb",
        "Message": "' . addslashes($data) . '",
        "MessageAttributes": {
          "Test": {
            "Type": "String",
            "Value": "TestString"
          },
          "TestBinary": {
            "Type": "Binary",
            "Value": "TestBinary"
          }
        },
        "Type": "Notification",
        "UnsubscribeUrl": "https://sns.us-east-2.amazonaws.com/?Action=Unsubscribe&amp;SubscriptionArn=arn:aws:sns:us-east-2:111122222:test-lambda:aaaaa-bbbbb",
        "TopicArn" : "arn:aws:sns:ap-southeast-2:1111222222:topic-name-goes-here",
        "Subject": "TestInvoke"
      }
    }
  ]
}';
        $event = json_decode($overallJson, true);

        $response = $handler($event, new Context('', 300, '', ''));
        $this->assertEquals('OK', $response);
    }

More unit-tests can obviously be added below, but the basics of this test that there’s no errors, unhandled exceptions, etc which you hadn’t fully tested otherwise

Serverless – creating DNS entries for API Gateway

The following can be included in your serverless.yml file to create a sub-domain in Route53, and link it upto your API Gateway (HTTP) endpoint.

The following variables are required in your ‘custom’ block in the serverless.yml file;

  • certificate_arn – this is the ARN of your AWS Certificate Manager SSL certificate. For regional endpoints, this should be a cert created in the same region as your API Gateway.
  • domain_hosted_zone – the zone name of your domain name (eg. if your subdomain you want is abcd.myexample.com, the domain_hosted_zone will be example.com
  • domain_name – this is the complete sub-domain (eg. abcd.myexample.com)

Eg;

custom:
  domain_hosted_zone: 'example.com.'
  domain_name: 'abcd.example.com'
  certificate_arn: 'arn:aws:acm:ap-southeast-2:1233456:certificate/abc123'
resources:
  Resources:
    APIDomainName:
      Type: 'AWS::ApiGatewayV2::DomainName'
      Properties:
        DomainNameConfigurations:
          - CertificateArn: ${self:custom.certificate_arn}
        DomainName: ${self:custom.domain_name}

    APIDomainMapping:
      Type: 'AWS::ApiGatewayV2::ApiMapping'
      Properties:
        ApiId: !Ref HttpApi
        DomainName: !Ref APIDomainName
        Stage: !Ref HttpApiStage
      DependsOn: [ APIDomainName ]

    APIDomain:
      Type: AWS::Route53::RecordSetGroup
      Properties:
        HostedZoneName: ${self:custom.domain_hosted_zone}
        RecordSets:
          - Name: !Ref APIDomainName
            Type: A
            AliasTarget:
              DNSName: !GetAtt APIDomainName.RegionalDomainName
              HostedZoneId: !GetAtt APIDomainName.RegionalHostedZoneId

Ref; https://theburningmonk.com/cloudformation-ref-and-getatt-cheatsheet/

Including the git tag as an environment var in AWS Lambda (via Bitbucket Pipeline Deployments & serverless)

When deploying with the Serverless framework (which Bitbucket Pipelines can do), I wanted to include a version number (or other vars & options passed in the Serverless CLI) which triggered the deploy (via Bitbucket Pipelines).

In my case, this is shown in the footer of a Symfony web-app (more on that below).

Here’s how this can be achieved;

Serverless

In serverless.yml, we need to define our env-var within the function (or as i’ve done, for all functions, by placing it in the ‘provider’ -> ‘environment’ variables);

DEPLOY_VERSION: ${opt:deploy-version, 'unknown'}

In the above example, my ENV file will be called ‘DEPLOY_VERSION’

The ‘${opt:…} basically gets an option we’ve specified in the serverless deploy command-line (eg. serverless deploy –deploy-version v1.2.3 )

This allows us to pass environment vars from the command line, to our functions (in our case, we’re saying version 1.2.3 of our software is getting deployed).

Then, in Bitbucket;

Next, in our bitbucket-pipelines.yml file, we need to include some extra vars in the ‘atlassian/serverless-deploy:…’ pipe – eg;

EXTRA_ARGS: '... --deploy-version $BITBUCKET_TAG'

Here, we just specify our own option called ‘deploy-version’ (eg. ‘–deploy-version’), and used a variable which bitbucket includes at deploy-time (in our case, it’s called BITBUCKET_TAG).

In my case, i’m using tags to deploy new version of an app (eg. v1.2.3)

Using it with Symfony

From there, it’s upto you how your AWS Lambda function actually uses the environment variable. In my case, i’m using Symfony (with Bref to run it on Lambda). This requires an additional couple of steps;

In the .env file, I need to specify my default value for the env file (eg. when i’m developing it locally, etc);

DEPLOY_VERSION=dev-master

From there, in my case I then include it as a global variable in my templates, by adding it to my ‘config/packages/twig.yaml‘ file;

parameters:
    deploy_version: '%env(DEPLOY_VERSION)%'

twig:
    globals:
        deploy_version: '%deploy_version%'

And then in the footer of my pages, I can include it (eg. base.twig.html);

<p><small>Version: {{ deploy_version }}</small></p>

Done!

In summary, now when we deploy via Bitbucket Pipelines, we’ll have the version number used in the tag, included in our Symfony app (or whatever Lambda function you have).

Of course this could be used for any variable available in Bitbucket Pipelines (or event via the command-line in the Serverless framework)

Enjoy!

Serverless Framework / API Gateway Quirks

So, the Serverless framework is pretty awesome!

But … out of the box, it needs a few options setup to work as well as a regular server!

  • Compression
  • Serving binary files (images/pdf files/etc – stuff your app generates and tries to send to the user)

Binary files

By default API Gateway will have all sorts of encoding issues if you don’t set this up, and try to send binary files to your users. To set it up;

provider:
  apiGateway:
    binaryMediaTypes:
      - '*/*'

Compression

This is one which I hadn’t even thought of until I was browsing the site on a slowish connection!

By default content will be sent from API Gateway uncompressed. Whilst your users might not see much of a different, you could find yourself sending a lot more data than is needed (I had over a 10x saving in bandwidth … from 100kb to 6kb for JSON data).

To enable it, set;

provider:
  name: aws
  apiGateway:
    minimumCompressionSize: 1024

1024 (1kb) is used as a minimum size at which compression is used. You can set it to ‘0’ to compress everything, but the docs mention if you do-so, some small responses (less than 1kb) might actually be larger.

Ref;

Ref for these, and more options; https://serverless.com/framework/docs/providers/aws/events/apigateway/

Symfony with Bref (and AWS Lambda)

Bref is a serverless framework allowing you to use AWS Lambda with PHP sites, including Symfony apps.

In the serverless.yml file, include the following to allow the headers from the API Gateway to be picked up (eg. so Symfony knows you’re using https rather than http … and makes your absolute URLs with https accordingly);

provider:
  environment:
    TRUSTED_PROXIES: 127.0.0.1,127.0.0.2

Ref; https://symfony.com/doc/current/deployment/proxies.html

As part of this, some optimisations need to be made to the php.ini file, to get Symfony running a little faster;

; maximum memory that OPcache can use to store compiled PHP files
opcache.memory_consumption=256

; maximum number of files that can be stored in the cache
opcache.max_accelerated_files=20000

; don't check timestamps for php files in cache (comment out if php files are getting edited on the server)
; needs a clear-cache script prepared
opcache.validate_timestamps=0

; maximum memory allocated to store the results
realpath_cache_size=4096K

; save the results for 10 minutes (600 seconds)
realpath_cache_ttl=600

Ref; https://symfony.com/doc/current/performance.html

Adding the imagick extension to Bref

Recently the ‘GD’ extension was added to Bref, allowing images to be created.

Continuing on that theme, the ‘imagick’ extension was also going to be a nice addition.

To add it in, you need to do the following;

  1. Checkout bref from Github
  2. Add in the lines suggested here, into the ‘runtimes/php-intermediary.Dockerfile’; https://github.com/brefphp/bref/issues/267#issuecomment-476578009
  3. Go into the ‘runtimes’ dir and run ‘make layers’

This will make a few zip files in the ‘runtimes/export’ dir, which are the layers (runtimes) which need to be uploaded to Lambda. That can be done by;

aws lambda publish-layer-version --region=ap-southeast-2 --layer-name=php-73-fpm-with-imagick --zip-file fileb://php-73-fpm.zip --compatible-runtimes provided --output text
aws lambda publish-layer-version --region=ap-southeast-2 --layer-name=php-73-with-imagick --zip-file fileb://php-73.zip --compatible-runtimes provided --output text

Don’t forget to change the region name in the above commands!

Once that’s done, you’ll see the layers in the ‘Lambda’ ‘Layers’ page. Copy the ARN from there into your serverless .yml file, and away you go!

Don’t forget to enable the extension – this is done by creating a ‘php.ini’ file at /php/conf.d/php.ini (in your code-base), with the following;

extension=imagick

If you want to make sure it works, include the following in your PHP file and it’ll output a full list of info about the PHP environment, including a section on imagick if it’s been enabled & setup properly.

Enjoy!

Using AWS Cognito for authentication on your app

The aim here is to use AWS Cognito to authenticate users on your Symfony app, using oAuth2 so all the auth happens externally on AWS Cognito.

I’m not storing user data locally with this — it just makes sure that they’re valid users. Groups functionality would need to be added separately if required (it’s referenced in the MyBuilder article and has instructions on how groups can be obtained).

I’ve used the knpuniversity bundle as I tried to get HWI OAuth package to work but there wasn’t any providers already written up to support Cognito.

Composer packages used;

  • knpuniversity/oauth2-client-bundle
  • cakedc/oauth2-cognito

So – to get it going, follow the KNPUniversity instructions on their git-hub account.

From there, use the following for the Authenticator class;

<?php

namespace App\Security;

use KnpU\OAuth2ClientBundle\Client\ClientRegistry;
use KnpU\OAuth2ClientBundle\Client\OAuth2Client;
use KnpU\OAuth2ClientBundle\Security\Authenticator\SocialAuthenticator;
use Symfony\Component\HttpFoundation\RedirectResponse;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\RouterInterface;
use Symfony\Component\Security\Core\Authentication\Token\TokenInterface;
use Symfony\Component\Security\Core\Exception\AuthenticationException;
use Symfony\Component\Security\Core\User\UserProviderInterface;

class CognitoAuthenticator extends SocialAuthenticator
{
    private $clientRegistry;
    private $router;

    public function __construct(ClientRegistry $clientRegistry, RouterInterface $router)
    {
        $this->clientRegistry = $clientRegistry;
        $this->router = $router;
    }

    public function supports(Request $request)
    {
        // continue ONLY if the current ROUTE matches the check ROUTE
        return $request->attributes->get('_route') === 'connect_cognito_check';
    }

    public function getCredentials(Request $request)
    {
        // this method is only called if supports() returns true
        return $this->fetchAccessToken($this->getClient());
    }

    /**
     * @return OAuth2Client
     */
    private function getClient()
    {
        return $this->clientRegistry
            ->getClient('cognito');
    }

    public function getUser($credentials, UserProviderInterface $userProvider)
    {
        return $userProvider->loadUserByUsername($this->getClient()->fetchUserFromToken($credentials)->getId());
    }

    public function onAuthenticationSuccess(Request $request, TokenInterface $token, $providerKey)
    {
        $targetUrl = $this->router->generate('app_default_index');

        return new RedirectResponse($targetUrl);

        // or, on success, let the request continue to be handled by the controller
        //return null;
    }

    public function onAuthenticationFailure(Request $request, AuthenticationException $exception)
    {
        $message = strtr($exception->getMessageKey(), $exception->getMessageData());

        return new Response($message, Response::HTTP_FORBIDDEN);
    }

    /**
     * Called when authentication is needed, but it's not sent.
     * This redirects to the 'login'.
     *
     * @param Request $request
     * @param AuthenticationException|null $authException
     * @return RedirectResponse
     */
    public function start(Request $request, AuthenticationException $authException = null)
    {
        return new RedirectResponse(
            $this->router->generate('connect_cognito_start'),
            // might be the site, where users choose their oauth provider
            Response::HTTP_TEMPORARY_REDIRECT
        );
    }
}

My ‘SecurityCognitoController’ class looks like this;

<?php

namespace App\Controller;

use KnpU\OAuth2ClientBundle\Client\ClientRegistry;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;

class SecurityCognitoController extends AbstractController
{
    /**
     * Link to this controller to start the "connect" process
     *
     * @Route("/security/connect-cognito", name="connect_cognito_start")
     */
    public function connectAction(ClientRegistry $clientRegistry)
    {
        // will redirect to AWS Cognito!
        return $clientRegistry
            ->getClient('cognito') // key used in config/packages/knpu_oauth2_client.yaml
            ->redirect();
    }

    /**
     * After going to Facebook, you're redirected back here
     * because this is the "redirect_route" you configured
     * in config/packages/knpu_oauth2_client.yaml
     *
     * @Route("/security/cognito/check", name="connect_cognito_check")
     */
    public function connectCheckAction(Request $request, ClientRegistry $clientRegistry)
    {
        // ** if you want to *authenticate* the user, then
        // leave this method blank and create a Guard authenticator
        
    }
}

My security.yaml file looks something like this;

security:
    providers:
        oauth:
            id: knpu.oauth2.user_provider

    firewalls:
        main:
            anonymous: true
            guard:
                authenticators:
                    - App\Security\CognitoAuthenticator
    access_control:
        - { path: ^/security/connect-cognito, roles: IS_AUTHENTICATED_ANONYMOUSLY }

And the following config was used in the ‘pnpu_oauth2_client.yaml’ file;

knpu_oauth2_client:
    clients:
        # configure your clients as described here: https://github.com/knpuniversity/oauth2-client-bundle#configuration
        cognito:
            type:       'generic'
            provider_class:       '\CakeDC\OAuth2\Client\Provider\Cognito'
            client_id:  '<your client id here>'
            client_secret: '<your client secret here>'
            redirect_route: connect_cognito_check
            provider_options:
                region: <your region here>
                cognitoDomain: <your cognito domain here>
                scope: 'email'

A few things are above;

  • client id; This is found in the ‘App integration’ -> ‘App client settings’ page
  • client_secret: This is in the ‘General settings’ -> ‘App clients’ page, and generated when the ‘app client’ is added
  • region: the region your Cognito user-pool is in
  • scope: I’ve just used email, but if you want you can expand it to capture other information as well.
  • redirect_route: In my case I set it to ‘connect_cognito_check’ … this endpoint is used to receive the ‘OK’ from AWS Cognito that your user has been authenticated and pass back a code which internally is used to retrieve the actual account info of the person which was authenticated.

In AWS Cognito, in your ‘App client’ you’ve setup, make sure you have the following settings;

In AWS Cognito, in your ‘App client’ you’ve setup, make sure you have the following settings;

In this case, the ‘Authorization code grant’ is required as part of the oAuth process
The scopes ticked here are what you’ve specified in the config above.
The sign-in & out URLs should be roughly like the above. (obviously this is for dev … use a prod url for your prod environment.
The callback URLs should match your ‘check’ url

Thanks to the following sites which helped get this far;

https://tech.mybuilder.com/managing-authentication-in-your-symfony-project-with-aws-cognito/
https://github.com/knpuniversity/oauth2-client-bundle
https://github.com/CakeDC/oauth2-cognito