Symfony with Bref (and AWS Lambda)

Bref is a serverless framework allowing you to use AWS Lambda with PHP sites, including Symfony apps.

In the serverless.yml file, include the following to allow the headers from the API Gateway to be picked up (eg. so Symfony knows you’re using https rather than http … and makes your absolute URLs with https accordingly);

provider:
  environment:
    TRUSTED_PROXIES: 127.0.0.1,127.0.0.2

Ref; https://symfony.com/doc/current/deployment/proxies.html

As part of this, some optimisations need to be made to the php.ini file, to get Symfony running a little faster;

; maximum memory that OPcache can use to store compiled PHP files
opcache.memory_consumption=256

; maximum number of files that can be stored in the cache
opcache.max_accelerated_files=20000

; don't check timestamps for php files in cache (comment out if php files are getting edited on the server)
; needs a clear-cache script prepared
opcache.validate_timestamps=0

; maximum memory allocated to store the results
realpath_cache_size=4096K

; save the results for 10 minutes (600 seconds)
realpath_cache_ttl=600

Ref; https://symfony.com/doc/current/performance.html

Adding the imagick extension to Bref

Recently the ‘GD’ extension was added to Bref, allowing images to be created.

Continuing on that theme, the ‘imagick’ extension was also going to be a nice addition.

To add it in, you need to do the following;

  1. Checkout bref from Github
  2. Add in the lines suggested here, into the ‘runtimes/php-intermediary.Dockerfile’; https://github.com/brefphp/bref/issues/267#issuecomment-476578009
  3. Go into the ‘runtimes’ dir and run ‘make layers’

This will make a few zip files in the ‘runtimes/export’ dir, which are the layers (runtimes) which need to be uploaded to Lambda. That can be done by;

aws lambda publish-layer-version --region=ap-southeast-2 --layer-name=php-73-fpm-with-imagick --zip-file fileb://php-73-fpm.zip --compatible-runtimes provided --output text
aws lambda publish-layer-version --region=ap-southeast-2 --layer-name=php-73-with-imagick --zip-file fileb://php-73.zip --compatible-runtimes provided --output text

Don’t forget to change the region name in the above commands!

Once that’s done, you’ll see the layers in the ‘Lambda’ ‘Layers’ page. Copy the ARN from there into your serverless .yml file, and away you go!

Don’t forget to enable the extension – this is done by creating a ‘php.ini’ file at /php/conf.d/php.ini (in your code-base), with the following;

extension=imagick

If you want to make sure it works, include the following in your PHP file and it’ll output a full list of info about the PHP environment, including a section on imagick if it’s been enabled & setup properly.

Enjoy!

Managing disk space on MySQL

Below are a bunch of helpful SQL queries to find tables and DB’s using up disk space;

This finds the largest 10 tables across your MySQL server;

SELECT CONCAT(table_schema, '.', table_name),
       CONCAT(ROUND(table_rows / 1000000, 2), 'M')                                    rows,
       CONCAT(ROUND(data_length / ( 1024 * 1024 * 1024 ), 2), 'G')                    DATA,
       CONCAT(ROUND(index_length / ( 1024 * 1024 * 1024 ), 2), 'G')                   idx,
       CONCAT(ROUND(( data_length + index_length ) / ( 1024 * 1024 * 1024 ), 2), 'G') total_size,
       ROUND(index_length / data_length, 2)                                           idxfrac
FROM   information_schema.TABLES
ORDER  BY data_length + index_length DESC
LIMIT  10;

This gets the size of each of your databases, ordered smallest to largest;

SELECT table_schema, CONCAT(ROUND( sum((data_length+index_length)/1024/1024)/1024, 2), 'G') AS MB from information_schema.tables group by 1 order by sum(data_length+index_length) asc;

Ref; https://serverfault.com/questions/676669/amazon-rds-instance-slowly-losing-free-diskspace

Using AWS Cognito for authentication on your app

The aim here is to use AWS Cognito to authenticate users on your Symfony app, using oAuth2 so all the auth happens externally on AWS Cognito.

I’m not storing user data locally with this — it just makes sure that they’re valid users. Groups functionality would need to be added separately if required (it’s referenced in the MyBuilder article and has instructions on how groups can be obtained).

I’ve used the knpuniversity bundle as I tried to get HWI OAuth package to work but there wasn’t any providers already written up to support Cognito.

Composer packages used;

  • knpuniversity/oauth2-client-bundle
  • cakedc/oauth2-cognito

So – to get it going, follow the KNPUniversity instructions on their git-hub account.

From there, use the following for the Authenticator class;

<?php

namespace App\Security;

use KnpU\OAuth2ClientBundle\Client\ClientRegistry;
use KnpU\OAuth2ClientBundle\Client\OAuth2Client;
use KnpU\OAuth2ClientBundle\Security\Authenticator\SocialAuthenticator;
use Symfony\Component\HttpFoundation\RedirectResponse;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\RouterInterface;
use Symfony\Component\Security\Core\Authentication\Token\TokenInterface;
use Symfony\Component\Security\Core\Exception\AuthenticationException;
use Symfony\Component\Security\Core\User\UserProviderInterface;

class CognitoAuthenticator extends SocialAuthenticator
{
    private $clientRegistry;
    private $router;

    public function __construct(ClientRegistry $clientRegistry, RouterInterface $router)
    {
        $this->clientRegistry = $clientRegistry;
        $this->router = $router;
    }

    public function supports(Request $request)
    {
        // continue ONLY if the current ROUTE matches the check ROUTE
        return $request->attributes->get('_route') === 'connect_cognito_check';
    }

    public function getCredentials(Request $request)
    {
        // this method is only called if supports() returns true
        return $this->fetchAccessToken($this->getClient());
    }

    /**
     * @return OAuth2Client
     */
    private function getClient()
    {
        return $this->clientRegistry
            ->getClient('cognito');
    }

    public function getUser($credentials, UserProviderInterface $userProvider)
    {
        return $userProvider->loadUserByUsername($this->getClient()->fetchUserFromToken($credentials)->getId());
    }

    public function onAuthenticationSuccess(Request $request, TokenInterface $token, $providerKey)
    {
        $targetUrl = $this->router->generate('app_default_index');

        return new RedirectResponse($targetUrl);

        // or, on success, let the request continue to be handled by the controller
        //return null;
    }

    public function onAuthenticationFailure(Request $request, AuthenticationException $exception)
    {
        $message = strtr($exception->getMessageKey(), $exception->getMessageData());

        return new Response($message, Response::HTTP_FORBIDDEN);
    }

    /**
     * Called when authentication is needed, but it's not sent.
     * This redirects to the 'login'.
     *
     * @param Request $request
     * @param AuthenticationException|null $authException
     * @return RedirectResponse
     */
    public function start(Request $request, AuthenticationException $authException = null)
    {
        return new RedirectResponse(
            $this->router->generate('connect_cognito_start'),
            // might be the site, where users choose their oauth provider
            Response::HTTP_TEMPORARY_REDIRECT
        );
    }
}

My ‘SecurityCognitoController’ class looks like this;

<?php

namespace App\Controller;

use KnpU\OAuth2ClientBundle\Client\ClientRegistry;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;

class SecurityCognitoController extends AbstractController
{
    /**
     * Link to this controller to start the "connect" process
     *
     * @Route("/security/connect-cognito", name="connect_cognito_start")
     */
    public function connectAction(ClientRegistry $clientRegistry)
    {
        // will redirect to AWS Cognito!
        return $clientRegistry
            ->getClient('cognito') // key used in config/packages/knpu_oauth2_client.yaml
            ->redirect();
    }

    /**
     * After going to Facebook, you're redirected back here
     * because this is the "redirect_route" you configured
     * in config/packages/knpu_oauth2_client.yaml
     *
     * @Route("/security/cognito/check", name="connect_cognito_check")
     */
    public function connectCheckAction(Request $request, ClientRegistry $clientRegistry)
    {
        // ** if you want to *authenticate* the user, then
        // leave this method blank and create a Guard authenticator
        
    }
}

My security.yaml file looks something like this;

security:
    providers:
        oauth:
            id: knpu.oauth2.user_provider

    firewalls:
        main:
            anonymous: true
            guard:
                authenticators:
                    - App\Security\CognitoAuthenticator
    access_control:
        - { path: ^/security/connect-cognito, roles: IS_AUTHENTICATED_ANONYMOUSLY }

And the following config was used in the ‘pnpu_oauth2_client.yaml’ file;

knpu_oauth2_client:
    clients:
        # configure your clients as described here: https://github.com/knpuniversity/oauth2-client-bundle#configuration
        cognito:
            type:       'generic'
            provider_class:       '\CakeDC\OAuth2\Client\Provider\Cognito'
            client_id:  '<your client id here>'
            client_secret: '<your client secret here>'
            redirect_route: connect_cognito_check
            provider_options:
                region: <your region here>
                cognitoDomain: <your cognito domain here>
                scope: 'email'

A few things are above;

  • client id; This is found in the ‘App integration’ -> ‘App client settings’ page
  • client_secret: This is in the ‘General settings’ -> ‘App clients’ page, and generated when the ‘app client’ is added
  • region: the region your Cognito user-pool is in
  • scope: I’ve just used email, but if you want you can expand it to capture other information as well.
  • redirect_route: In my case I set it to ‘connect_cognito_check’ … this endpoint is used to receive the ‘OK’ from AWS Cognito that your user has been authenticated and pass back a code which internally is used to retrieve the actual account info of the person which was authenticated.

In AWS Cognito, in your ‘App client’ you’ve setup, make sure you have the following settings;

In AWS Cognito, in your ‘App client’ you’ve setup, make sure you have the following settings;

In this case, the ‘Authorization code grant’ is required as part of the oAuth process
The scopes ticked here are what you’ve specified in the config above.
The sign-in & out URLs should be roughly like the above. (obviously this is for dev … use a prod url for your prod environment.
The callback URLs should match your ‘check’ url

Thanks to the following sites which helped get this far;

https://tech.mybuilder.com/managing-authentication-in-your-symfony-project-with-aws-cognito/
https://github.com/knpuniversity/oauth2-client-bundle
https://github.com/CakeDC/oauth2-cognito

PHP GD on AWS Lambda (using Bref)

Warning: This is still a work in progress — i’ve tried to get as far as I could to but ran out of time.

The current error is the following, which I think is due to the GD extension using config from a later version of PHP, rather than the version Bref uses;

PHP Warning: PHP Startup: Unable to load dynamic library 'gd' (tried: /opt/bref/lib/php/extensions/no-debug-zts-20180731/gd (/opt/bref/lib/php/extensions/no-debug-zts-20180731/gd: cannot open shared object file: No such file or directory), /opt/bref/lib/php/extensions/no-debug-zts-20180731/gd.so (/opt/bref/lib/php/extensions/no-debug-zts-20180731/gd.so: undefined symbol: executor_globals)) in Unknown on line 0

Basically the following needs to be done to get GD added as a layer (and thus, extension) available to PHP, within AWS Lambda (using PHP Bref).

  1. Compile GD as a ‘bundle’ (including all required libraries as part of it).
  2. Zip up the gd.so bundle file, along with any other libraries it requires
    This is unzipped in the /opt dir in lambda, so the directory structure in the Zip file should be setup in whatever way makes it all work
  3. Publish the zip file as a layer, which is then included after the bref layer in lambda

Compile GD

I followed this guide for the commands;

Download and compile the following required libraries;

  • libwebp (GD compiles, but then complains the library isn’t there when you try to use the GD PHP extension if you don’t include this library)
    libxpm (GD doesn’t seem to compile without it, and it’s not included using the AWS AMI by default)

Follow the instructions on each of their respective websites to build, and run the ‘./configure’ command with ‘–with-prefix=/opt/…’ (replacing ‘…’ with the name of the library)

Download PHP from the php.net site, (get the same version of PHP as Bref uses).

Unzip, and go into the /modules/gd dir (in your unzipped PHP dir)

Run the following to generate a ./configure command and make files

sudo phpize

The following configures and compiles the GD extension, followed by ‘make test’ to check the extension is working.

It references the ‘libwebp’ and ‘libxpm’ directories which are already compiled and output in the /opt/… dir (using ./configure –with-prefix=/opt/.. command)

sudo ./configure --with-php-config=/usr/bin/php-config --with-gd --with-freetype-dir=/usr --with-jpeg-dir=/usr --with-xpm-dir=/opt/libxpm/ --with-png-dir=/usr --with-webp-dir=/opt/libwebp/
sudo make
make test

Zip up the gd bundle

For me, I include the following in the zip file;

  • libwebp (GD compiles, but then complains the library isn’t there when you try to use the GD PHP extension if you don’t include this library)
  • libxpm (GD doesn’t seem to compile without it, and it’s not included using the AWS AMI by default)
  • bref/lib/php/extensions/no-debug-zts-20180731/gd.so
    This is my compiled GD extension, in this directory (to coincide with the extensions dir bref uses — found using phpinfo(); )

The following command creates a ‘gd.zip’ file with the above libraries & files;

zip -r gd.zip lib* bref

Publish the zip file as an AWS lambda layer

This part’s the easy one;

aws lambda publish-layer-version --layer-name php-7-3-gd --description "gd php extension" --zip-file fileb://gd.zip

The lambda layer name can be whatever you like, along with the description. The file part (.zip) refers to the zip file you’ve created, stored on your local machine

AWS will spit out some JSON, with the ‘LayerVersionArn’ as well — this is what you need to include in your list of Lambda layers for your function, eg;

     ....
     "LayerVersionArn": "arn:aws:lambda:ap-southeast-2:1234567:layer:php-7-3-gd:16", 
     "Version": 16, 
     "Description": "gd php extension",
     ....
}

Sticky Date Pudding

Ingredients;

  • 2 cups dates
  • 2 cups water
  • 1 tsp bicarbonate of soda
  • 280 grams of butter
  • 1 1/2 cups caster sugar
  • 3 eggs
  • 2 cups self-raising flour
  • 1 cup brown sugar
  • 1 cup cream

Method

  1. Place dates, water and bicarbonate of soda in a saucepan and simmer for 5 minutes
  2. Put aside
  3. Beat together 80g of butter, caster sugar and eggs
  4. Add date mixture to butter and sugar alternately with flour
  5. Cook in 3 litre casserole on a low heat for 1/2 hour with the vent closed and 1/2 hour with the vent open. Alternately cook in a 20 cm (8 inch) cake tin at around 160 degrees for approximately 45 mins to 1 hour.

Sauce

  1. Mix brown sugar, remaining 200g butter and cream in a saucepan on a low heat. 
  2. When mixed, bring to the boil until thickened.

Credit goes to Scott for this amazing recipe!

Using Symfony on Lambda

For those going down the server-less route and using Symfony, this will hopefully give you a decent starting point.

I’ve added a ‘part 2‘ to this post, with performance optimisations, as well as handling being behind the API Gateway.

This uses a project called Bref – their website is a great starting point and has loads of info.

Ingredients (what you’ll need on-hand);

Method (how do we do this);

A lot of these instructions come from https://bref.sh

For this example, we’ll use ap-southeast-2 as the region (as all my scripts are written using it)

IMPORTANT: If you decide to use a different region (eg. one closer to home/your users), make sure you use the same region for S3 as well as the runtime (mentioned later). Otherwise you’ll run into permission issues you’ll never be able to solve!

2) Jump into the project dir (eg. my-project), and install bref using composer;

1) Install the Symfony skeleton project (see https://symfony.com/doc/current/setup.html) & check to make sure it works as expected.

composer require mnapoli/bref

You may also need to run the following;

vendor/bin/bref init

3) Create an AWS S3 bucket to store your ‘packaged’ files;

aws s3 mb s3://symfony-lambda --region=ap-southeast-2

4) Create a deploy.sh executable file to save yourself some typing;

#!/bin/bash

# Package up your files and send it to an S3 bucket you're going to use;
sam package --output-template-file .stack-symfony.yaml     --s3-bucket symfony-lambda --region=ap-southeast-2

# Deploy (using cloud-formation, which will create your lambda function, etc)
sam deploy --template-file .stack-symfony.yaml --stack-name symfony-lambda --capabilities CAPABILITY_IAM --region=ap-southeast-2

5) Create a cloudformation template file (we’ll call this ‘.stack-symfony.yaml’ … if you want to call it something different, put in the new name in the above command instead);

 AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31


Description: ''
Outputs:
  DemoHttpApi:
    Description: URL of our function in the *Prod* environment
    Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/'


Globals:
    Function:
        Environment:
            Variables:
                APP_ENV: prod
Resources:
  Website:
    Properties:
      CodeUri: s3://<your s3 bucket here>/94ce2f32ac156cd3f06208d3e05dca8f
      Description: ''
      Events:
        HttpRoot:
          Properties:
            Method: ANY
            Path: /
          Type: Api
        HttpSubPaths:
          Properties:
            Method: ANY
            Path: /{proxy+}
          Type: Api
      FunctionName: symfony-website
      Handler: public/index.php
      MemorySize: 1024
      Layers:
        - arn:aws:lambda:ap-southeast-2:416566615250:layer:php-72-fpm:8
      Runtime: provided
      Timeout: 30
    Type: AWS::Serverless::Function
  MyFunctionCli:
    Properties:
      CodeUri: s3://<your s3 bucket here>/94ce2f32ac156cd3f06208d3e05dca8f
      FunctionName: symfony-console
      Handler: bin/console
      Timeout: 120  # in seconds
      Layers:
        - arn:aws:lambda:ap-southeast-2:416566615250:layer:php-72-fpm:8
        - arn:aws:lambda:ap-southeast-2:416566615250:layer:console:4
      Runtime: provided
    Type: AWS::Serverless::Function

Runtimes (basically the PHP executable which is needed by Lambda (PHP isn’t built into lambda, so we supply it as a ‘layer’). See; https://bref.sh/docs/runtimes/

In the file above, add in your s3 bucket name, and replace the ‘layers’ mentioned above with the later from your region (if you’ve decided to use a different one). See; https://bref.sh/docs/runtimes/

6) Run your ./deploy.sh script and follow the prompts if there’s any issues.

I ran into multiple permission issues which were all easily solved by giving access to the user i’d created for this project.

See the Symfony bref guide for more details if you get stuck; https://bref.sh/docs/frameworks/symfony.html

Cost;

This article is of course free, but there’s some on-going costs you’ll need to consider with Lambda. The few websites i’ve read regarding this seem to indicate it’s cheaper than running an EC2 instance (both from an actual cost and a time-cost in maintaining the thing), but of course you can do a lot of things with EC2!

The cost estimates i’m using come from US East (Ohio), and were taken on 8/Feb/2019;

  • Lambda (for your actual PHP server-side code)
    • $3.50/million requests – first 333 million requests/month
    • AWS Lambda – Compute Free Tier – 400,000 GB-Seconds – US East (Ohio)12.163 Lambda-GB-Second
    • AWS Lambda – Requests Free Tier – 1,000,000 Requests – US East (Ohio)
  • S3 (for storage of your packaged site, as well as any assets – css/images/js/uploaded files);
    • $0.005 per 1,000 PUT, COPY, POST, or LIST requests
    • $0.004 per 10,000 GET and all other requests
    • $0.023 per GB – first 50 TB / month of storage used
  • Some others include;
    • Bandwidth
    • CloudFront (for serving up your assets if you use it later on)
    • Relational Database Service (for a database if you need one)
    • Route 53 (for your DNS needs)
    • Simple Email Service (for sending emails)

Inspiration from this comes from this YouTube vid – well worth watching;

The tale of the Mexican Fisherman and the Investment Banker

(Author Unknown)

An American investment banker was at the pier of a small coastal Mexican village when a small boat with just one fisherman docked. Inside the small boat were several large yellowfin tuna. The American complimented the Mexican on the quality of his fish and asked how long it took to catch them.

The Mexican replied, “only a little while.”

The American then asked why didn’t he stay out longer and catch more fish?

The Mexican said he had enough to support his family’s immediate needs.

The American then asked, “but what do you do with the rest of your time?”

The Mexican fisherman said, “I sleep late, fish a little, play with my children, take siestas with my wife, Maria, and stroll into the village each evening where I sip wine, and play guitar with my amigos. I have a full and busy life.”

The American scoffed. “I have an MBA from Harvard, and can help you,” he said. “You should spend more time fishing, and with the proceeds, buy a bigger boat. With the proceeds from the bigger boat, you could buy several boats, and eventually you would have a fleet of fishing boats. Instead of selling your catch to a middle-man, you could sell directly to the processor, eventually opening up your own cannery. You could control the product, processing, and distribution,” he said. “Of course, you would need to leave this small coastal fishing village and move to Mexico City, then Los Angeles, and eventually to New York City, where you will run your expanding enterprise.”

The Mexican fisherman asked, “But, how long will this all take?”

To which the American replied, “Oh, 15 to 20 years or so.”

“But what then?” asked the Mexican.

The American laughed and said, “That’s the best part. When the time was right, you would announce an IPO, and sell your company stock to the public and become very rich. You would make millions!”

“Millions – then what?”

The American said, “Then you could retire. Move to a small coastal fishing village where you could sleep late, fish a little, play with your kids, take siestas with your wife, and stroll to the village in the evenings where you could sip wine and play guitar with your amigos.”

Ref; http://renewablewealth.com/the-parable-of-the-mexican-fisherman/

Reverting/resetting a GIT commit

If something gets committed to a GIT repo, the following command can be run to reverse it;

$ git reset HEAD~

Eg;

$ git commit -m "Something terribly misguided" # (1)
$ git reset HEAD~ # (2)
<< edit files as necessary >> # (3)
$ git add ... # (4)
$ git commit -c ORIG_HEAD # (5)

If the commit has already been pushed to your external repo, you’ll need to add ‘–force’ next time you push if the commit is rejected (the repo is probably going to be ‘ahead’ of the commit you’re trying to push to it);

$ git push origin master --force

Ref; https://stackoverflow.com/questions/927358/how-to-undo-the-most-recent-commits-in-git