Overview

Sometimes we need to let a Lambda know when a new file was created in an S3 bucket. That's can be the case when we have a data lake and we want to process those files. But beware, when we talk about a data lake, we mean a huge amount of data so we shouldn't start processing immediately a file is written since our capacity can be limited. In that case, we should use a queue mechanism but that is out of the scope of this post so Let's concentrate on our specific problem: trigger a Lambda from S3.

We could do this from the console using point and click but as a good practice, let's automate this provisioning with Cloudformation. This way we will be able to move our code across different accounts and deal with environments easily. The Cloudformation approach can be slow at the beginning but once you get familiar with it, you will accelerate your dev process. Another big advantage of Cloudfromation is versioning. Since our output is a yml file, it can be easily add it to a github repo. Thus, tracking changes becomes a piece of cake. However, you may resolve your problem first from the console and then translate it to Cloudformation

Let’s do it step by step, building small pieces of Cloudformation components, deploy them and then put all together in one file.

      
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  lambda-s3-trigger

  Sample SAM Template for lambda-s3-trigger

Globals:
  Function:
    Timeout: 3

Parameters:
  Environment:
    Type: String
    Description: Environment name. Example, staging, dev, prod, etc.

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Role:
        Fn::GetAtt:
          - "MyRole"
          - "Arn"
    Tags:
      Name: !Sub "${Environment}-my-function"


  MyRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: AccessToS3Notifications
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:GetBucketNotification'
                  - 's3:PutBucketNotification'
                  - "s3:GetObject"
                Resource: !Sub 'arn:aws:s3:::${AWS::AccountId}-${Environment}-my-bucket'

  MyBucket:
    Type: AWS::S3::Bucket
    DependsOn:
      - MyFunction
    Properties:
      # the bucket name has the account as a prefix since it has to be unique globally at AWS level
      BucketName: !Sub "${AWS::AccountId}-${Environment}-my-bucket"
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: 's3:ObjectCreated:*'
            Function: !GetAtt MyFunction.Arn

  PermissionForEventsToInvokeLambda:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt MyFunction.Arn
      Action: "lambda:InvokeFunction"
      Principal: "s3.amazonaws.com"
      SourceAccount: !Ref 'AWS::AccountId'

      
    

Build and deploy

      
sam build
sam deploy --guided
      
    

Photo by Lucas van Oort on Unsplash