Audio transcoding

!Historical walkthrough

Elastic Transcoder shut down on 13 November 2025. The Node.js 10 runtime, Serverless 1 configuration and AWS SDK v2 examples below also need migration before use today. Prices, limits and billing increments refer to the original tests.

For a side project I’m converting WebM audio files to MP3. I initially started doing this with Amazon Elastic Transcoder. But after doing the same with FFmpeg and Lambda Layers, my testing showed that the latter was substantially cheaper and ~2 times faster for short audio recordings (for ~3 minute, ~3 MB files).

※Just want to read the code?

See github.com/upstandfm/audio-transcoder.

Use case

My side project is a web app that allows users to record their voice so others can listen to it. In the app I use the MediaStream Recording API (aka Media Recording API) to easily record audio from the user’s input device. It works really well, and you don’t have to use any external libraries!

There’s one catch though. At the time of this writing it only works in Firefox, Chrome and Opera. It “sort of” works in Safari¹. Even though that’s a bit disappointing, I’m okay with that for my use case.

※Note

Safari has since enabled MediaRecorder by default.

So after I had built something functional that allowed me to record my voice, it turned out that the audio file I ended up with had to be transcoded if I wanted to listen to it across a wide range of browsers and devices.

What does transcoding even mean?

Before I can answer that, we need to explore what an audio file is.

We can think of an audio file like a stream of data elements wrapped in a container. This container is formally called a media container format. It’s basically a file format (think file type) that can store different types of data elements (i.e. bits).

The container describes how this data “coexists” in a file. Some container formats only support audio, like WAVE (usually referred to as WAV). Others support both audio and video, like WebM.

So a container “wraps” data to store it in a file, but information can be stored in different ways. We’ll also want to compress the data to optimize for storage and/or bandwidth by encoding it (i.e. converting it from one “form” to another).

This is where a codec (coder/decoder) comes into play. It handles all the processing that’s required to encode (compress) and decode (decompress) the audio data.

So when describing an audio or video file, we usually need to specify both its container and its codec². For example, AAC audio in an MP4 container is commonly stored as an .m4a file. An MP3 file contains MPEG-1 or MPEG-2 Audio Layer III frames: it does not need an MP4 container.

So what does transcoding mean? It’s the process of converting one encoding into another. If we convert one container format into another, this process is called transmuxing.

There are a lot of codecs available. Each codec will have a different effect on the quality, size and/or compatibility of the audio file³.

Why do you need to transcode audio?

You might be wondering (like I was), if we can record audio directly in the browser and immediately use the result in our app, why do we even have to transcode it?

The answer is: to optimize for compatibility. Because the Media Recording API can not record audio in all media formats.

For example, MP3 has good compatibility across browsers and devices for playback, but is not supported by the Media Recording API. What formats are supported depend on the browser’s specific implementation of said API.

We can use the isTypeSupported method to figure out if we can record in a specific media type by calling it with a MIME type. Run the following code in the web console (e.g. in Firefox) to see it in action:

MediaRecorder.isTypeSupported("audio/mpeg") // false

Okay, MP3 isn’t supported. Which format can we use to record in then? It looks like WebM is a good choice:

MediaRecorder.isTypeSupported("audio/webm") // true

Also note that you can specify the codec in addition to the container:

MediaRecorder.isTypeSupported("audio/webm;codecs=opus") // true

So if we want to end up with MP3 files of the recordings, we need to transcode (and technically also transmux) the WebM audio recordings.

How will we do this?

We’ll explore two implementations that both convert a WebM audio file to MP3:

1. Using Amazon Elastic Transcoder
2. Using FFmpeg and Lambda Layers

For both implementations we’ll use the Serverless Framework and Node.js to write the code for the Lambda function that converts an audio file.

Before we get started, make sure you have Node.js installed. Then use npm to install the Serverless Framework globally:

npm i -g serverless

Additionally, we’ll need two S3 buckets to process and store the converted audio files:

• An input bucket to upload WebM audio files.
• An output bucket to store transcoded MP3 files.

Using Amazon Elastic Transcoder

Amazon Elastic Transcoder is a fully managed and highly scalable AWS service that can be used to transcode audio and video files.

We can use this service to schedule a transcoding job in a pipeline. The pipeline knows from which bucket to read a file that needs to be converted, and to which bucket the converted file should be written. The job contains instructions on which file to transcode, and to what format it should be converted.

We’ll create a Lambda function that will “listen” to the S3 input bucket. Whenever a new object is created in that bucket, Lambda will schedule a transcoder job to create the MP3 file.

So the flow will be like this:

• A WebM audio file is uploaded to the input bucket.
• The Lambda function is triggered, and uses the key of the created S3 object to schedule a transcoder job.
• A job is scheduled in the pipeline, where Amazon Elastic Transcoder:
  • Fetches the WebM audio file from the input bucket.
  • Transcodes the WebM audio file to MP3.
  • Uploads the MP3 file to the output bucket.

!Warning

At the time of this writing AWS CloudFormation has no support for Amazon Elastic Transcoder. So you’ll have to use the AWS web console to create and configure your pipeline(s).

We’ll go through the following steps to get it up and running:

• 1. Create a pipeline
• 2. Choose a preset
• 3. Create an IAM Policy
• 4. Create a Serverless project
• 5. Implement the Lambda function
• 6. Release the Lambda function
• 7. Schedule a job

1. Create a pipeline

Navigate to the Elastic Transcoder service in the AWS web console. Select a region (we’ll use eu-west-1) and click on “Create New Pipeline”.

Create the pipeline and take note of the ARN and Pipeline ID. We’ll need both to configure the Lambda function later on.

2. Choose a preset

The pipeline we created in the previous step requires a preset to work. Presets contain settings we want to be applied during the transcoding process. Lucky for us, AWS already has system presets to convert to MP3 files.

In the web console, click on “Presets” and filter on the keyword “MP3”. Select one and take note of its ARN and Preset ID. We’ll also need these to configure the Lambda function.

3. Create an IAM Policy

AWS will already have created an IAM Role named Elastic_Transcoder_Default_Role. But in order for the pipeline to read objects from the input bucket and write objects to the output bucket, we need to make sure the role has the required permissions to do so.

Create a new IAM Policy with the following configuration:

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Action": "s3:GetObject",
7
      "Resource": "arn:aws:s3:::raw.recordings/*"
8
    },
9
    {
10
      "Effect": "Allow",
11
      "Action": "s3:PutObject",
12
      "Resource": "arn:aws:s3:::transcoded.recordings/*"
13
    },
14
    {
15
      "Effect": "Allow",
16
      "Action": "s3:ListBucket",
17
      "Resource": "arn:aws:s3:::transcoded.recordings"
18
    }
19
  ]
20
}

Make sure the resource ARNs of your input and output buckets are named correctly. After the Policy has been created, attach it to Elastic_Transcoder_Default_Role.

4. Create a Serverless project

Create a new project named “audio-transcoder”. Move into this directory and create a Serverless manifest in the project root:

1
service: audio-transcoder
2

3
provider:
4
  name: aws
5
  runtime: nodejs10.x
6

7
package:
8
  exclude:
9
    - ./*
10
    - ./**/*.test.js
11
  include:
12
    - node_modules
13
    - src

Add the Elastic Transcoder Pipeline ID, MP3 Preset ID and region (from step 1 and step 2) as environment variables:

1
service: audio-transcoder
2

3
provider:
4
  name: aws
5
  runtime: nodejs10.x
6
  environment:
7
    TRANSCODE_AUDIO_PIPELINE_ID: "1572538082044-xmgzaa"
8
    TRANSCODER_MP3_PRESET_ID: "1351620000001-300040"
9
    ELASTIC_TRANSCODER_REGION: "eu-west-1"
10

11
package:
12
  exclude:
13
    - ./*
14
    - ./**/*.test.js
15
  include:
16
    - node_modules
17
    - src

Use the Elastic Transcoder Pipeline ARN and MP3 Preset ARN (from step 1 and step 2) to configure the Lambda with the required IAM permissions, so it can create transcoder jobs:

1
service: audio-transcoder
2

3
provider:
4
  name: aws
5
  runtime: nodejs10.x
6
  environment:
7
    TRANSCODE_AUDIO_PIPELINE_ID: "1572538082044-xmgzaa"
8
    TRANSCODER_MP3_PRESET_ID: "1351620000001-300040"
9
    ELASTIC_TRANSCODER_REGION: "eu-west-1"
10
  iamRoleStatements:
11
    - Effect: Allow
12
      Action:
13
        - elastictranscoder:CreateJob
14
      Resource:
15
        - YOUR_PIPELINE_ARN # Replace this with the ARN from step 1
16
        - YOUR_PRESET_ARN # Replace this with the ARN from step 2
17

18
package:
19
  exclude:
20
    - ./*
21
    - ./**/*.test.js
22
  include:
23
    - node_modules
24
    - src

Finally, add the Lambda function definition. This Lambda will be executed whenever an object is created in the input bucket:

1
service: audio-transcoder
2

3
provider:
4
  name: aws
5
  runtime: nodejs10.x
6
  environment:
7
    TRANSCODE_AUDIO_PIPELINE_ID: "1572538082044-xmgzaa"
8
    TRANSCODER_MP3_PRESET_ID: "1351620000001-300040"
9
    ELASTIC_TRANSCODER_REGION: "eu-west-1"
10
  iamRoleStatements:
11
    - Effect: Allow
12
      Action:
13
        - elastictranscoder:CreateJob
14
      Resource:
15
        - YOUR_PIPELINE_ARN # Replace this with the ARN from step 1
16
        - YOUR_PRESET_ARN # Replace this with the ARN from step 2
17

18
package:
19
  exclude:
20
    - ./*
21
    - ./**/*.test.js
22
  include:
23
    - node_modules
24
    - src
25

26
functions:
27
  transcodeToMp3:
28
    handler: src/handler.transcodeToMp3
29
    description: Transcode an audio file to MP3
30
    events:
31
      - s3:
32
          bucket: "raw.recordings"
33
          event: "s3:ObjectCreated:*"
34
          existing: true

5. Implement the Lambda function

In order to match the Lambda function definition in the Serverless manifest, create a file named handler.js in src, and export a method named transcodeToMp3:

1
"use strict"
2

3
module.exports.transcodeToMp3 = async () => {
4
  try {
5
    // Implementation goes here.
6
  } catch (err) {
7
    console.log("Transcoder Error: ", err)
8
    throw err
9
  }
10
}

In the previous step we configured the Lambda to be executed whenever an object is created in the input bucket. This means that AWS will call the Lambda with an event message that contains a list of Records. Each Record will contain an s3 object with information about the s3:ObjectCreated event:

// "event" object:
{
  "Records":[
    // "Record" object:
    {
      "s3":{
        // Contains information about the "s3:ObjectCreated" event.
      }
    }
  ]
}

The s3 object will contain a property called key, which is the “name” of the file that was created in the input bucket. For example, if we upload a file named test.webm to the S3 bucket, the value of key will be the (URL encoded!) string test.webm.

You can see the entire event message structure in the AWS S3 docs.

Also be aware that you can get more than one Record. So always process all of them:

1
"use strict"
2

3
module.exports.transcodeToMp3 = async (event) => {
4
  try {
5
    for (const Record of event.Records) {
6
      const { s3 } = Record
7
      if (!s3) {
8
        continue
9
      }
10

11
      const { object: s3Object = {} } = s3
12
      const { key } = s3Object
13
      if (!key) {
14
        continue
15
      }
16

17
      const decodedKey = decodeURIComponent(key.replace(/\+/g, " "))
18
      // TODO: use "decodedKey" to schedule transcoder job.
19
    }
20
  } catch (err) {
21
    console.log("Transcoder Error: ", err)
22
    throw err
23
  }
24
}

Finally, initialize the transcoder client and schedule a transcoder job for every created object in the input bucket:

1
"use strict"
2

3
const ElasticTranscoder = require("aws-sdk/clients/elastictranscoder")
4

5
const { ELASTIC_TRANSCODER_REGION, TRANSCODE_AUDIO_PIPELINE_ID, TRANSCODER_MP3_PRESET_ID } =
6
  process.env
7

8
const transcoderClient = new ElasticTranscoder({
9
  region: ELASTIC_TRANSCODER_REGION,
10
})
11

12
module.exports.transcodeToMp3 = async (event) => {
13
  try {
14
    for (const Record of event.Records) {
15
      const { s3 } = Record
16
      if (!s3) {
17
        continue
18
      }
19

20
      const { object: s3Object = {} } = s3
21
      const { key } = s3Object
22
      if (!key) {
23
        continue
24
      }
25

26
      const decodedKey = decodeURIComponent(key.replace(/\+/g, " "))
27
      await transcoderClient
28
        .createJob({
29
          PipelineId: TRANSCODE_AUDIO_PIPELINE_ID,
30
          Input: {
31
            Key: decodedKey,
32
          },
33
          Outputs: [
34
            {
35
              Key: decodedKey.replace(/\.webm$/i, ".mp3"),
36
              PresetId: TRANSCODER_MP3_PRESET_ID,
37
            },
38
          ],
39
        })
40
        .promise()
41
    }
42
  } catch (err) {
43
    console.log("Transcoder Error: ", err)
44
    throw err
45
  }
46
}

You can read more about the createJob API in the AWS JavaScript SDK docs.

6. Release the Lambda function

In order to upload the Lambda to AWS, make sure you have your credentials configured. Then run the following command from the project root to release the Lambda:

sls deploy --region eu-west-1 --stage prod

7. Schedule a job

With everything up and running, we can now upload a WebM audio file to the input bucket to schedule a transcoder job. Navigate to the S3 service in the AWS web console:

• Select your input bucket.
• Click “Upload”.
• Add a WebM audio file.
• Click on “Upload” again.

This action will trigger an s3:ObjectCreated event. AWS will execute the Lambda function we deployed in the previous step, and it will schedule a transcoder job.

To get more information about a scheduled job, navigate to the Elastic Transcoder service in the AWS web console. Click on “Jobs”, select your pipeline and click “Search”. Here you can select a job to get more details about it.

If it has status “Complete”, there should be a file named test.mp3 in the output bucket!

Using FFmpeg and Lambda Layers

FFmpeg is a cross-platform solution that can be used to convert audio and video files. Since it’s a binary, we’ll use a Lambda Layer to execute it from the Lambda function.

What’s a Lambda Layer?

Lambda Layers allow us to “pull in” extra dependencies into Lambda functions. A layer is basically a ZIP archive that contains some code. In order to use a layer we first must create and publish one.

After we publish a layer we can configure any Lambda function to use it⁴. AWS will then extract the layer to a special directory called /opt. The Lambda function runtime will be able to execute it.

How different is this implementation?

We’re basically “swapping out” Amazon Elastic Transcoder with FFmpeg. Other than that the flow is still the same.

So since we’re still converting a WebM audio file to MP3 whenever it’s uploaded to the input bucket, we can reuse the Lambda from the previous implementation by making these changes:

• Replace Amazon Elastic Transcoder with FFmpeg.
• Within the Lambda we will:
  • Retrieve the WebM audio file from the input bucket whenever it’s uploaded.
  • Convert the retrieved WebM audio file to MP3 using FFmpeg.
  • Write the converted MP3 file to the output bucket.

We’ll apply these changes by going through the following steps:

• 1. Create and publish FFmpeg Lambda Layer
• 2. Update the Serverless manifest
• 3. Update the Lambda function
• 4. Release the updated Lambda function
• 5. Upload another WebM audio file
• 6. Optimize the Lambda function

1. Create and publish FFmpeg Lambda Layer

The Serverless Framework makes it very easy to work with layers. To get started create a new project named “lambda-layers”. Move into this directory and create a Serverless manifest in the project root:

1
service: lambda-layers
2

3
provider:
4
  name: aws
5
  runtime: nodejs10.x
6

7
package:
8
  exclude:
9
    - ./*
10
  include:
11
    - layers
12

13
layers:
14
  ffmpeg:
15
    path: layers
16
    description: FFmpeg binary
17
    compatibleRuntimes:
18
      - nodejs10.x
19
    licenseInfo: GPL v2+, for more info see https://github.com/FFmpeg/FFmpeg/blob/master/LICENSE.md

The layer is named ffmpeg and the path property dictates that the layer code will reside in a directory named layers. Match this structure in the project by creating that directory first.

Move into the layers directory and download a static build of FFmpeg from johnvansickle.com/ffmpeg⁵.

Use the recommended ffmpeg-git-amd64-static.tar.xz master build:

curl -O https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz

Extract the files from the downloaded archive:

tar -xvf ffmpeg-git-amd64-static.tar.xz

Remove the downloaded archive:

rm ffmpeg-git-amd64-static.tar.xz

Rename the extracted directory to ffmpeg, so it matches the configured layer name in the Serverless manifest. For example:

mv ffmpeg-git-20191029-amd64-static ffmpeg

You should now have the following files and folder structure:

lambda-layers
  ├── layers
  │   └── ffmpeg
  │       ├── GPLv3.txt
  │       ├── ffmpeg
  │       ├── ffprobe
  │       ├── manpages
  │       ├── model
  │       ├── qt-faststart
  │       └── readme.txt
  └── serverless.yml

Publish the layer by running the following command from the project root:

sls deploy --region eu-west-1 --stage prod

When Serverless finishes deploying, navigate to the Lambda service in the AWS web console and click on “Layers”. Here you should see the published layer. Click on it and take note of the ARN. We’ll need it in the next step.

2. Update the Serverless manifest

We’ll now be modifying the manifest file of the audio-transcoder project.

First change the environment variables, and add the names of your input and output buckets.

Then change the IAM permissions so the Lambda function can read from the input bucket and write to the output bucket.

Finally, change the Lambda function to use the FFmpeg layer with the ARN from the previous step:

1
service: audio-transcoder
2

3
provider:
4
  name: aws
5
  runtime: nodejs10.x
6
  environment:
7
    S3_INPUT_BUCKET_NAME: "raw.recordings"
8
    S3_OUTPUT_BUCKET_NAME: "transcoded.recordings"
9
  iamRoleStatements:
10
    - Effect: Allow
11
      Action:
12
        - s3:GetObject
13
      Resource: arn:aws:s3:::raw.recordings/*
14
    - Effect: Allow
15
      Action:
16
        - s3:PutObject
17
      Resource: arn:aws:s3:::transcoded.recordings/*
18

19
package:
20
  exclude:
21
    - ./*
22
    - ./**/*.test.js
23
  include:
24
    - node_modules
25
    - src
26

27
functions:
28
  transcodeToMp3:
29
    handler: src/handler.transcodeToMp3
30
    description: Transcode an audio file to MP3
31
    events:
32
      - s3:
33
          bucket: "raw.recordings"
34
          event: "s3:ObjectCreated:*"
35
          existing: true
36
    layers:
37
      - YOUR_FFMPEG_LAYER_ARN # Replace this with the ARN from step 1

3. Update the Lambda function

Since we have to read from the input bucket and write to the output bucket, replace the Elastic Transcoder client with the S3 client. Use the decodedKey to get the WebM recording from the input bucket:

1
"use strict"
2

3
const S3 = require("aws-sdk/clients/s3")
4
const { S3_INPUT_BUCKET_NAME, S3_OUTPUT_BUCKET_NAME } = process.env
5
const s3Client = new S3()
6

7
module.exports.transcodeToMp3 = async (event) => {
8
  try {
9
    for (const Record of event.Records) {
10
      const { s3 } = Record
11
      if (!s3) {
12
        continue
13
      }
14

15
      const { object: s3Object = {} } = s3
16
      const { key } = s3Object
17
      if (!key) {
18
        continue
19
      }
20

21
      const decodedKey = decodeURIComponent(key.replace(/\+/g, " "))
22
      const webmRecording = await s3Client
23
        .getObject({
24
          Bucket: S3_INPUT_BUCKET_NAME,
25
          Key: decodedKey,
26
        })
27
        .promise()
28
    }
29
  } catch (err) {
30
    console.log("Transcoder Error: ", err)
31
    throw err
32
  }
33
}

The S3 client returns an object that contains a Body property. The value of Body is a blob, which we’ll feed to the FFmpeg layer and convert it to MP3.

We’ll do this via a helper function that will spawn a synchronous child process which allows us to execute the ffmpeg “command” (provided by the FFmpeg layer):

1
"use strict"
2

3
const { spawnSync } = require("child_process")
4

5
module.exports = {
6
  convertWebmToMp3(webmBlob) {
7
    spawnSync(
8
      "/opt/ffmpeg/ffmpeg", // "/opt/:LAYER_NAME/:BINARY_NAME"
9
      [
10
        // FFmpeg command arguments go here.
11
      ],
12
      { stdio: "inherit" },
13
    )
14

15
    // Rest of the implementation goes here.
16
  },
17
}

For this implementation, we’ll give ffmpeg files to read and write. We’ll use a “special” directory called /tmp⁶ for this.

First write the WebM blob to /tmp so FFmpeg can read it. Then tell it to write the produced MP3 file back to the same directory:

1
"use strict"
2

3
const { spawnSync } = require("child_process")
4
const { writeFileSync } = require("fs")
5

6
module.exports = {
7
  convertWebmToMp3(webmBlob) {
8
    const now = Date.now()
9
    const input = `/tmp/${now}.webm`
10
    const output = `/tmp/${now}.mp3`
11

12
    writeFileSync(input, webmBlob)
13

14
    spawnSync("/opt/ffmpeg/ffmpeg", ["-i", input, output], {
15
      stdio: "inherit",
16
    })
17

18
    // TODO: cleanup and return MP3 blob.
19
  },
20
}

Now check that FFmpeg succeeded, read the produced MP3 file, and return the MP3 blob. Use a unique temporary directory and clean it in finally, so failures also release the temporary files:

1
"use strict"
2

3
const { spawnSync } = require("child_process")
4
const { mkdtempSync, readFileSync, writeFileSync, existsSync, unlinkSync, rmdirSync } =
5
  require("fs")
6

7
module.exports = {
8
  convertWebmToMp3(webmBlob) {
9
    const directory = mkdtempSync("/tmp/transcode-")
10
    const input = `${directory}/input.webm`
11
    const output = `${directory}/output.mp3`
12
    try {
13
      writeFileSync(input, webmBlob)
14

15
      const result = spawnSync("/opt/ffmpeg/ffmpeg", ["-i", input, output], {
16
        stdio: "inherit",
17
      })
18
      if (result.error) {
19
        throw result.error
20
      }
21
      if (result.status !== 0) {
22
        throw new Error(`FFmpeg failed: ${result.signal || result.status}`)
23
      }
24

25
      return readFileSync(output)
26
    } finally {
27
      for (const file of [input, output]) {
28
        if (existsSync(file)) unlinkSync(file)
29
      }
30
      rmdirSync(directory)
31
    }
32
  },
33
}

Finally, use the MP3 blob in the handler to write it to the output bucket:

1
"use strict"
2

3
const S3 = require("aws-sdk/clients/s3")
4
const ffmpeg = require("./ffmpeg")
5
const { S3_INPUT_BUCKET_NAME, S3_OUTPUT_BUCKET_NAME } = process.env
6
const s3Client = new S3()
7

8
module.exports.transcodeToMp3 = async (event) => {
9
  try {
10
    for (const Record of event.Records) {
11
      const { s3 } = Record
12
      if (!s3) {
13
        continue
14
      }
15

16
      const { object: s3Object = {} } = s3
17
      const { key } = s3Object
18
      if (!key) {
19
        continue
20
      }
21

22
      const decodedKey = decodeURIComponent(key.replace(/\+/g, " "))
23
      const webmRecording = await s3Client
24
        .getObject({
25
          Bucket: S3_INPUT_BUCKET_NAME,
26
          Key: decodedKey,
27
        })
28
        .promise()
29

30
      const mp3Blob = ffmpeg.convertWebmToMp3(webmRecording.Body)
31
      await s3Client
32
        .putObject({
33
          Bucket: S3_OUTPUT_BUCKET_NAME,
34
          Key: decodedKey.replace(/\.webm$/i, ".mp3"),
35
          ContentType: "audio/mpeg",
36
          Body: mp3Blob,
37
        })
38
        .promise()
39
    }
40
  } catch (err) {
41
    console.log("Transcoder Error: ", err)
42
    throw err
43
  }
44
}

Rethrowing errors lets Lambda treat failed work as a failure and apply its asynchronous retry behavior. S3 events can be delivered more than once, and a retry can repeat records already processed by this handler. A production version should make repeated processing safe, especially when scheduling a paid transcoding job. The finally block handles ordinary failures, but cannot run if Lambda terminates the invocation on timeout.

4. Release the updated Lambda function

Run the same command like before from the project root to release the Lambda:

sls deploy --region eu-west-1 --stage prod

5. Upload another WebM audio file

When Serverless is done deploying, upload another WebM audio file to the input bucket.

But nothing happens… Where’s the MP3 file?

Let’s find out why this is happening by checking the Lambda function’s log files in the AWS web console:

• Go to the Lambda service.
• Click on the audio-transcoder-prod-transcodeToMp3 function.
• Click on the “Monitoring” tab.
• Click the “View logs in CloudWatch” button.
• Select the latest log group.

Here you should see the logs of the Lambda function.

The logs tell us that FFmpeg is executing (hooray!) but that it doesn’t complete (boo!).

In the middle of the transcoding process the logs just say END. On the last line we see that the Lambda had a duration of 6006.17 ms.

What’s happening? The Lambda function takes “too long” to finish executing. This Serverless setup defaults to a timeout of 6 seconds⁷: AWS Lambda itself defaults to 3 seconds. After 6 seconds the Lambda function is still not done transcoding, so AWS terminates it.

How do we solve this? By optimizing the Lambda function!

6. Optimize the Lambda function

First let’s just set the timeout to a larger value. For example, 180 seconds. This way we can see how long it would actually take to complete the transcoding process:

functions:
  transcodeToMp3:
    timeout: 180

Deploy again. When Serverless is done, upload another WebM audio file, and check the logs.

This time we see FFmpeg completes the transcoding process and that the Lambda had a duration of 7221.95 ms. If we check the output bucket now, we’ll see the MP3 file!

Optimizing further

Transcoding the audio file in ~7 seconds isn’t bad. Actually, it’s very similar to Amazon Elastic Transcoder. But we can do better.

Something that’s very important when working with Lambda, is to always performance tune your functions. Or in other words, always make sure that a Lambda function has the optimum memory size configured.

This is important because when you choose a higher memory setting, AWS will also give you an equivalent resource boost (like CPU). This can reduce the Lambda function’s runtime duration. But cost depends on both allocated memory and billed duration, so faster execution only saves money if it outweighs the higher memory allocation.

By default a Lambda function has a memory setting of 128 MB. So lets increase it and compare results. A good strategy is usually to keep doubling memory and measure the duration. But for the sake of brevity, I’m jumping ahead to 2048 MB:

functions:
  transcodeToMp3:
    memorySize: 2048

Deploy again. When Serverless is done, upload another WebM audio file and check the logs.

Great, it’s even faster now! Does this mean we can just keep increasing the memory and reap the benefits? Sadly, no. There’s a tipping point where increasing the memory won’t make it run faster.

For example, increasing the memory to 3008 MB (the maximum memory limit at the time of this writing) will result in a similar runtime duration:

Memory 2048 MB

Memory 3008 MB

When done optimizing, make sure to apply a sensible value for the Lambda timeout. In this case, 6 seconds covered these test runs, though production needs headroom for slower inputs and cold starts.

Comparing costs

To compare costs between both implementations, I did a couple of test runs converting a 3-minute (2.8 MB) WebM audio file to MP3.

!Warning

The following comparison uses the 2019 rates and measured billed durations (including the 100 ms billing increments used then). It is by no means very extensive, and your mileage may vary. The free-tier calculations assume the monthly allowances are available to this workload. S3 storage and request costs are excluded.

Amazon Elastic Transcoder costs

Elastic Transcoder charged per minute of output audio (rounded up to whole minutes) with 20 free audio minutes per month. The rate used for these calculations in eu-west-1 was $0,00522 per output minute. Processing time affects how soon the file is ready, but is not the billing unit.

These are the timing results of the test runs:

So the average transcoding time of the audio file would be:

7638 + 6663 + 7729 + 6595 + 8752 + 7216 + 7167 + 6605 + 6718 + 8700 = 73 783 ms
73783 / 10 = 7378,3 ms
7378,3 / 1000 = 7,3783 sec

Let’s say we would be transcoding 100 000 of these audio files per month, with each output exactly 3 minutes long. That would amount to:

3 * 100 000 = 300 000 output minutes

Since we pay $0,00522 per minute, the costs without free tier would be:

300 000 * 0,00522 = $1 566

And with free tier it would cost:

(300 000 - 20) * 0,00522 = $1 565,8956

What about Lambda costs?

We’re using Lambda to schedule Amazon Elastic Transcoder jobs. So we also have to calculate those (minor if not negligible) costs.

The Lambda pricing page tells us we pay for the number of requests and the duration (which depends on memory setting).

The request rate was $0,20 per million requests, with 1 million requests free each month. For 100 000 invocations, that’s $0,02 without free tier, or zero with an unused allowance. We’ll calculate duration costs below and include the request charge in the final totals.

These are the Lambda durations (with 128 MB memory) for the accompanying transcoder test runs:

The average billed duration would be:

500 + 200 + 200 + 200 + 200 + 300 + 200 + 200 + 400 + 300 = 2700 ms
2700 / 10 = 270 ms
270 / 1000 = 0,27 sec

In eu-west-1 we pay $0,000 016 6667 for every GB-second (GB-s). That means we first have to calculate “how much” memory the Lambda function uses for its runtime duration.

For 100 000 transcoding jobs per month (with 128 MB memory) that would be:

100 000 * 0,27 = 27000 sec
(128 / 1024) * 27000 = 3375 GB-s

The free tier included 400 000 GB-s every month, so depending on your scale you may or may not have to include it in your calculations. But without free tier it would cost:

3375 * 0,000 016 6667 = $0,056 250 113

FFmpeg and Lambda Layers costs

These are the Lambda durations (with 2048 MB memory) of the test runs:

The average billed duration would be:

4100 + 3900 + 4000 + 3800 + 3900 + 3900 + 3900 + 3800 + 3900 + 3900 = 39100 ms
39100 / 10 = 3910 ms
3910 / 1000 = 3,91 sec

In eu-west-1 we pay $0,000 016 6667 for every GB-s. For 100 000 transcoding jobs (with 2048 MB memory) that would be:

100 000 * 3,91 = 391 000 sec
(2048 / 1024) * 391 000 = 782 000 GB-s

Without free tier it would cost:

782 000 * 0,000 016 6667 = $13,033 3594

With free tier it would cost:

(782 000 - 400 000) * 0,000 016 6667 = $6,366 6794

What about data transfer costs?

Otherwise data transferred into and out of Lambda functions will be charged at the EC2 data transfer rates as listed under the “Data transfer” section.

Putting it all together

Costs of transcoding 100 000 3-minute (2.8 MB) WebM audio files to MP3 per month: