[@aws-sdk/client-lambda] - Payload data in uint8array format - how to resolve data to expected output?

[NickLiffen]

Describe the bug

When using the new AWS Client Lambda NPM Module, and when invoking a lambda, data from Payload is returned in a Uint8Array. I can't find a way to resolve/get the data in the original JSON format that it should be?

Your environment

SDK version number

"@aws-sdk/client-lambda": "^3.12.0",

Is the issue in the browser/Node.js/ReactNative?

NodeJS

Details of the browser/Node.js/ReactNative version

Node 14.6.1

Steps to reproduce

Lambda A tried to invoke Lambda GITHUB_TOKEN_FUNCTION

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');

module.exports = {
  getGitHubToken: async () => {
    const client = new LambdaClient({ region: process.env.REGION });

    const params = {
      FunctionName: process.env.GITHUB_TOKEN_FUNCTION,
      LogType: 'Tail',
      Payload: '',
    };

    const command = new InvokeCommand(params);

    try {
      const { Payload } = await client.send(command);
      console.log(Payload)
      return str;
    } catch (error) {
      console.log(error.message);
      throw error;
    }
  },
};

GITHUB_TOKEN_FUNCTION should return this:

const res = {
      statusCode: 200,
      body: JSON.stringify({
        token,
      }),

However, when I console.log(Payload) I get this:

In the old module (v2) I never got this, I just got the data back as returned from the GITHUB_TOKEN_FUNCTION. Which should be:

const res = {
      statusCode: 200,
      body: JSON.stringify({
        token,
      }),

I have checked the CloudWatch logs for GITHUB_TOKEN_FUNCTION and it is successfully sending that object. Lambda A just receives it in a different format that I can't seem to get into the expected one.

Observed behavior

I get some sort of Unit8Array back that I can't find out how to turn into the data I expect.

Expected behavior

Payload ===

const res = {
      statusCode: 200,
      body: JSON.stringify({
        token,
      }),

My Question:

How do I resolve data from Unit8Array to the data I was expecting from the Lambda response? Which is a JSON Object?

[NickLiffen]

Okay, I got it working:

const asciiDecoder = new TextDecoder('ascii');
const data = asciiDecoder.decode(Payload);

A question, why is this module not doing this 🤔 / why is there not an option to do this? 🤔 The reason we use this module is to not have to interact with the apis and services natively. It would be good if this module handled it.

[NickLiffen]

Sorry to keep on moaning here but also it makes testing really hard 🤔 We use Jest and there are a bunch of problems with TextDecode and TextEncode. As found here and here.

[NickLiffen]

I have looked at the "style" of this library and I think It isn't intended for the module to "by default" decode for the user. I think when it comes to Lambda Invoke especially; it would be good if you could maybe pass in a boolean or something to:

    const params = {
      FunctionName: process.env.LAMBDA_ARN,
      LogType: 'Tail',
      Payload: '',
    };

called: decode: true/false.

The value of this module is the abstraction and I think this would really help out people using this module 👍 Just a thought.

[NickLiffen]

I see there is some sort of encoder interface, is this exposed by any chance to the client so I can run this within my code?

[plummet555]

I came across the same issue. It does seem odd that we have to decode and then re-parse the output like this:
const asciiDecoder = new TextDecoder('ascii');
const creds = JSON.parse(asciiDecoder.decode(data.Payload));

[ffxsam]

Just a slight correction to the above (for those using Node.js, where ascii doesn't work as an argument):

const asciiDecoder = new TextDecoder('utf-8');

This method works too, and requires a little less code:

const creds = JSON.parse(Buffer.from(data.Payload)); // if using TypeScript, you'll need to use Buffer.from(data.Payload).toString() to be type-safe

I agree, I'm not sure why the decision was made to use Uint8Array. It's really inconvenient.

[ajredniwja]

@NickLiffen thanks for opening this issue, apologies for late reply, I think this is the expected behavior for this SDK as you can see payload is now of type unit8array

aws-sdk-js-v3/clients/client-lambda/models/models_0.ts

Line 3490 in 3bd30bb

Payload?: Uint8Array;

.

I have verified that the @ffxsam code which seems to work.

With that being said, I think this behavior can be documented but I believe its not a bug. I will mark it as a documentation issue.

[dgurns]

Agreed with the above posters. In a JavaScript/TypeScript environment, being able to access the payload data as JSON, even by passing in a flag, would be a nice quality of life improvement.

[toto922]

This offical util can be uesd to decode and encode uint8array format ↓

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_util_utf8_node.html

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_util_utf8_browser.html

[sebastien-abbal]

For typescript, I don't have errors if I use that kind of payload :

lambda.invoke({
  FunctionName: 'my-lamba',
  InvocationType: 'RequestResponse',
  Payload: Buffer.from(JSON.stringify({ foo: 'bar' })),
});

Hope it will be usefull :)

[iRoachie]

Can anyone shed light on this decision? @ajredniwja Seems very unorthodox especially coming from v2.

[juicycool92]

i still want to hear why they decide to use uint8array instead string. :/
does this changed type cost less or something?

[l3wy]

This still isn't documented (or at least well documented). The migration from v2 to v3 page indicates that it should be mostly a transparent migration, no mention of breaking changes (which this definitely is).

That said, I don't agree that this is a documentation issue. Just because someone decided to define the Payload type as uint8array in an output interface doesn't mean that it was an appropriate decision to make. It should have defaulted to the old Payload format and had an optional encoding param if you desired the output (or wanted to provide the input) in a different way.

Also worth noting that the input payload can still be provided as a string (which further confuses the issue).

[Unintendedz]

For me who is new to cloud development, it is very frustrating to not be able to find the correct example from SDK Document and have to go through search engines for lambda payload Uint8Array and find the solution from the comments of this issue.

[kuhe]

The reason for the response type not being string is because Lambda functions are user controlled and may return a response that should not be converted into a string.

Upstream, AWS Lambda's definition for the InvocationResponse::Payload field is Blob, and not string. While AWS SDK JS v2 was more convenient for string responses, this did not support other response types. V3 aims to both support other response types and more closely follow the AWS service's definitions for input and output types.

As to how we could make this more convenient, there was a similar issue with S3::getObject #1877 returning a stream, described in README for which the solution was to apply a "mixin" that added helper methods to convert the response to a string. Something similar could be done here.

• add mixin to help convert Blob responses such as Lambda::invocationResponse::Payload to string Add mixin to help convert blobs to string #4265

---

For now, please use one of the suggestions above (@ffxsam) to convert the value to a string.

Where data is the invocation response,

// browser
const textDecoder = new TextDecoder('utf-8');
const object = JSON.parse(textDecoder.decode(data.Payload));

// Node.js
const object = JSON.parse(Buffer.from(data.Payload).toString());

Omit or replace JSON.parse if you are not using a JSON serialization.

[RanVaknin]

Seems like @kuhe answered the question. I feel like this can be closed. If more questions arise, please feel free to open a new issue.

Thanks,
Ran~

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

[kuhe]

in a recently released version https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.357.0, Lambda::invoke and all other payload blobs are no longer restricted to Uint8Array, and can be sent in as strings, and converted to strings on output too:

import { InvokeCommand, LambdaClient } from "@aws-sdk/client-lambda";
import { Uint8ArrayBlobAdapter } from "@aws-sdk/util-stream";

const lambda = new LambdaClient({
  region: "us-east-1",
});

const invoke = lambda.send(
  new InvokeCommand({
    FunctionName: "echo",
    Payload: JSON.stringify({
      hello: "world",
    }),
  })
);

const payload: Uint8ArrayBlobAdapter = (await invoke).Payload!;

console.log(JSON.parse(payload?.transformToString() ?? "{}")); // { "hello": "world" }