How to use this package to AWS Lambda?

[cjnoname]

We’re interested in using this package within an AWS Lambda environment, but we’re unsure whether it’s compatible or how to properly deploy it.

Is there any documentation or guidance on how to get started with Lambda support for this package?

Thanks in advance!

[felixmagnus]

We're interested in this as well.

Currently, our lambdas error as soon as the '@confluentinc/kafka-javascript' package is imported.

This is the error:

INIT_REPORT Init Duration: 1652.67 ms	Phase: init	Status: error	Error Type: Runtime.Unknown

To me, it reads like the runtime of the lambda is incompatible with the package.

This is our lambda runtime configuration:

• Memory 1024 MB
• Runtime: Node.js 22.x
• Architecture: arm64

I can find Node 22 and arm64 on here, https://docs.confluent.io/kafka-clients/javascript/current/overview.html but I'm not sure if that matches lambdas architecture...

[milindl]

I have two preliminary suggestions:

1. is it possible to increase the verbosity of the available logging? The error is quite uninformative
2. Seeing this document https://docs.aws.amazon.com/lambda/latest/dg/nodejs-package.html#nodejs-package-create-dependencies it seems like you have to make a zip file with the installed node modules that's sent to Lambda. While installing, our package checks the current architecture to determine which prebuilt binary to fetch.
   So, for instance, you create your zip file on a macOS system with Node 18, it won't work on linux with Node 22. If this is the case, then you could use a docker image with the right OS, node version, and architecture to create the zip file for submission. I've seen similar errors earlier in different contexts.

[cjnoname]

Hi @milindl , thanks for the response. I hope there’s a guideline in the documentation that shows how to deploy to Lambda—either including the steps to create a layer or providing an official layer for people to use.

[felixmagnus]

Hi @milindl

is it possible to increase the verbosity of the available logging? The error is quite uninformative

Good suggestion, thanks! I increased the verbosity to the maximum possible for lambda. See attached logs at the end.

[...] So, for instance, you create your zip file on a macOS system with Node 18, it won't work on linux with Node 22. [...]

That's also a good hint, thanks a lot. All our lambdas are built in github actions, using the following runner configuration:

• OS: Ubuntu 24.04.2 LTS
• image: ubuntu-24.04 - 20250527.1.0
• Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250527.1/images/ubuntu/Ubuntu2404-Readme.md
• Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250527.1

We use the same node version at buildtime & at runtime (node 22).

I tried both possible lambda runtime architectures (arm64 and x86_64) and both do currently not work (and produce the same error).

I also tried both available bundling options for the lambda (CJS and ESM, ESM being our default) and they produce different errors currently.

Lambda logs (JSON format) for the ESM bundling (same for arm64 and x86_64):

{
    "time": "2025-06-02T08:33:13.231Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:e4b97d4055c2afa813f5bda7d8e92ef971a7605f10be17ef7ae967743a21ba7d",
        "functionName": "<FUNCTION_NAME>",
        "functionVersion": "$LATEST"
    }
}
file:///var/task/index.mjs:431265
if (fileName !== __filename) {
^
ReferenceError: __filename is not defined
(Use `node --trace-uncaught ...` to show where the exception was thrown)
Node.js v22.15.0
{
    "time": "2025-06-02T08:33:16.034Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "status": "error",
        "errorType": "Runtime.ExitError"
    }
}
{
    "time": "2025-06-02T08:33:16.034Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "status": "error",
        "errorType": "Runtime.ExitError",
        "metrics": {
            "durationMs": 2803.685
        }
    }
}
{
    "time": "2025-06-02T08:33:16.034Z",
    "type": "platform.start",
    "record": {
        "requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
        "functionArn": "<FUNCTION_ARN>",
        "version": "$LATEST",
        "tracing": {
            "spanId": "69765a8e17d79988",
            "type": "X-Amzn-Trace-Id",
            "value": "Root=1-683d61c9-3a9685c768d1be1f76d7404a;Parent=27d52784c5996cbf;Sampled=1"
        }
    }
}
{
    "time": "2025-06-02T08:33:16.035Z",
    "type": "platform.runtimeDone",
    "record": {
        "requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
        "status": "error",
        "errorType": "Runtime.ExitError",
        "metrics": {
            "durationMs": 0,
            "producedBytes": 0
        }
    }
}
{
    "time": "2025-06-02T08:33:16.035Z",
    "type": "platform.report",
    "record": {
        "requestId": "2ea4f82b-5fda-483c-8ed8-427d5908c82a",
        "metrics": {
            "durationMs": 2805.027,
            "billedDurationMs": 2806,
            "memorySizeMB": 1024,
            "maxMemoryUsedMB": 202
        },
        "tracing": {
            "spanId": "69765a8e17d79988",
            "type": "X-Amzn-Trace-Id",
            "value": "Root=1-683d61c9-3a9685c768d1be1f76d7404a;Parent=27d52784c5996cbf;Sampled=1"
        },
        "status": "error",
        "errorType": "Runtime.ExitError"
    }
}
{
    "time": "2025-06-02T08:33:16.063Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:e4b97d4055c2afa813f5bda7d8e92ef971a7605f10be17ef7ae967743a21ba7d",
        "functionName": "FUNCTION_NAME",
        "functionVersion": "$LATEST"
    }
}
file:///var/task/index.mjs:431265
if (fileName !== __filename) {
^
ReferenceError: __filename is not defined
(Use `node --trace-uncaught ...` to show where the exception was thrown)
Node.js v22.15.0
{
    "time": "2025-06-02T08:33:17.668Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.ExitError"
    }
}
{
    "time": "2025-06-02T08:33:17.668Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.ExitError",
        "metrics": {
            "durationMs": 1605.514
        }
    }
}

When looking at the bundled lambda's index.mjs, this is the code trace that produces the above error:

require_librdkafka -> require_bindings -> getFileName. Those are the relevant code parts:

// node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js
var require_librdkafka = __commonJS({
  "node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js"(exports, module) {
    var kafka = require_bindings()("confluent-kafka-javascript");
    module.exports = kafka;
  }
});

// node_modules/.pnpm/bindings@1.5.0/node_modules/bindings/bindings.js
var require_bindings = __commonJS({
  "node_modules/.pnpm/bindings@1.5.0/node_modules/bindings/bindings.js"(exports, module) {
    var fs = __require("fs");
    var path = __require("path");
    var fileURLToPath2 = require_file_uri_to_path();
    var join = path.join;
    var dirname = path.dirname;
    var exists = fs.accessSync && function(path2) {
      try {
        fs.accessSync(path2);
      } catch (e) {
        return false;
      }
      return true;
    } || fs.existsSync || path.existsSync;
    var defaults = {
      arrow: process.env.NODE_BINDINGS_ARROW || " \u2192 ",
      compiled: process.env.NODE_BINDINGS_COMPILED_DIR || "compiled",
      platform: process.platform,
      arch: process.arch,
      nodePreGyp: "node-v" + process.versions.modules + "-" + process.platform + "-" + process.arch,
      version: process.versions.node,
      bindings: "bindings.node",
      try: [
        // node-gyp's linked version in the "build" dir
        ["module_root", "build", "bindings"],
        // node-waf and gyp_addon (a.k.a node-gyp)
        ["module_root", "build", "Debug", "bindings"],
        ["module_root", "build", "Release", "bindings"],
        // Debug files, for development (legacy behavior, remove for node v0.9)
        ["module_root", "out", "Debug", "bindings"],
        ["module_root", "Debug", "bindings"],
        // Release files, but manually compiled (legacy behavior, remove for node v0.9)
        ["module_root", "out", "Release", "bindings"],
        ["module_root", "Release", "bindings"],
        // Legacy from node-waf, node <= 0.4.x
        ["module_root", "build", "default", "bindings"],
        // Production "Release" buildtype binary (meh...)
        ["module_root", "compiled", "version", "platform", "arch", "bindings"],
        // node-qbs builds
        ["module_root", "addon-build", "release", "install-root", "bindings"],
        ["module_root", "addon-build", "debug", "install-root", "bindings"],
        ["module_root", "addon-build", "default", "install-root", "bindings"],
        // node-pre-gyp path ./lib/binding/{node_abi}-{platform}-{arch}
        ["module_root", "lib", "binding", "nodePreGyp", "bindings"]
      ]
    };
    function bindings(opts) {
      if (typeof opts == "string") {
        opts = { bindings: opts };
      } else if (!opts) {
        opts = {};
      }
      Object.keys(defaults).map(function(i2) {
        if (!(i2 in opts)) opts[i2] = defaults[i2];
      });
      if (!opts.module_root) {
        opts.module_root = exports.getRoot(exports.getFileName());
      }
      if (path.extname(opts.bindings) != ".node") {
        opts.bindings += ".node";
      }
      var requireFunc = typeof __webpack_require__ === "function" ? __non_webpack_require__ : __require;
      var tries = [], i = 0, l = opts.try.length, n, b, err2;
      for (; i < l; i++) {
        n = join.apply(
          null,
          opts.try[i].map(function(p) {
            return opts[p] || p;
          })
        );
        tries.push(n);
        try {
          b = opts.path ? requireFunc.resolve(n) : requireFunc(n);
          if (!opts.path) {
            b.path = n;
          }
          return b;
        } catch (e) {
          if (e.code !== "MODULE_NOT_FOUND" && e.code !== "QUALIFIED_PATH_RESOLUTION_FAILED" && !/not find/i.test(e.message)) {
            throw e;
          }
        }
      }
      err2 = new Error(
        "Could not locate the bindings file. Tried:\n" + tries.map(function(a) {
          return opts.arrow + a;
        }).join("\n")
      );
      err2.tries = tries;
      throw err2;
    }
    module.exports = exports = bindings;
    exports.getFileName = function getFileName(calling_file) {
      var origPST = Error.prepareStackTrace, origSTL = Error.stackTraceLimit, dummy = {}, fileName;
      Error.stackTraceLimit = 10;
      Error.prepareStackTrace = function(e, st) {
        for (var i = 0, l = st.length; i < l; i++) {
          fileName = st[i].getFileName();
          if (fileName !== __filename) {
            if (calling_file) {
              if (fileName !== calling_file) {
                return;
              }
            } else {
              return;
            }
          }
        }
      };
      Error.captureStackTrace(dummy);
      dummy.stack;
      Error.prepareStackTrace = origPST;
      Error.stackTraceLimit = origSTL;
      var fileSchema = "file://";
      if (fileName.indexOf(fileSchema) === 0) {
        fileName = fileURLToPath2(fileName);
      }
      return fileName;
    };
    exports.getRoot = function getRoot(file) {
      var dir = dirname(file), prev;
      while (true) {
        if (dir === ".") {
          dir = process.cwd();
        }
        if (exists(join(dir, "package.json")) || exists(join(dir, "node_modules"))) {
          return dir;
        }
        if (prev === dir) {
          throw new Error(
            'Could not find module root given file: "' + file + '". Do you have a `package.json` file? '
          );
        }
        prev = dir;
        dir = join(dir, "..");
      }
    };
  }
});

Lambda logs (JSON format) for the CJS bundling (same for arm64 and x86_64):

{
    "time": "2025-06-02T14:37:23.727Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
        "functionName": "<FUNCTION_NAME>",
        "functionVersion": "$LATEST"
    }
}
{
    "timestamp": "2025-06-02T14:37:25.373Z",
    "level": "ERROR",
    "message": {
        "errorType": "Error",
        "errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
        "stackTrace": [
            "Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
            "    at Function.getRoot (/var/task/index.js:427074:17)",
            "    at bindings (/var/task/index.js:427001:37)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
            "    at __require (/var/task/index.js:9:50)"
        ]
    }
}
{
    "time": "2025-06-02T14:37:25.469Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.Unknown"
    }
}
{
    "time": "2025-06-02T14:37:25.469Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.Unknown",
        "metrics": {
            "durationMs": 1742.111
        }
    }
}
{
    "time": "2025-06-02T14:37:25.491Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
        "functionName": "<FUNCTION_NAME>",
        "functionVersion": "$LATEST"
    }
}
{
    "timestamp": "2025-06-02T14:37:27.903Z",
    "level": "ERROR",
    "message": {
        "errorType": "Error",
        "errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
        "stackTrace": [
            "Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
            "    at Function.getRoot (/var/task/index.js:427074:17)",
            "    at bindings (/var/task/index.js:427001:37)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
            "    at __require (/var/task/index.js:9:50)"
        ]
    }
}
{
    "time": "2025-06-02T14:37:28.072Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "status": "error",
        "errorType": "Runtime.Unknown"
    }
}
{
    "time": "2025-06-02T14:37:28.073Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "invoke",
        "status": "error",
        "errorType": "Runtime.Unknown",
        "metrics": {
            "durationMs": 2581.309
        }
    }
}
{
    "time": "2025-06-02T14:37:28.073Z",
    "type": "platform.start",
    "record": {
        "requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
        "functionArn": "<FUNCTION_ARN>",
        "version": "$LATEST",
        "tracing": {
            "spanId": "169d7dcf1f466182",
            "type": "X-Amzn-Trace-Id",
            "value": "Root=1-683db723-6a1d131b6e9d9ab108adf796;Parent=321f6f8ae8e37911;Sampled=1"
        }
    }
}
{
    "time": "2025-06-02T14:37:28.073Z",
    "type": "platform.runtimeDone",
    "record": {
        "requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
        "status": "error",
        "errorType": "Runtime.Unknown",
        "metrics": {
            "durationMs": 0,
            "producedBytes": 0
        }
    }
}
{
    "time": "2025-06-02T14:37:28.073Z",
    "type": "platform.report",
    "record": {
        "requestId": "ab5fdd36-8d16-486c-8be2-7efb0c2ecca1",
        "metrics": {
            "durationMs": 2582.303,
            "billedDurationMs": 2583,
            "memorySizeMB": 1024,
            "maxMemoryUsedMB": 163
        },
        "tracing": {
            "spanId": "169d7dcf1f466182",
            "type": "X-Amzn-Trace-Id",
            "value": "Root=1-683db723-6a1d131b6e9d9ab108adf796;Parent=321f6f8ae8e37911;Sampled=1"
        },
        "status": "error",
        "errorType": "Runtime.Unknown"
    }
}
{
    "time": "2025-06-02T14:37:28.103Z",
    "type": "platform.initStart",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "runtimeVersion": "nodejs:22.v43",
        "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:fd2e05b324f99edd3c6e17800b2535deb79bcce74b7506d595a94870b3d9bd2e",
        "functionName": "<FUNCTION_NAME>",
        "functionVersion": "$LATEST"
    }
}
{
    "timestamp": "2025-06-02T14:37:29.507Z",
    "level": "ERROR",
    "message": {
        "errorType": "Error",
        "errorMessage": "Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
        "stackTrace": [
            "Error: Could not find module root given file: \"/var/task/index.js\". Do you have a `package.json` file? ",
            "    at Function.getRoot (/var/task/index.js:427074:17)",
            "    at bindings (/var/task/index.js:427001:37)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/librdkafka.js (/var/task/index.js:427088:35)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/client.js (/var/task/index.js:427562:17)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/kafka-consumer.js (/var/task/index.js:428182:18)",
            "    at __require (/var/task/index.js:9:50)",
            "    at node_modules/.pnpm/@confluentinc+kafka-javascript@1.3.0/node_modules/@confluentinc/kafka-javascript/lib/rdkafka.js (/var/task/index.js:429503:25)",
            "    at __require (/var/task/index.js:9:50)"
        ]
    }
}
{
    "time": "2025-06-02T14:37:29.598Z",
    "type": "platform.initRuntimeDone",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.Unknown"
    }
}
{
    "time": "2025-06-02T14:37:29.598Z",
    "type": "platform.initReport",
    "record": {
        "initializationType": "on-demand",
        "phase": "init",
        "status": "error",
        "errorType": "Runtime.Unknown",
        "metrics": {
            "durationMs": 1494.505
        }
    }
}

For additional context, we use pnpm as a package manager.

The CJS error message "Could not find module root given file: \"/var/task/index.js\". Do you have a package.json file? " is (maybe?) understandable, since in our lambdas, we do not have a package.json (only the bundled index.js file for the function).

Hope that helps 👍

Would be great if you could find a solution for lambdas bundled with ESM. The architecture (arm64 or x86_64) is not that important to us.

[hakuno]

but we’re unsure whether it’s compatible

Yes. It's compatible. I run Lambda with:

• Purely ESM (MJS extension);
• NodeJS 22.x;
• ARM64 container.

[felixmagnus]

@hakuno which package manager do you use?

[cjnoname]

but we’re unsure whether it’s compatible

Yes. It's compatible. I run Lambda with:

• Purely ESM (MJS extension);
• NodeJS 22.x;
• ARM64 container.

Do you use Layer or you bundle the code using webpack or other tools?

[milindl]

To just use the library, I got it to work by creating a layer - it was straightforward, and I didn't encounter any errors. I'm posting the steps for it below, and then, I'll be figuring out if I can get it to work when the code is being bundled.

@felixmagnus , how are you bundling your code?

---

Layer creation (run it on the right architecture and with the target node version, using a docker container if necessary):

mkdir layer && cd layer
mkdir nodejs && cd nodejs
npm init -y
npm install @confluentinc/kafka-javascript
cd ../
zip -r layer.zip nodejs/

The zip file should look like this internally:

layer.zip
  - nodejs
    - package.json
    - package-lock.json
    - node_modules/ (lots of subfolders)

I created a layer by uploading this to create a new layer (make sure to match platform and version).

And then the following index.mjs file for the function, configured to use my layer, worked correctly (I'm just printing the library version to check the import):

import Kafka from '@confluentinc/kafka-javascript'

export const handler = async (event) => {
  // TODO implement
  const response = {
    statusCode: 200,
    body: Kafka.librdkafkaVersion,
  };
  return response;
};

[felixmagnus]

Thanks for looking into this!

We're bundling our lambdas using AWS CDK, with the following settings / configs:

Monorepo / Package manager:

• pnpm 10.12.1
• turborepo 2.5.4

package.json

• aws-cdk version: 2.1012.0
• aws-cdk-lib version: 2.193.0

cdk.json

• app command: npx tsx infra/main.ts

Lambda construct:

• NodejsFunction from aws-cdk-lib/aws-lambda-nodejs with the following settings:

{
    runtime: Runtime.NODEJS_22_X,
    architecture: Architecture.ARM_64,
    memorySize: 1024,
    bundling: {
        format: OutputFormat.ESM,
        mainFields: ['module', 'main'],
    },
    ...,
}

@milindl is there any more information I can provide?