(aws-lambda-python-alpha): I'm failing to bundle a Python 3.12 function using pipenv.

[kamaD-y]

Describe the bug

When trying to export the Pipfile to requirements.txt, I encounter the following error:
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

Expected Behavior

I expect to successfully bundle.

Current Behavior

Traceback (most recent call last):
  File "/usr/app/venv/bin/pipenv", line 5, in <module>
    from pipenv import cli
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/__init__.py", line 57, in <module>
    from .cli import cli
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/cli/__init__.py", line 1, in <module>
    from .command import cli  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/cli/command.py", line 7, in <module>
    from pipenv.cli.options import (
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/cli/options.py", line 3, in <module>
    from pipenv.project import Project
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/project.py", line 21, in <module>
    from pipenv.core import system_which
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/core.py", line 30, in <module>
    from pipenv.utils.resolver import venv_resolve_deps
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 14, in <module>
    from pipenv.vendor.requirementslib import Pipfile, Requirement
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/requirementslib/__init__.py", line 7, in <module>
    from .models.lockfile import Lockfile
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/requirementslib/models/lockfile.py", line 14, in <module>
    from ..utils import is_editable, is_vcs, merge_items
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/requirementslib/utils.py", line 11, in <module>
    import pip_shims.shims
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/__init__.py", line 26, in <module>
    from . import shims
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/shims.py", line 12, in <module>
    from .models import (
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/models.py", line 775, in <module>
    Command.add_mixin(SessionCommandMixin)
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/models.py", line 689, in add_mixin
    mixin = mixin.shim()
            ^^^^^^^^^^^^
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/models.py", line 737, in shim
    result = self.traverse(top_path)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/models.py", line 729, in traverse
    result = shim.shim()
             ^^^^^^^^^^^
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/models.py", line 575, in shim
    imported = self._import()
               ^^^^^^^^^^^^^^
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/models.py", line 600, in _import
    result = self._import_module(self.calculated_module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/vendor/pip_shims/models.py", line 352, in _import_module
    imported = importlib.import_module(module)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lang/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/patched/notpip/_internal/cli/req_command.py", line 15, in <module>
    from pipenv.patched.notpip._internal.cache import WheelCache
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/patched/notpip/_internal/cache.py", line 13, in <module>
    from pipenv.patched.notpip._internal.exceptions import InvalidWheelFilename
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/patched/notpip/_internal/exceptions.py", line 7, in <module>
    from pipenv.patched.notpip._vendor.pkg_resources import Distribution
  File "/usr/app/venv/lib/python3.12/site-packages/pipenv/patched/notpip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

Reproduction Steps

I'm using the following stack.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { aws_lambda as lambda } from 'aws-cdk-lib';
import { PythonFunction } from '@aws-cdk/aws-lambda-python-alpha';

export class TestStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    const entry = 'src'
    new PythonFunction(this, 'TestFunction', {
      functionName: "testFunction",
      runtime: lambda.Runtime.PYTHON_3_12,
      entry,
      index: 'index.py',
      handler: 'handler',
      bundling: {
        assetExcludes: ['.venv'],
      },
    });
  }
}

When I run cdk synth with the Pipfile placed in the src entry point, it results in an error.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"

[dev-packages]

[requires]
python_version = "3.12"
python_full_version = "3.12.2"

Possible Solution

There is the following description in the Dockerfile.

# pipenv 2022.4.8 is the last version with Python 3.6 support
    pip install pipenv==2022.4.8 poetry==$POETRY_VERSION && \

I think it would be fine to upgrade the version of pipenv, as Python 3.6 is no longer supported by Lambda.
The same error was present up to pipenv==2023.3.20, but it seems that from pipenv==2023.4.20 onwards, the error does not occur

Additional Information/Context

No response

CDK CLI Version

2.141.0 (build 3d1c06e)

Framework Version

No response

Node.js Version

v18.16.0

OS

Windows (WSL2)

Language

TypeScript

Language Version

Typescript (5.4.5)

Other information

No response

[Leo10Gama]

Unfortunately, I wasn't able to reproduce the issue on my machine using the provided snippets, both using CDK 2.141.0 and the current CDK 2.142.1. Using the provided stack and Pipfile, with Pipfile both under the src directory and the root directory, it synthesizes and deploys. Can you provide any additional details on how we can reproduce the issue?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[kamaD-y]

@Leo10Gama

Please check the following repository for the directory structure.
https://github.com/kamaD-y/cdk-python-lambda-test

I am using pyenv and the version is as follows.

pyenv 2.4.0-2-ge1f07da7

The version of the Docker command is as follows.

Docker version 26.1.0, build 9714adc

Please let me know if there is any other information you need.

[Leo10Gama]

Thanks for providing the repository! I can confirm the error on my end as well, and will look into it.

In the meantime, it seems like you can work around this by downgrading to Python 3.11.9 or lower, since it seems like the issue sprouted from a change in Python 3.12 that's not playing nice with older versions of pipenv.

[jlosito]

I'm getting a similar error when bundling, but in my case I'm using a requirements.txt. Currently, I'm using cdk-lib v2.124.0 due to other project constraints I'm working on.

The workaround for me was to drop the Lambda Function's runtime to 3.11.

[Mexflubber]

Can we get an update here ?