Add support for Python 3.10

[michael-k]

Latest official update for anyone who missed it in the wall of comments:

We have today [2023-02-28] published a Lambda container base image for Python 3.10, for both x86_64 and arm64 architectures. This image is suitable for production workloads.

Instructions for how to use Lambda container base images for Python are given in the ‘Usage’ tab at https://gallery.ecr.aws/lambda/python.

We expect to publish managed runtime support for Python 3.10 within 90 days.

Previous official update:

We recognize the extended delay in providing Python 3.10 support in AWS Lambda, and apologize for the inconvenience this has caused. Our plan is to publish a base container images for Python 3.10 this month [February 2023], with the managed runtime and tooling support (SAM CLI, CFN, CDK, AWS CLI, console, etc) support to follow. We will provide an ETA for the managed runtime when we publish the container image.

---

Python 3.10 was released on 2021-10-04

• Announcement: https://pythoninsider.blogspot.com/2021/10/python-3100-is-available.html
  • 3.10.1 announcement: https://pythoninsider.blogspot.com/2021/12/python-3101-is-available.html
  • 3.10.2 announcement: https://pythoninsider.blogspot.com/2022/01/python-3102-3910-and-3110a4-are-now.html
  • 3.10.3 announcement: https://pythoninsider.blogspot.com/2022/03/python-3103-3911-3813-and-3713-are-now.html
  • 3.10.4 announcement: https://pythoninsider.blogspot.com/2022/03/python-3104-and-3912-are-now-available.html
  • 3.10.5 announcement: https://pythoninsider.blogspot.com/2022/06/python-3105-is-available_6.html
  • 3.10.6 announcement: https://pythoninsider.blogspot.com/2022/08/python-3106-is-available.html
  • 3.10.7 announcement: https://pythoninsider.blogspot.com/2022/09/python-releases-3107-3914-3814-and-3714.html
  • 3.10.8 announcement: https://pythoninsider.blogspot.com/2022/10/python-versions-3108-3915-3815-3715-now.html
  • 3.10.9 announcement: https://pythoninsider.blogspot.com/2022/12/python-3111-3109-3916-3816-3716-and.html
  • 3.10.10 announcement: https://pythoninsider.blogspot.com/2023/02/python-3112-python-31010-and-3120-alpha.html
  • 3.10.11 announcement: https://pythoninsider.blogspot.com/2023/04/its-time-for-another-set-of-python.html
• Release schedule: https://www.python.org/dev/peps/pep-0619/
• What’s new in Python 3.10: https://docs.python.org/3.10/whatsnew/3.10.html

[andreif]

Please?

[volphy]

Any implementation timeline?

[nikitajz]

some features look interesting, looking forward!

[leslie-alldridge]

keen !

[fadesibert]

+1
as much as I love building python from source - maybe 3.7 is a little long in the tooth to be the standard...

[alimcmaster1]

+1 as much as I love building python from source - maybe 3.7 is a little long in the tooth to be the standard...

Agree - looks like @jbesw did the work for 3.9... maybe he can help with 3.10 support?

https://aws.amazon.com/blogs/compute/python-3-9-runtime-now-available-in-aws-lambda/

[jtuliani]

Thank you everyone for your feedback regarding AWS Lambda support for Python 3.10. We are aware of the ask, however, we don't currently have a timeline to share.

[acdha]

Python 3.9 just hit the security-only stage of its maintenance lifecycle:

https://pythoninsider.blogspot.com/2022/05/python-3913-is-now-available.html

[HWiese1980]

Two months since the last update. Any (hopefully good) news here? @jtuliani

[jwgstr]

I had a desire to update my lambda builds to 3.10 the other day and began looking into this more. Since this issue has been open for a little while I thought I might take a stab at building this from the amazonlinux base image (as the python 3.9 builds are). None of the guides I found quite made this copy / paste possible.

So I came up with this, which I tested and seems to work just fine at least for my use cases which includes things like numpy. But depending on your project complexity your mileage may vary. It doesn't follow all the same conventions with putting task logic in /var/task or app logic in /var/runtime but that didn't cause any conflicts for me. It's kind of a hodgepodge of various AWS docs and articles I found. Wouldn't surprise me if it can be cleaned up a bit.

I don't think I have access to contribute this back anywhere otherwise I would create PR. @jtuliani please feel free to point me in the right direction for that and I'm happy to help iterate on a better vetted and reviewed 3.10 image.

In the meantime I hope this helps others.

ARG FUNCTION_DIR="/app/"
ARG AWS_LINUX_VERSION="2022"
ARG PYTHON_VERSION="3.10.4"


FROM amazonlinux:${AWS_LINUX_VERSION} as python-layer

ARG PYTHON_VERSION

# install python
RUN yum update -y
RUN yum groupinstall "Development Tools" -y
RUN yum install openssl1.1 openssl1.1-devel libffi-devel bzip2-devel wget -y
RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz
RUN tar -xf Python-${PYTHON_VERSION}.tgz
RUN cd Python-${PYTHON_VERSION}/ && \
    ./configure --enable-optimizations && \
    make install
RUN ln -s /Python-${PYTHON_VERSION}/python /usr/bin/python
RUN python -m pip install --upgrade pip


FROM amazonlinux:${AWS_LINUX_VERSION} as base-layer

# copy over python
ARG PYTHON_VERSION
COPY --from=python-layer /Python-${PYTHON_VERSION} /Python-${PYTHON_VERSION}
COPY --from=python-layer /usr/local/bin /usr/local/bin
COPY --from=python-layer /usr/local/lib /usr/local/lib
RUN ln -s /Python-${PYTHON_VERSION}/python /usr/bin/python

ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}


FROM base-layer as build-layer

######## YOUR OWN SETUP PROCESS HERE ########################
# copy over requirements and install those
COPY setup.py .
RUN pip install . --target "${FUNCTION_DIR}"

# copy over configuration and service code then install it
COPY config/ ${FUNCTION_DIR}/config/
COPY service/ ${FUNCTION_DIR}/
RUN pip install . --target "${FUNCTION_DIR}"
######## ########################### ########################

# install lambda runtime interface client for python
RUN pip install awslambdaric --target "${FUNCTION_DIR}"


FROM base-layer as runtime-layer

# copy in the built dependencies
ARG FUNCTION_DIR
COPY --from=build-layer ${FUNCTION_DIR} ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}

# (optional) add lambda runtime interface emulator
#ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
#RUN chmod 755 /usr/bin/aws-lambda-rie
#ENTRYPOINT [ "/usr/bin/aws-lambda-rie", "python", "-m", "awslambdaric" ]

ENTRYPOINT [ "python", "-m", "awslambdaric" ]

######## REFERENCE YOUR OWN HANDLER HERE ########################
CMD [ "main.app" ]```
######## ############################### ########################

[matthewdeanmartin]

@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular, yum update fails on a self-hosted Gitlab.)

[jwgstr]

@jwgstr - can you post this Dockerfile as a gist or repo so I can discuss issues with it w/o cluttering the main thread? (in particular, yum update fails on a self-hosted Gitlab.)

excellent idea: https://gist.github.com/jwgstr/4dcc9f16c52529486de0f6fa78c55aef

[sjzabel]

This is now an issue for users of Gitlab who use the Docker:latest image to build lambda images using DinD. The Docker image is built off of Alpine 3.16 which now only supports python3.10 natively.

Our workaround is to use pyenv to install a python 3.9 version... but that adds significant time to our builds.

[BjoernPetersen]

You could also just not use the latest tag and continue with the older image (e.g. the docker:20.10.16-dind-alpine3.15 tag). Using the latest tag is generally not a great idea, because it'll often lead to issues like these in the worst possible moment.

Not saying that it isn't ridiculous for AWS to still not support Python 3.10 in Lambdas, just offering my opinion on best practices when using container images.

[sjzabel]

Let me try that. (Thanks, I had gone through the list of available Docker tags and I didn't see that one.)

Ty,
Stephen

[obataku]

now that 3.10 is the only available runtime for the latest codebuild images (aws/aws-codebuild-docker-images#505), one cannot use these to build + package lambdas without manually downgrading or upgrading Python (e.g. via pyenv) in codebuild or lambda, respectively, or risk bundling incompatible C extension binaries between build and deployment. @jtuliani @jbesw any updates?

[mikelane]

Half way around the sun since this was opened and 10 months since 3.10 final was released. Hey AWS, any chance we can get a 3.10 image sometime soon?? @jtuliani?

[HWiese1980]

Yeah, it can't be that hard...

[jtuliani]

We are working on support for Py3.10 in Lambda. However, I don’t currently have a date we can share. We appreciate your feedback and we are also investing in process improvements to enable us to ship runtimes more promptly in future.

[leslie-alldridge]

Thanks for the update

[michael-k]

I don’t currently have a date we can share.

Does this mean it's more than 3 months away? Because for Node.js 16 you made the following announcement:

Work is in progress, and we are planning to ship a Node 16 managed runtime and base container image within the next 90 days.

Source: #14 (comment)

Python 3.11 will be released in ~2 months, see https://peps.python.org/pep-0664/#schedule

[HWiese1980]

Valid question. If your release cycle of Lambda containers with new Python versions takes longer than that of completely new Python versions, I'd say, there's something wrong with your release process.

[HWiese1980]

Valid question. If your release cycle of Lambda containers with new Python versions takes longer than that of completely new Python versions, I'd say, there's something wrong with your release process.

I don't think you understand the definition of the word question. Refer to the comment made here a few days ago by jtuliani, a few comments above yours "we are also investing in process improvements to enable us to ship runtimes more promptly in future"

Excuse me? Is "Does this mean, it's more than 3 months away?" not a question?

[leslie-alldridge]

Can you guys argue on Facebook instead? Most of us are watching to see a PR be merged not two people trying to get one up on each other. Thanks.

[jetttz]

thank you it's live now

[LucasHartridge]

Thanks team 🙏, it is live now

[dliwustl]

Did they upgrade the manylinux platform version for 3.10? I believe it was manylinux2010_x86_64 in 3.9. Did they go to 2014? I'm not sure where to look to figure that out, but I want to know if I can change the platform when building Layers.

[kinghuang]

So, that's 18 months 20 months to go from 3.9 to 3.10? I shudder to think how long 3.11 will take.

Reference: AWS Lambda adds support for Python 3.9 (Aug 16, 2021)

[tongclement]

Finally! 🎉 Now onto 3.11

[mnapoli]

Just a quick note since https://github.com/brefphp was mentioned. As the author of Bref, while I understand (and share) the frustration, I don’t condone aggressive comments. We can use better words to express that. Just wanted to clarify, sorry for the off-topic comment.

[re-thc]

Under the hood it could be something "Debian stable"-related?

It's Amazon Linux everywhere for AWS i.e. Redhat...

[miketheman]

Released: https://aws.amazon.com/blogs/compute/python-3-10-runtime-now-available-in-aws-lambda/

[jpmatusik]

Thanks AWS team! (normally wouldn't post but tons of the comments were negative so wanted to balance it out a bit lol)

[davidlee-ca]

Echoing @jpmatusik - congratulations AWS team!

[jtuliani]

I'm happy to announce that Python 3.10 support in AWS Lambda is now GA: https://aws.amazon.com/blogs/compute/python-3-10-runtime-now-available-in-aws-lambda/

We have also shipped a preview base container image for Python 3.11 support in AWS Lambda. The purpose of this image is to enable customers to get an early look at Python 3.11 support and provide any feedback. This image is subject to change and should not be used for production workloads. To provide feedback on this image, and for future updates on Python 3.11 support, see #62 .

Thank you all for your patience and feedback. We recognize that we've not met customer expectations regarding timely runtime launches, and are working hard to catch up.

[alxsavard]

Error: expected runtime to be one of [... python3.6 python3.7 python3.8 python3.9 ...], got python3.10

Seems like the api for creating functions don't know about 3.10 yet. Hopefully it won't take too long to be added, I couldn't find the info in the blog post 🤞

[acbellini]

Chiming in with @jpmatusik, kudos for the release!

[QuinnyPig]

Very glad to see this, dismayed at a few comments not keeping it "above the belt" with the criticism of the delay. There are humans on both ends here.

[sdaves]

@jtuliani Thank you AWS team for releasing this update. I would love to hear the details of what was the blocker, it would make a great postmortem to prevent this in the future.

[HWiese1980]

Finally... We Germans like to say "Was lange währt, wird endlich gut". Thank you, AWS, for basically doing your job. It's not easy sometimes, I know.

But seriously, AWS, you should really, I mean, like, REALLY check on and fix your delivery/upgrade process. There's something terribly flawed and smelly if a from the viewpoint of your customers pretty simple upgrade to a new runtime version takes one and a half year.

[BTripp1986]

@jtuliani Thank you AWS team for releasing this update. I would love to hear the details of what was the blocker, it would make a great postmortem to prevent this in the future.

This would be great, even if we just got the fully digested conclusions of the postmortem. I doubt that it was the runtime itself causing the issue, which would make a postmortem even more insightful for those of us on the outside. I suspect that this took so long because it wasn't just an update to the Python Lambda runtime, but a massive update across a large part of the system(evidenced by the large number of updates that came with this, and the fact that the 3.11 base image is already in preview).

It would be great if we could learn with AWS about what the stumbling blocks were so that when we need to plan for a massive update at our own companies we can apply those lessons.

[kinghuang]

But seriously, AWS, you should really, I mean, like, REALLY check on and fix your delivery/upgrade process. There's something terribly flawed and smelly if a from the viewpoint of your customers pretty simple upgrade to a new runtime version takes one and a half year.

As a customer, it’s worrying that it’s taking AWS more than 12 months to update for a system that has a fixed 12 month upgrade cycle. If AWS continues the way they have for Python 3.9 and 3.10, Lambda is going to continually fall further and further behind on Python versions.

[fitzoh]

I'm not gonna be the one to do it, but for the people requesting some kind of post-mortem it might make sense to open a new top level issue.
This one is now closed and probably less likely to receive a substantial response.

[jtuliani]

@alxsavard Thanks for flagging the doc issue, we're looking into that. However, this should be a documentation issue only, the CreateFunction API does support Python 3.10. Are you seeing an error when calling the API? If so, in which region?

[alxsavard]

@jtuliani I did get an error yesterday (in us-east-1) but it seems to be fixed now! Thanks for asking!

[jtuliani]

@alxsavard Thanks for confirming. It takes quite a while to roll out a runtime release safely across all regions (which is why folks on this thread found the release before the official announcement). I expect you tried it before the us-east-1 deployment was complete. We appreciate the quick follow-up, thank you.

[michael-k]

Is there an ETA for Python 3.10 on Lambda@Edge? The announcement doesn't mention Lambda@Edge and the documentation doesn't list Python 3.10 (but might be outdated).

Or do I have to ask the CloudFront team?

[AntonKyrychekCrit]

@Felixoid you need to bump TF version to =>0.13.2 and aws provider version =>4.64 then you will see it
UPD sorry missed a part with Lambda@Edge

[Felixoid]

Kindly excuse me, but it's the web console.

update

the message was deleted. There was a screenshot about the inability to use python 3.10 as a lambda@edge function

[mcdonnnj]

@Felixoid you need to bump TF version to =>0.13.2 and aws provider version =>4.64 then you will see it

@AntonKyrychekCrit They are trying to create a Lambda@Edge function which has further restrictions. Please see the Lambda@Edge supported runtimes and note that Python 3.10 is not listed.