[BUG] Setuptools via pyproject.toml parses maintainers list badly #3663

nabriis · 2022-11-04T23:27:17Z

setuptools version

65.5.1

Python version

OS

Additional environment information

setuptools via isolated environment as part of pyproject.toml.

Description

Our releases build via setuptools using a pyproject.toml file have just now started to fail because the maintainer list added to METADATA is some odd string rather than the list of maintainers.

It has been working at least a few days ago.

The pyproject.toml file is here https://github.com/CUQI-DTU/CUQIpy/blob/main/pyproject.toml. Relevant parts are:

[build-system]
requires = ["setuptools>=42", "wheel", "versioneer[toml]==0.26"]
build-backend = "setuptools.build_meta"

[project]
maintainers = [
    {name = "Nicolai A. B. Riis", email = "[email protected]"},
    {name = "Jakob S. Jørgensen", email = "[email protected]"},
    {name = "Amal M. Alghamdi", email = "[email protected]"}
]

Somehow in the METADATA after running python -m build --sdist --wheel --outdir dist/ . maintainers is written as follows.

'=?utf-8?b?Ik5pY29sYWkgQS4gQi4g...bWRpIiA8YW1hYWxAZHR1LmRrPg==?='

Error thrown by PyPI when attempting to upload release can be viewed here:
https://github.com/CUQI-DTU/CUQIpy/actions/runs/3397562988/jobs/5649833961

Perhaps related to the ø in one of our names?

Expected behavior

Expect to correctly parse the maintainer list and provide METADATA file with correct format of maintainers.

How to Reproduce

Create Python project with pyproject.toml file and maintainer list as written above.
Build wheels etc. for release
Check METADATA file

Output

The text was updated successfully, but these errors were encountered:

abravalheri · 2022-11-09T13:02:01Z

Hi @nabriis thank you very much for reporting this.

I checked your reproducer and it does seem that setuptools is writing the PKG-INFO file in the sdist. That is curious.

Motivated by this result, I did some investigation in #3666, and I believe that setuptools is (surprisingly 😅) doing the right thing here.

Since setuptools uses another package pypa/wheel to create the METADATA file, I started to suspect that there might be a problem with pypa/wheel. To verify that, I did the following experiment:

rm -rf /tmp/helloworld
mkdir /tmp/helloworld
cd /tmp/helloworld
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "helloworld"
version = "42"
maintainers = [
    {name = "Nicolai A. B. Riis", email = "[email protected]"},
    {name = "Jakob S. Jørgensen", email = "[email protected]"},
    {name = "Amal M. Alghamdi", email = "[email protected]"}
]
EOF
virtualenv -p python3.10 .venv
.venv/bin/python -m pip install -U pip setuptools wheel build

cat <<EOF > investigation.patch
--- .venv/lib/python3.8/site-packages/wheel/bdist_wheel.py      2022-11-09 12:07:41.250225200 +0000
+++ /tmp/modified_bdist_wheel.py        2022-11-09 12:07:29.936182800 +0000
@@ -508,10 +508,16 @@
             # .egg-info is a single file
             pkginfo_path = egginfo_path
             pkg_info = pkginfo_to_metadata(egginfo_path, egginfo_path)
+            with open(pkginfo_path, "r", encoding="utf-8") as inp:
+                print("\n\n\n~-------------------------~\nBefore wheel conversion:\n")
+                print(inp.read())
             os.mkdir(distinfo_path)
         else:
             # .egg-info is a directory
             pkginfo_path = os.path.join(egginfo_path, "PKG-INFO")
+            with open(pkginfo_path, "r", encoding="utf-8") as inp:
+                print("\n\n\n~-------------------------~\nBefore wheel conversion:\n")
+                print(inp.read())
             pkg_info = pkginfo_to_metadata(egginfo_path, pkginfo_path)

             # ignore common egg metadata that is useless to wheel
@@ -537,6 +543,10 @@
         with open(pkg_info_path, "w", encoding="utf-8") as out:
             Generator(out, mangle_from_=False, maxheaderlen=0).flatten(pkg_info)

+        with open(pkg_info_path, "r", encoding="utf-8") as out:
+            print("\n\n\n~-------------------------~\nAfter wheel conversion:\n")
+            print(out.read())
+
         for license_path in self.license_paths:
             filename = os.path.basename(license_path)
             shutil.copy(license_path, os.path.join(distinfo_path, filename))
EOF

patch -u .venv/lib/python3.10/site-packages/wheel/bdist_wheel.py -i investigation.patch
.venv/bin/pyproject-build --no-isolation

The following is part of the output of the experiment:

# ...
Copying helloworld.egg-info to build/bdist.linux-x86_64/wheel/helloworld-42-py3.10.egg-info
running install_scripts



~-------------------------~
Before wheel conversion:

Metadata-Version: 2.1
Name: helloworld
Version: 42
Maintainer-email: "Nicolai A. B. Riis" <[email protected]>, "Jakob S. Jørgensen" <[email protected]>, "Amal M. Alghamdi" <[email protected]>




~-------------------------~
After wheel conversion:

Metadata-Version: 2.1
Name: helloworld
Version: 42
Maintainer-email: =?utf-8?b?Ik5pY29sYWkgQS4gQi4gUmlpcyIgPG5hYnJAZHR1LmRrPiwgIkpha29iIFMuIErDuHJnZW5zZW4iIDxqYWtqQGR0dS5kaz4sICJBbWFsIE0uIEFsZ2hhbWRpIiA8YW1hYWxAZHR1LmRrPg==?=


creating build/bdist.linux-x86_64/wheel/helloworld-42.dist-info/WHEEL
#

The output does seems to indicate that the change in the encoding is indeed happening inside of pypa/wheel. So I think the next step is to report it to that repository (I plan to do it soon, but please feel free to beat me to that).

nabriis · 2022-11-09T13:32:08Z

Hi @nabriis thank you very much for reporting this.

I checked your reproducer and it does seem that setuptools is writing the PKG-INFO file in the sdist. That is curious.

Motivated by this result, I did some investigation in #3666, and I believe that setuptools is (surprisingly 😅) doing the right thing here.

Since setuptools uses another package pypa/wheel to create the METADATA file, I started to suspect that there might be a problem with pypa/wheel. To verify that, I did the following experiment:

rm -rf /tmp/helloworld
mkdir /tmp/helloworld
cd /tmp/helloworld
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "helloworld"
version = "42"
maintainers = [
    {name = "Nicolai A. B. Riis", email = "[email protected]"},
    {name = "Jakob S. Jørgensen", email = "[email protected]"},
    {name = "Amal M. Alghamdi", email = "[email protected]"}
]
EOF
virtualenv -p python3.10 .venv
.venv/bin/python -m pip install -U pip setuptools wheel build

cat <<EOF > investigation.patch
--- .venv/lib/python3.8/site-packages/wheel/bdist_wheel.py      2022-11-09 12:07:41.250225200 +0000
+++ /tmp/modified_bdist_wheel.py        2022-11-09 12:07:29.936182800 +0000
@@ -508,10 +508,16 @@
             # .egg-info is a single file
             pkginfo_path = egginfo_path
             pkg_info = pkginfo_to_metadata(egginfo_path, egginfo_path)
+            with open(pkginfo_path, "r", encoding="utf-8") as inp:
+                print("\n\n\n~-------------------------~\nBefore wheel conversion:\n")
+                print(inp.read())
             os.mkdir(distinfo_path)
         else:
             # .egg-info is a directory
             pkginfo_path = os.path.join(egginfo_path, "PKG-INFO")
+            with open(pkginfo_path, "r", encoding="utf-8") as inp:
+                print("\n\n\n~-------------------------~\nBefore wheel conversion:\n")
+                print(inp.read())
             pkg_info = pkginfo_to_metadata(egginfo_path, pkginfo_path)

             # ignore common egg metadata that is useless to wheel
@@ -537,6 +543,10 @@
         with open(pkg_info_path, "w", encoding="utf-8") as out:
             Generator(out, mangle_from_=False, maxheaderlen=0).flatten(pkg_info)

+        with open(pkg_info_path, "r", encoding="utf-8") as out:
+            print("\n\n\n~-------------------------~\nAfter wheel conversion:\n")
+            print(out.read())
+
         for license_path in self.license_paths:
             filename = os.path.basename(license_path)
             shutil.copy(license_path, os.path.join(distinfo_path, filename))
EOF

patch -u .venv/lib/python3.10/site-packages/wheel/bdist_wheel.py -i investigation.patch
.venv/bin/pyproject-build --no-isolation

The following is part of the output of the experiment:

# ...
Copying helloworld.egg-info to build/bdist.linux-x86_64/wheel/helloworld-42-py3.10.egg-info
running install_scripts



~-------------------------~
Before wheel conversion:

Metadata-Version: 2.1
Name: helloworld
Version: 42
Maintainer-email: "Nicolai A. B. Riis" <[email protected]>, "Jakob S. Jørgensen" <[email protected]>, "Amal M. Alghamdi" <[email protected]>




~-------------------------~
After wheel conversion:

Metadata-Version: 2.1
Name: helloworld
Version: 42
Maintainer-email: =?utf-8?b?Ik5pY29sYWkgQS4gQi4gUmlpcyIgPG5hYnJAZHR1LmRrPiwgIkpha29iIFMuIErDuHJnZW5zZW4iIDxqYWtqQGR0dS5kaz4sICJBbWFsIE0uIEFsZ2hhbWRpIiA8YW1hYWxAZHR1LmRrPg==?=


creating build/bdist.linux-x86_64/wheel/helloworld-42.dist-info/WHEEL
#

The output does seems to indicate that the change in the encoding is indeed happening inside of pypa/wheel. So I think the next step is to report it to that repository (I plan to do it soon, but please feel free to beat me to that).

Thanks for the investigation! Seems this was just reported for summary META data in the linked issue. Looks like these issues are very similar. If you have some more in depth info to add please feel free. Thanks again.

abravalheri · 2022-11-09T19:02:45Z

I proposed a fix to pypa/wheel, which seems to solve the problem (I might be wrong though...).

For the time being I think the best is to close this issue in favour of pypa/wheel#488.

We can revisit and reopen it in the future if something else needs to be changed in the setuptools code base.

nabriis added bug Needs Triage Issues that need to be evaluated for severity and status. labels Nov 4, 2022

nabriis changed the title ~~[BUG] Setuptools parses maintainer badly~~ [BUG] Setuptools parses maintainers list badly Nov 4, 2022

nabriis changed the title ~~[BUG] Setuptools parses maintainers list badly~~ [BUG] Setuptools via pyproject.toml parses maintainers list badly Nov 4, 2022

di mentioned this issue Nov 8, 2022

Unable to upload package - "Error : Use a valid email address" pypi/warehouse#12490

Closed

frague59 mentioned this issue Nov 9, 2022

twine check passes but pypi rejects pypa/twine#937

Closed

abravalheri mentioned this issue Nov 9, 2022

Ensure pyproject.toml accepts UTF-8 in maintainers #3666

Merged

2 tasks

nabriis mentioned this issue Nov 9, 2022

Incorrect summary in produced wheels starting from 0.38.1 pypa/wheel#488

Closed

This was referenced Nov 9, 2022

UTF-8 author name leads to Author-email field being encoded pypa/build#531

Closed

Allow METADATA file to contain UTF-8 chars pypa/wheel#489

Merged

abravalheri closed this as completed Nov 9, 2022

abravalheri mentioned this issue Aug 30, 2023

[BUG] pyproject.toml: Unicode character in a license file prevents dependencies from being installed #4033

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Setuptools via pyproject.toml parses maintainers list badly #3663

[BUG] Setuptools via pyproject.toml parses maintainers list badly #3663

nabriis commented Nov 4, 2022 •

edited

Loading

abravalheri commented Nov 9, 2022

nabriis commented Nov 9, 2022 •

edited

Loading

abravalheri commented Nov 9, 2022

[BUG] Setuptools via pyproject.toml parses maintainers list badly #3663

[BUG] Setuptools via pyproject.toml parses maintainers list badly #3663

Comments

nabriis commented Nov 4, 2022 • edited Loading

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

abravalheri commented Nov 9, 2022

nabriis commented Nov 9, 2022 • edited Loading

abravalheri commented Nov 9, 2022

nabriis commented Nov 4, 2022 •

edited

Loading

nabriis commented Nov 9, 2022 •

edited

Loading