From dd59afe6ccec95acb01f014fd77bf353c9f85c76 Mon Sep 17 00:00:00 2001 From: leycec Date: Wed, 16 Oct 2019 02:14:33 -0400 Subject: [PATCH] Resolve pypa/pip#6163 and pypa/pip#6434. This commit circumvents a countably infinite number of issues introduced by both pip 19.0.0 and 19.1.0 - all pertaining to PEP 517 and 518, which is to say "pyproject.toml". The mere existence of a "pyproject.toml" file fundamentally breaks pip in inexplicable, horrifying ways: a significantly worse state of affairs than the pre-PEP 517 and 518 days of setuptools yore. Say what you will of setuptools ("...so much hate"), but at least it sort of fundamentally worked. The same cannot be said of recent pip versions, which absolutely do not work whatsoever. Die, pip! Specifically, this commit: * Mandatory dependencies bumped: * setuptools >= 38.2.0. The same version of setuptools required by BETSEE is now required by BETSE, ensuring parity between the two codebases and avoiding painful dependency conflicts. For safety, this dependency is repeated in the top-level "pyproject.toml" file. * Installation improved: * pypa/pip#6163 resolved. All files and subdirectories of the project directory containing the top-level "setup.py" script are now safely registered to be importable modules and packages. Technically, this should not be required. The current build framework (e.g., pip, setuptools) should implicitly guarantee this to be the case. Indeed, the obsolete setuptools-based "easy_install" script does so. Sadly, pip >= 19.0.0 fails to do so for projects defining a top-level "pyproject.toml" file. Upstream purports to have resolved this, but the most recent stable release of pip continues to suffer this. * pypa/pip#6434 resolved. The top-level "pyproject.toml" file now explicitly declares a default value for the "build-backend" key. Doing so safeguards backward compatibility with pip 19.1.0, which erroneously violated PEP 51{7,8} by failing to fallback to a sane default value in the absence of this key. If this key is left undeclared, pip 19.1.0 fails on attempting to perform an editable (i.e., developer-specific) installation of this application. * Documentation revised: * pip-based editable installation. The top-level "README.rst" file now advises developers to leverage "pip" rather than "setuptools" when performing an editable installation of this application. * "setup.cfg"-based PyPI documentation. The top-level "setup.cfg" file now transcludes the contents of the top-level "README.rst" file, a substantial improvement over the prior code-based approach strewn throughout the codebase (e.g., "setup.py", "betse_setup.buputil"). * API generalized: * Defined a new "betse.util.py.module.pyimport" submodule: * Renamed the prior betse.util.py.pys.add_import_dirname() function to register_dir() for brevity and clarity. * Git maintenance: * Ignored all top-level pip-specific temporary directories (e.g., "pip-wheel-metadata") with respect to Git tracking. (Contemptible contusions of the combustible dirigible!) --- .gitignore | 3 ++ README.rst | 4 +- betse/metadeps.py | 66 +++++++++++++++++++++++--- betse/util/py/module/pyimport.py | 77 +++++++++++++++++++++++++++++++ betse/util/py/module/pymodname.py | 8 ++-- betse/util/py/module/pymodule.py | 8 +++- betse/util/py/pys.py | 47 ------------------- betse_setup/buputil.py | 63 ------------------------- pyproject.toml | 39 +++++++++++++++- setup.cfg | 5 +- setup.py | 65 +++++++++++++++++++++++--- 11 files changed, 251 insertions(+), 134 deletions(-) create mode 100644 betse/util/py/module/pyimport.py diff --git a/.gitignore b/.gitignore index 986ae1b1..a37908ec 100644 --- a/.gitignore +++ b/.gitignore @@ -22,6 +22,9 @@ /freeze/build/ /freeze/dist/ +# Ignore all top-level pip-specific temporary directories. +/pip-wheel-metadata/ + # Ignore all top-level py.test-specific temporary directories. /.cache/ /.pytest_cache/ diff --git a/README.rst b/README.rst index ca32421e..5e712f20 100644 --- a/README.rst +++ b/README.rst @@ -230,14 +230,14 @@ repository and prior stable releases – is manually installable as follows: .. code-block:: console - sudo python3 setup.py develop + sudo pip install --editable . - **Non-editably,** installing a physical copy of the current BETSE codebase. Modifications to this code are ignored and thus require reinstallation. .. code-block:: console - sudo python3 setup.py install + sudo pip install . #. (\ *Optional*\ ) **Test BETSE,** running all modelling phases of a sample simulation from a new directory. diff --git a/betse/metadeps.py b/betse/metadeps.py index 20610a29..bed88978 100644 --- a/betse/metadeps.py +++ b/betse/metadeps.py @@ -27,20 +27,72 @@ from collections import namedtuple # ....................{ LIBS ~ install : mandatory }.................... +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +# WARNING: To avoid dependency conflicts between "pip", "setuptools", BETSE, +# and BETSEE, the value of this global variable *MUST* be synchronized (i.e., +# copied) across numerous files in both codebases. Specifically, the following +# strings *MUST* be identical: +# * "betse.metadeps.SETUPTOOLS_VERSION_MIN". +# * "betsee.guimetadeps.SETUPTOOLS_VERSION_MIN". +# * The "build-backend" setting in: +# * "betse/pyproject.toml". +# * "betsee/pyproject.toml". +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + # This public global is externally referenced by "setup.py". -SETUPTOOLS_VERSION_MIN = '36.7.2' +SETUPTOOLS_VERSION_MIN = '38.2.0' ''' Minimum version of :mod:`setuptools` required at both application install- and runtime as a human-readable ``.``-delimited string. Motivation ---------- -This application requires the -:meth:`setuptools.command.easy_install.ScriptWriter.get_args` class method and -hence at least the oldest version of :mod:`setuptools` to have this method. -Since official setuptools documentation fails to specify the exact version that -first defined this method, we fallback to a sufficiently old version from 2017 -known to define this method. +This version derives from both: + +* BETSE requirements. BETSE requires the + :meth:`setuptools.command.easy_install.ScriptWriter.get_args` class method + and hence at least the oldest version of :mod:`setuptools` to have this + method. Since official setuptools documentation fails to specify the exact + version that first defined this method, we fallback to a sufficiently old + version from 2017 known to define this method: :mod:`setuptools` >= 36.7.2. +* BETSEE requirements. BETSEE requires :mod:`PySide2`, which is distributed as + a wheel and thus requires wheel support, which in turns requires either + ``pip`` >= 1.4.0 or :mod:`setuptools` >= 38.2.0. While ``pip`` 1.4.0 is + relatively ancient, :mod:`setuptools` 38.2.0 is comparatively newer. If the + current version of :mod:`setuptools` is *not* explicitly validated at + installation time, older :mod:`setuptools` versions fail on attempting to + install :mod:`PySide2` with non-human-readable fatal errors resembling: + + $ sudo python3 setup.py develop + running develop + running egg_info + writing betsee.egg-info/PKG-INFO + writing dependency_links to betsee.egg-info/dependency_links.txt + writing entry points to betsee.egg-info/entry_points.txt + writing requirements to betsee.egg-info/requires.txt + writing top-level names to betsee.egg-info/top_level.txt + reading manifest template 'MANIFEST.in' + writing manifest file 'betsee.egg-info/SOURCES.txt' + running build_ext + Creating /usr/lib64/python3.6/site-packages/betsee.egg-link (link to .) + Saving /usr/lib64/python3.6/site-packages/easy-install.pth + Installing betsee script to /usr/bin + changing mode of /usr/bin/betsee to 755 + + Installed /home/leycec/py/betsee + Processing dependencies for betsee==0.9.2.0 + Searching for PySide2 + Reading https://pypi.python.org/simple/PySide2/ + No local packages or working download links found for PySide2 + error: Could not find suitable distribution for Requirement.parse('PySide2') + +Since these two versions superficially conflict, this string global reduces to +the version satisfying both constraints (i.e., the newest of these two +versions). Equivalently: + +.. code-block:: console + + SETUPTOOLS_VERSION_MIN == max('36.7.2', '38.2.0') == '38.2.0' ''' # ....................{ LIBS ~ runtime : mandatory }.................... diff --git a/betse/util/py/module/pyimport.py b/betse/util/py/module/pyimport.py new file mode 100644 index 00000000..a43b3730 --- /dev/null +++ b/betse/util/py/module/pyimport.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +# --------------------( LICENSE )-------------------- +# Copyright 2014-2019 by Alexis Pietak & Cecil Curry. +# See "LICENSE" for further details. + +''' +Low-level module and package importation facilities. + +This submodule *only* defines functions implementing non-standard and +occasionally risky "black magic" fundamentally modifying Python's standard +importation semantics and mechanics. This submodule does *not* define +commonplace functions for dynamically importing modules or testing or +validating that importation. + +See Also +---------- +:mod:`betse.util.py.module.pymodname` + Related submodule defining functions importing modules by name as well as + testing and validating that importation. +''' + +# ....................{ IMPORTS }.................... +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +# WARNING: To raise human-readable exceptions on missing mandatory dependencies, +# the top-level of this module may import *ONLY* from packages guaranteed to +# exist at installation time -- which typically means *ONLY* BETSE packages and +# stock Python packages. +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +import sys +from betse.util.io.log import logs +from betse.util.type.types import type_check + +# ....................{ REGISTRARS }.................... +@type_check +def register_dir(dirname: str) -> None: + ''' + Register all files and subdirectories of the directory with the passed + dirname to be importable modules and packages (respectively) for the + remainder of the active Python process if this directory has yet to be + registered *or* reduce to a noop otherwise (i.e., if this directory is + registered already). + + Specifically, this function appends this dirname to the current + :data:`sys.path` listing (in order) the dirnames of all directories to be + iteratively searched for any module or package on first importing that + module or package. To comply with Python standards in which the first item + of this list is either the dirname of the directory containing the script + from which this process was invoked *or* the empty string (signifying the + current directory), this list is appended to rather than prepended to. + + Parameters + ---------- + dirname : str + Absolute or relative path of the directory to be registered. + ''' + + # Avoid circular import dependencies. + from betse.util.path import dirs + + # Log this addition. + logs.log_debug('Registering import directory: %s', dirname) + + # If this directory does *NOT* exist or is unreadable, raise an exception. + dirs.die_unless_dir(dirname) + + # If the current PYTHONPATH already contains this directory... + if dirname in sys.path: + # Log this edge case. + logs.log_debug('Ignoring already registered import directory.') + + # Reduce to a noop. + return + # Else, the current PYTHONPATH does *NOT* already contain this directory. + + # Append this directory to the current PYTHONPATH. + sys.path.append(dirname) diff --git a/betse/util/py/module/pymodname.py b/betse/util/py/module/pymodname.py index c4d88e20..c3ba5a79 100644 --- a/betse/util/py/module/pymodname.py +++ b/betse/util/py/module/pymodname.py @@ -6,13 +6,13 @@ ''' Low-level module and package name-based facilities. -All functions defined by this submodule accept only fully-qualified module -names. +All functions defined by this submodule require that modules be passed as +fully-qualified names (i.e., ``.``-delimited). See Also ---------- -:mod:`betse.util.py.module.pyname` - Related submodule whose functions accept imported module objects. +:mod:`betse.util.py.module.pymodule` + Related submodule whose functions accept already imported module objects. ''' # ....................{ IMPORTS }.................... diff --git a/betse/util/py/module/pymodule.py b/betse/util/py/module/pymodule.py index e53a9d6f..e995e19e 100644 --- a/betse/util/py/module/pymodule.py +++ b/betse/util/py/module/pymodule.py @@ -6,8 +6,12 @@ ''' Low-level module and package facilities. -All functions defined by this submodule accept at least a previously imported -module object; most also accept the fully-qualified name of a module. +All functions defined by this submodule require that modules be passed as +fully-qualified names (i.e., ``.``-delimited). + +All functions defined by this submodule require that modules be at least passed +as previously imported module objects; most also accept fully-qualified names +of modules where applicale. See Also ---------- diff --git a/betse/util/py/pys.py b/betse/util/py/pys.py index 88d01ce2..ff41c5cf 100644 --- a/betse/util/py/pys.py +++ b/betse/util/py/pys.py @@ -260,53 +260,6 @@ def get_metadata() -> 'OrderedArgsDict': 'frozen', pyfreeze.is_frozen(), ) -# ....................{ ADDERS }.................... -#FIXME: Shift this utility function elsewhere. While useful, it's rather -#low-level and thoroughly dissimilar from *ALL* of the other functionality -#provided by this high-level submodule. -@type_check -def add_import_dirname(dirname: str) -> None: - ''' - Register all files and subdirectories of the directory with the passed path - to be importable modules and packages (respectively) for the remainder of - the current Python process if this directory has not already been - registered *or* noop otherwise. - - Specifically, this function appends this dirname to the current - :data:`sys.path` listing (in order) the dirnames of all directories to be - iteratively searched for any module or package on first importing that - module or package. To comply with Python standards in which the first item - of this list is either the dirname of the directory containing the script - from which this process was invoked *or* the empty string (signifying the - current directory), this list is appended to rather than prepended to. - - Parameters - ---------- - dirname : str - Absolute or relative path of the directory to be registered. - ''' - - # Avoid circular import dependencies. - from betse.util.path import dirs - - # Log this addition. - logs.log_debug('Registering import directory: %s', dirname) - - # If this directory does *NOT* exist or is unreadable, raise an exception. - dirs.die_unless_dir(dirname) - - # If the current PYTHONPATH already contains this directory... - if dirname in sys.path: - # Log this edge case. - logs.log_debug('Ignoring already registered import directory.') - - # Noop. - return - # Else, the current PYTHONPATH does *NOT* already contain this directory. - - # Append this directory to the current PYTHONPATH. - sys.path.append(dirname) - # ....................{ RUNNERS }.................... @type_check def rerun_or_die( diff --git a/betse_setup/buputil.py b/betse_setup/buputil.py index 5e3b6716..d9830d05 100644 --- a/betse_setup/buputil.py +++ b/betse_setup/buputil.py @@ -73,69 +73,6 @@ def die_unless_setuptools_version_at_least( 'setuptools {} found.'.format( setuptools_version_min, setuptools.__version__)) -# ....................{ GETTERS }.................... -def get_chars(filename: str, encoding: str = 'utf-8') -> str: - ''' - String of all characters contained in the plaintext file with the passed - filename encoded with the passed encoding. - - Parameters - ---------- - filename : str - Relative or absolute path of the plaintext text to be read. - encoding : optional[str] - Name of the encoding to be used. Defaults to UTF-8. - - Returns - ---------- - str - String of all characters decoded from this file's byte content. - ''' - assert isinstance(filename, str), '"{}" not a string.'.format(filename) - assert isinstance(encoding, str), '"{}" not a string.'.format(encoding) - - with open(filename, mode='rt', encoding=encoding) as text_file: - return text_file.read() - - -def get_description() -> str: - ''' - Human-readable multiline description of this application in - reStructuredText (reST) format. - - To minimize synchronization woes, this description is identical to the - contents of the :doc:`/README.rst` file. When submitting this application - package to PyPI, this description is re-used verbatim as this package's - front matter. - - Caveats - ---------- - This function is I/O intensive and hence should be called sparingly -- - ideally, only once by this application's top-level ``setup.py`` script. - ''' - - # Relative path of this application's front-facing documentation in - # reStructuredText format, required by PyPI. This path resides outside this - # application's package tree and hence is inlined here rather than provided - # by the "betsee.guiappmeta" submodule. - DESCRIPTION_FILENAME = 'README.rst' - - # Description read from this description file. - try: - description = get_chars(DESCRIPTION_FILENAME) - # print('description: {}'.format(_DESCRIPTION)) - # If this file is *NOT* readable, print a non-fatal warning and reduce this - # description to the empty string. While unfortunate, this description is - # *NOT* required for most operations and hence mostly ignorable. - except Exception as exception: - description = '' - _output_warning( - 'Description file "{}" not found or unreadable:\n{}'.format( - DESCRIPTION_FILENAME, exception)) - - # Retcurn this description. - return description - # ....................{ SANITIZERS }.................... def sanitize_classifiers( classifiers: list, diff --git a/pyproject.toml b/pyproject.toml index 54e09153..cdd29de3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,6 +20,43 @@ # ....................{ BUILDING }.................... [build-system] +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +# WARNING: To avoid dependency conflicts between "pip", "setuptools", BETSE, +# and BETSEE, the value of this global variable *MUST* be synchronized (i.e., +# copied) across numerous files in both codebases. Specifically, the following +# strings *MUST* be identical: +# * "betse.metadeps.SETUPTOOLS_VERSION_MIN". +# * "betsee.guimetadeps.SETUPTOOLS_VERSION_MIN". +# * The "build-backend" setting in: +# * "betse/pyproject.toml". +# * "betsee/pyproject.toml". +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + # List of all Python packages required to build (i.e., install) this project # from both codebase tarballs and binary wheels. -requires = ['setuptools', 'wheel'] +requires = [ + 'setuptools>=38.2.0', + 'wheel', +] + +# Explicitly notify "pip" that we leverage the top-level "setuptools"-backed +# "setup.py" script as our installation infrastructure. +# +# Note that this is explicitly required for temporary backward compatibility +# with "pip" 19.1.0, which erroneously violated PEP 51{7,8} by failing to +# fallback to a sane default value in the absence of this setting. If this +# setting is left unspecified here, "pip" 19.1.0 fails on attempting to perform +# an editable (i.e., developer-specific) install with the following error: +# +# $ pip install --user --editable . +# Obtaining file:///home/leycec/py/betse +# ERROR: Error installing 'file:///home/leycec/py/betse': editable mode is +# not supported for pyproject.toml-style projects. pip is processing this +# project as pyproject.toml-style because it has a pyproject.toml file. +# Since the project has a setup.py and the pyproject.toml has no +# "build-backend" key for the "build_system" value, you may pass +# --no-use-pep517 to opt out of pyproject.toml-style processing. See PEP +# 517 for details on pyproject.toml-style projects. +# +# See also: https://github.com/pypa/pip/issues/6434 +build-backend = 'setuptools.build_meta' diff --git a/setup.cfg b/setup.cfg index 6cfa01af..0d1476cf 100644 --- a/setup.cfg +++ b/setup.cfg @@ -16,5 +16,8 @@ [metadata] -# Relative path of the file providing this project's license. +# Relative filename of the file defining this project's license. license_file = LICENSE + +# Relative filename of the file describing this project. +long_description = file: README.rst diff --git a/setup.py b/setup.py index 54617534..e8f9ae73 100755 --- a/setup.py +++ b/setup.py @@ -70,6 +70,49 @@ #Ubuntu) for automated packaging of PySide2 applications. See also: # https://build-system.fman.io/ +# ....................{ KLUDGES }.................... +# Explicitly register all files and subdirectories of the root directory +# containing this top-level "setup.py" script to be importable modules and +# packages (respectively) for the remainder of this Python process if this +# directory has yet to be registered. +# +# Technically, this should *NOT* be required. The current build framework +# (e.g., "pip", "setuptools") should implicitly guarantee this to be the case. +# Indeed, the "setuptools"-based "easy_install" script does just that. +# Unfortunately, "pip" >= 19.0.0 does *NOT* guarantee this to be the case for +# projects defining a "pyproject.toml" file -- which, increasingly, is all of +# them. Although "pip" purports to have resolved this upstream, current stable +# release appear to suffer the same deficiencies. See also: +# https://github.com/pypa/pip/issues/6163 +# +# Note this logic necessarily duplicates the implementation of the +# betse.util.py.module import pyimport.register_dir() function. *sigh* + +# Isolate this kludge to a private function for safety. +def _register_dir() -> None: + + # Avert thy eyes, purist Pythonistas! + import os, sys + + # Absolute dirname of this directory, inspired by the following + # StackOverflow answer: https://stackoverflow.com/a/8663557/2809027 + setup_dirname = os.path.dirname(os.path.realpath(__file__)) + + # If the current PYTHONPATH does *NOT* already contain this directory... + if setup_dirname not in sys.path: + # Print this registration. + print( + 'WARNING: Registering "setup.py" directory for importation under ' + 'broken installer (e.g., pip >= 19.0.0)...', + file=sys.stderr) + # print('setup_dirname: {}\nsys.path: {!r}'.format(setup_dirname, sys.path)) + + # Append this directory to the current PYTHONPATH. + sys.path.append(setup_dirname) + +# Kludge us up the bomb. +_register_dir() + # ....................{ IMPORTS }.................... #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! # WARNING: To avoid race conditions during setuptools-based installation, this @@ -79,11 +122,11 @@ # installed at some later time in the installation. # # Technically, this script may import from all subpackages and submodules of -# the this application's eponymous package. By Python mandate, the first -# element of the "sys.path" list is guaranteed to be the directory containing -# this script. Python necessarily searches this directory for imports from the -# local version of this application *BEFORE* any other directories (including -# system directories containing older versions of this application). To quote: +# this application's eponymous package. By Python mandate, the first item of +# the "sys.path" list is guaranteed to be the directory containing this script. +# Python necessarily searches this directory for imports from the local version +# of this application *BEFORE* any other directories (including system +# directories containing older versions of this application). To quote: # # "As initialized upon program startup, the first item of this list, # path[0], is the directory containing the script that was used to invoke @@ -159,7 +202,16 @@ # setuptools or distutils must be added to the above dictionary instead. _SETUP_OPTIONS = { # ..................{ CORE }.................. - # Self-explanatory metadata. + # Self-explanatory metadata. Note that the following metadata keys are + # instead specified by the "setup.cfg" file, + # + # * "license_file", for unknown reasons. We should probably reconsider + # * "long_description", since "setup.cfg" supports convenient + # "file: ${relative_filename}" syntax for transcluding the contents of + # arbitrary project-relative files into metadata values. Attempting to do + # so here would require safely opening this file with a context manager, + # reading the contents of this file into a local variable, and passing + # that variable's value as this metadata outside of that context. (Ugh.) 'name': metadata.PACKAGE_NAME, 'version': metadata.VERSION, 'author': metadata.AUTHORS, @@ -167,7 +219,6 @@ 'maintainer': metadata.AUTHORS, 'maintainer_email': metadata.AUTHOR_EMAIL, 'description': metadata.SYNOPSIS, - 'long_description': buputil.get_description(), 'url': metadata.URL_HOMEPAGE, 'download_url': metadata.URL_DOWNLOAD,