pip completely broken on Windows

[chrullrich]

• Pip version: a25c26c
• Python version: 3.5.2
• Operating System: Windows 10

Description:

Since fc4109e (PR #3906), pip does not work anymore on Windows. Other (non-Linux) platforms, too, possibly.
(I first reported this as a comment on bug #3823, but that is closed and so probably nobody noticed.)

What I've run:

(pipve) C:\Daten>python -m pip list
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\runpy.py", line 174, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\Program Files\Python35\lib\runpy.py", line 133, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "C:\Program Files\Python35\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "c:\daten\projekte\other\pip\pip\__init__.py", line 16, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "c:\daten\projekte\other\pip\pip\vcs\mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "c:\daten\projekte\other\pip\pip\download.py", line 40, in <module>
    from pip._vendor import distro
  File "c:\daten\projekte\other\pip\pip\_vendor\distro.py", line 1040, in <module>
    _distroi = LinuxDistribution()
  File "c:\daten\projekte\other\pip\pip\_vendor\distro.py", line 591, in __init__
    if include_lsb else {}
  File "c:\daten\projekte\other\pip\pip\_vendor\distro.py", line 925, in _lsb_release_info
    raise subprocess.CalledProcessError(rc, cmd, err)
subprocess.CalledProcessError: Command 'lsb_release -a' returned non-zero exit status 1

platform.linux_distribution() did not attempt to run an external command to get LSB release information; the new code does, and then stumbles when the attempt to run a non-existing command fails with exit code 1 rather than 127.

If an lsb_release command returning the correct exit status (127) is provided to simulate the expected failure mode, the next failure is this:

(pipve) C:\Daten>python -m pip list
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\runpy.py", line 174, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\Program Files\Python35\lib\runpy.py", line 133, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "C:\Program Files\Python35\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "c:\daten\projekte\other\pip\pip\__init__.py", line 16, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "c:\daten\projekte\other\pip\pip\vcs\mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "c:\daten\projekte\other\pip\pip\download.py", line 40, in <module>
    from pip._vendor import distro
  File "c:\daten\projekte\other\pip\pip\_vendor\distro.py", line 1040, in <module>
    _distroi = LinuxDistribution()
  File "c:\daten\projekte\other\pip\pip\_vendor\distro.py", line 592, in __init__
    self._distro_release_info = self._distro_release_info()
  File "c:\daten\projekte\other\pip\pip\_vendor\distro.py", line 977, in _distro_release_info
    basenames = os.listdir(_UNIXCONFDIR)
FileNotFoundError: [WinError 3] The system cannot find the path specified: '/etc'

Note that the full distro package (as opposed to the version integrated into pip) defends itself aggressively against failing in this manner:

(pipve) C:\Daten>python
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import distro
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\daten\projekte\other\distro\distro.py", line 39, in <module>
    raise ImportError('Unsupported platform: {0}'.format(sys.platform))
ImportError: Unsupported platform: win32

[jayvdb]

This is completely intentional wrt distro. See python-distro/distro#124 for discussion about it. And a bit in python-distro/distro#132

[chrullrich]

Sorry if I misunderstand, but this is not about distro refusing to work on non-Linux platforms. distro provides information about Linux environments and clearly has no reason, or theoretical ability, to work anywhere else.

This is about the fact that pip has made itself utterly dependent on distro, and hence is now broken on all platforms where distro does not work, i.e. anything but Linux.

[jayvdb]

Yes, my point is that distro choose its current behaviour intentionally, and are not likely to change their API now.

[pfmoore]

Thanks @chrullrich - duelling PRs :-)

I'm not sure my PR is any better than yours, but as you've closed yours I'll merge #3947 once the tests pass. Thanks for the report, and apologies for not getting to it sooner.

[chrullrich]

Yours is better; I hadn't thought of importing it where it is used ...

[pfmoore]

BTW, @dstufft why aren't the appveyor tests required to pass for new PRs? Is there a setting somewhere that needs to be changed to run them and require them to work? I thought @xavfernandez had done a bunch of work to get the tests working on Windows, but it looks like they've bitrotted. Even if we only had a minimal set of Windows tests, they would have caught this issue, so I'd rather mark a bunch of tests as "skip on Windows" for now, than have no tests at all.

If there's a reason for not enabling appveyor, then that's fine, otherwise I'll open an issue for this.

[dstufft]

Are you asking why we don't run Appveyor at all or why are they not required before a PR can merge? AFAICT they aren't running at all though I don't think there is any specific reason for that except maybe I didn't know how to Windows good enough. If they're running and you just want them toggled to required that should be easy too.

[pfmoore]

@dstufft I thought it was enabled, but not marked as "required for a merge". But it's possible I'm wrong - I don't know how to navigate Travis' CI config (for instance, where do I find the CI runs for the current head of the master branch? I wasn't even able to find that to confirm Appveyor was running)

Anyhow, if you're not aware of a specific reason it's not running, I'll do some digging and enable it. Might need some help from you or whoever knows the pypa account password when it comes to setting up an appveyor account, if it turns out we don't even have that...

[dstufft]

• Current branches and the last or pending runs for them: https://travis-ci.org/pypa/pip/branches
• Current PRs: https://travis-ci.org/pypa/pip/pull_requests
• Whatever is running right now: https://travis-ci.org/pypa/pip

[pfmoore]

Ah, OK, that's why I couldn't find them, I was expecting a link in github somewhere (like we have for PRs). Thanks.

[pfmoore]

Fixed via #3947