NuGet cross plat authentication plugin

[infin8x]

Spec: Implemented

This is a fork of #6410 to track related work for separate prioritization.

Users restore NuGet packages using a variety of codepaths and clients: Visual Studio NuGet Package Manager, NuGet.exe, dotnet(.exe), and msbuild /restore and /t:restore. We want to make it seamless to use authenticated feeds (e.g. from VSTS Package Management, MyGet, or Artifactory) in all of those codepaths, without the NuGet team or authenticated feed providers having to write duplicate auth code.

In addition, we want to bundle an authentication plugin with msbuild (when installed with VS or via the Build Tools SKU) and dotnet (when installed via the installer) so that VSTS auth works seamlessly on first run of e.g. dotnet restore, similar to how the Git Credential Manager automatically authenticates VSTS, GitHub, and Bitbucket users after a standard Git for Windows installation.

As part of this work, we should investigate whether to move from the old VSTS NuGet Credential Provider, which is Windows-only, to standardize on the Git Credential Manager, which is cross-plat. And, we want to consider using @dtivel's plugin model (developed for large package support), so there's a single NuGet plugin model.

/cc @anangaur @emgarten @xavierdecoster

[maartenba]

Just an FYI: this already works cross-platform with JetBrains Rider's last preview version - www.jetbrains.com/rider/eap and with the existing plugin model, so wondering what work is left to do here other than hooking up a different executable to prompt for credentials?

[anangaur]

That’s good to know. Just curious: are you using a cross plat (.NET Core) credential provider or different versions for different platforms?
This workitem is to make the current (VSTS) credential provider work for all the platforms with VS, dotnet.exe and msbuild.exe. In addition, we would also be exploring ways to make the plugin model with NuGet better.
We plan to publish the spec soon.

[maartenba]

FYI here's the announcement post we did: https://blog.jetbrains.com/dotnet/2018/03/06/credential-providers-private-nuget-feeds-rider-2018-1-eap/

We've plugged into the existing mechanism and are using a cross-platform JVM-based browser window + OAuth flow. Simple interface similar to what NuGet currently has:

public interface NuGetCredentialProvider {
    public abstract fun getProviderResponse(
        project: com.intellij.openapi.project.Project,
        uri: java.net.URI,
        isRetry: kotlin.Boolean): NuGetCredentialProviderResponse?
}

(PS: in building 33 today in case you want to learn more)

[anangaur]

/cc: @nkolev92 @jeffkl

[maartenba]

2 big questions on the new spec:

1. Will the old mechanism continue to work?
2. Will there be a way to plug in a provider that does not require a separate process? Imagine I am building something using the SDK, and I want to add a credential provider plugin as part of my application. Can that credential provider be part of my application (e.g. same assembly just plugging an interface), or do I have to go the IPC route?

[infin8x]

1. For a while, yes, but eventually we'll deprecate that in favor of the new, more secure plugin mechanism.

[maartenba]

Ok, then 2 becomes interesting :-)

[nkolev92]

@maartenba

There are currently no plans for a plugin to be started in process.

We considered it but decided to go the standalone process route for a couple of reasons:

• Versioning would be painful. The plugins would need to depend on a specific version NuGet dlls and it would have to be rebuilt with every release of NuGet.
  Having a standalone process avoids special assembly loading logic, minimizes the interoperability concerns between the client and the plugin, as they're required to only respect the same protocol.
• Concerns with the potential vulnerabilities of loading a foreign assembly in the product.

If you're using the client SDKs directly you have the ability to override the CredentialService itself.

[maartenba]

Ok so it is possible to override that one?

[nkolev92]

@maartenba

Yes, it's normally setup by the application, not at the API level.
Relevant code.

[jainaashish]

Yes it's intended to be used by NuGet plugins authors...

As Andrew suggested we can proceed with adding ConfigureAwait(false) for Plugin library and discussed with @dtivel as well.

[dtretyakov]

@nkolev92, another error was received while running NuGet restore on mono:

Starting: /usr/bin/mono-sgen /home/jetbrains/Downloads/buildAgent/tools/NuGet.CommandLine.4.8.0-rtm.5369/tools/NuGet.exe restore /home/jetbrains/Downloads/buildAgent/work/71412ac3a5525267/NuGetFeedTest.sln -NoCache -Verbosity detailed -Source http://localhost:8080/httpAuth/app/nuget/feed/NuGetFeedTest/default/v3/index.json -Source http://localhost:8080/httpAuth/app/nuget/feed/_Root/default/v3/index.json -Source https://www.nuget.org/api/v2/
[17:33:30][restore] in directory: /home/jetbrains/Downloads/buildAgent/work/71412ac3a5525267
[17:33:31][restore] NuGet Version: 4.8.0.5369
[17:33:31][restore] MSBuild auto-detection: using msbuild version '15.0' from '/usr/lib/mono/msbuild/15.0/bin'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild.
[17:33:32][restore] MSBuild P2P timeout [ms]: 120000
[17:33:32][restore] /usr/bin/msbuild "/home/jetbrains/Downloads/buildAgent/temp/buildTmp/NuGetScratch/70my6m9f.lkk.nugetinputs.targets" /t:GenerateRestoreGraphFile /nologo /nr:false /v:q /p:NuGetRestoreTargets="/home/jetbrains/Downloads/buildAgent/temp/buildTmp/NuGetScratch/ouvua8kp.ika.nugetrestore.targets" /p:RestoreUseCustomAfterTargets="True" /p:RestoreTaskAssemblyFile="/home/jetbrains/Downloads/buildAgent/tools/NuGet.CommandLine.4.8.0-rtm.5369/tools/NuGet.exe" /p:RestoreSources=\"http://localhost:8080/httpAuth/app/nuget/feed/NuGetFeedTest/default/v3/index.json\;http://localhost:8080/httpAuth/app/nuget/feed/_Root/default/v3/index.json\;https://www.nuget.org/api/v2/\;\"  /p:RestoreSolutionDirectory="/home/jetbrains/Downloads/buildAgent/work/71412ac3a5525267/" /p:SolutionDir="/home/jetbrains/Downloads/buildAgent/work/71412ac3a5525267/" /p:SolutionName="NuGetFeedTest" /p:RestoreBuildInParallel="False" /p:RestoreUseSkipNonexistentTargets="False"
[17:33:34][restore] 
[17:33:34][restore] Restoring NuGet package NuGetFeedTest.0.0.326-beta.1.
[17:33:35][restore] Using /home/jetbrains/Downloads/buildAgent/plugins/nuget-agent/bin/credential-plugin/net46/CredentialProvider.TeamCity.exe as a credential provider plugin.
[17:33:35][restore]   GET https://www.nuget.org/api/v2/Packages(Id='NuGetFeedTest',Version='0.0.326-beta.1')
[17:33:36][restore]   NotFound https://www.nuget.org/api/v2/Packages(Id='NuGetFeedTest',Version='0.0.326-beta.1') 698ms
[17:33:36][restore]   GET https://www.nuget.org/api/v2/FindPackagesById()?id='NuGetFeedTest'&semVerLevel=2.0.0
[17:33:37][restore]   OK https://www.nuget.org/api/v2/FindPackagesById()?id='NuGetFeedTest'&semVerLevel=2.0.0 711ms
[17:33:37][restore] WARNING: Unable to find version '0.0.326-beta.1' of package 'NuGetFeedTest'.
[17:33:37][restore]   http://localhost:8080/httpAuth/app/nuget/feed/NuGetFeedTest/default/v3/index.json: Unable to load the service index for source http://localhost:8080/httpAuth/app/nuget/feed/NuGetFeedTest/default/v3/index.json.
[17:33:37][restore]   401 ()
[17:33:37][restore]   http://localhost:8080/httpAuth/app/nuget/feed/_Root/default/v3/index.json: wintrust.dll
[17:33:37][restore]   https://www.nuget.org/api/v2/: Package 'NuGetFeedTest.0.0.326-beta.1' is not found on source 'https://www.nuget.org/api/v2/'.

OS: Ubuntu 16.04
Mono: 5.12.0.301

Under Verbosity logging NuGet.exe writes only "wintrust.dll" as a error message. Under which mono and os versions did you verified work of the commit NuGet/NuGet.Client@318ddad?

[dtretyakov]

@nkolev92, it looks on mono could be used the AuthenticodeDeformatter for signature verification.

[nkolev92]

@dtretyakov
Mono on linux currently does not work (and it won't work in 4.8).
Thanks for the pointer. I'll look into it.

[dtretyakov]

@nkolev92, so as I understand in the mentioned change you've tried to disable plugins usage under any mono runtime? Or they work fine on mono in macos? If not maybe you could write meaningful message that plugin would not work under mono istead of this runtime exception?

Do you have a feature freeze time in NuGet 4.8?

[dtretyakov]

@nkolev92 , @jainaashish another thought about change NuGet/NuGet.Client@318ddad#diff-e24eb77aabfdbad8c040b73608ca477b: it could be a breaking from the Plugins v1 perspective. They could start failing in NuGet 4.8+ on Mono runtime.

[dtretyakov]

@nkolev92, another significant problem is that NuGet authentication plugin is not used in dotnet nuget push / dotnet nuget delete commands:

[11:58:35][nuget push] Starting: "C:\Program Files\dotnet\dotnet.exe" nuget push C:\TeamCity\buildAgent\work\71412ac3a5525267\NuGetFeedNetCore\NuGetFeedNetCoreLib\bin\Debug\NuGetFeedNetCoreLib.0.0.16.nupkg --api-key ******* --source http://localhost:8080/httpAuth/app/nuget/feed/NuGetFeedTest/default/v3/index.json
[11:58:35][nuget push] in directory: C:\TeamCity\buildAgent\work\71412ac3a5525267
[11:58:36][nuget push] error: Unable to load the service index for source http://localhost:8080/httpAuth/app/nuget/feed/NuGetFeedTest/default/v3/index.json.
[11:58:36][nuget push] error:   Response status code does not indicate success: 401 ().

.NET CLI: 2.1.401-preview-009229

With NuGet 4.8.0 it works fine. Do you plan to address in the scope of .NET CLI 2.1.4xx?

[nkolev92]

@dtretyakov
4.8/2.1.400/15.8 are locked and they are shipping very soon.
There will be no more changes there.

Regarding the mono change, it's not breaking the V1 plugins since the V1 plugins used the fallback signature verifier (always failed validation), which means it's not any worse than before.
The behavior is inconsistent across different installations and versions of mono (versions and OSs), so that'll be addressed in the next release. We'll call out the mono behavior in 4.8 in our release notes/docs.

[AndreyAkinshin]

@nkolev92 is it possible to use credential providers with MSBuild /t:Restore for MSBuild 15.8?

[infin8x]

No, Credential Provider support is rolling out as part of the 15.9 previews.

Per @nkolev92 below, this should be possible with 15.8.

[nkolev92]

Small correction.
15.8 assemblies and it's equivalent ship vehicles do have support for the credential providers.

As part of the 15.9 previews, we have VSTS support within Visual Studio.
//cc @rrelyea

[dtretyakov]

List of current problems:

• Lack of ConfigureAwait(false) in NuGet.Protocol library causes deadlocks while sending log messages
• Credentials plugin does not work under Mono
• dotnet nuget push does not use credentials plugin. Fixed in Setup the credential providers for xplat push NuGet.Client#2395
• dotnet nuget delete does not use credentials plugin. Fixed in Make dotnet delete use the new credential provider plugins NuGet.Client#2414

@nkolev92, @jainaashish, please let me know when these problems will be resolved.

[rrelyea]

I'm planning on closing this issue, to properly reflect that it has shipped in this 4.8 milestone.

@nkolev92 - can you please create/move remaining issues to 4.9 and 5.0 as appropriate.

[phillip-haydon]

@rrelyea so whats the fix? How can I install dependencies from authenticated feeds?

[infin8x]

@phillip-haydon we've just released the Azure Artifacts Credential Provider. If you haven't updated to the required version of dotnet/nuget/msbuild (see readme), you can also use the old mechanism of a plain-text PAT: https://docs.microsoft.com/en-us/azure/devops/artifacts/nuget/dotnet-exe?view=vsts

[bennettfactor]

@alexmullans This may be a silly question, but does this work for non-Azure feeds? The linked repo description implies its Azure-specific.

[infin8x]

@bennettfactor the Azure Artifacts Credential Provider is specific to Azure Artifacts feeds. However, the plug-in mechanism that the Credential Provider uses to plug into NuGet/dotnet/MSBuild is generic. Another feed provider (e.g. MyGet) could write their own credential provider plug-in.

[maartenba]

@alexmullans @nkolev92 With just the netcore plugin, it looks like no credential providers are picked up, and instead an exception is thrown? Does the netcore provider support interactive login (or at least device flow)?

Repro:
MultiThreadedCredentialsRepro.zip (need to make sure no netfx plugin is installed)

NuGet.Protocol.Core.Types.FatalProtocolException
  HResult=0x80131500
  Message=Failed to retrieve metadata from source 'https://xxxxxxxxxxxx.pkgs.visualstudio.com/_packaging/17c08682-207e-46f3-8265-4d7fd84104bb/nuget/v3/query2/?q=&skip=0&take=10&prerelease=true&semVerLevel=2.0.0'.
  Source=NuGet.Protocol
  StackTrace:
   at NuGet.Protocol.RawSearchResourceV3.<SearchPage>d__3.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.Protocol.RawSearchResourceV3.<Search>d__4.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.Protocol.PackageSearchResourceV3.<SearchAsync>d__3.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at MultiThreadedCredentialsRepro.Program.<Main>d__0.MoveNext() in C:\Users\maart\Desktop\MultiThreadedCredentialsRepro\MultiThreadedCredentialsRepro\Program.cs:line 66
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MultiThreadedCredentialsRepro.Program.<Main>(String[] args)

It looks like discovery is hardcoded to netfx and .exe providers when using the NuGet SDK tools? Is there planned support to also have it load .NET Core plugins? (which would make it cross platform)

[nkolev92]

@maartenba
It already is supported, it works with dotnet.exe.
For discovery,
https://github.com/NuGet/NuGet.Client/blob/720f0669d33d4c9874344745629efab12116599b/src/NuGet.Core/NuGet.Protocol/Plugins/PluginDiscoveryUtility.cs#L88
and https://github.com/NuGet/NuGet.Client/blob/720f0669d33d4c9874344745629efab12116599b/src/NuGet.Core/NuGet.Protocol/Plugins/PluginDiscoverer.cs

For launching:
https://github.com/NuGet/NuGet.Client/blob/720f0669d33d4c9874344745629efab12116599b/src/NuGet.Core/NuGet.Protocol/Plugins/PluginFactory.cs#L159-L168

Quickly skimming at your repro, you might be seeing #7438.

[maartenba]

Thanks, that helps!

Our scenario is that we have desktop .NET targeted, running under Mono on Linux. So for discovery and launching, behaviour is based on the NuGet.Protocol target used? So full .NET can not launch a .NET Core provider? And vice-versa?

(This also means nuget.exe under Mono currently does not support the netcore plugins?)

[Edit] further investigation shows plugins are loaded based on the target framework NuGet.Protocol is compiled against. That's perhaps not ideal, but by plugging in some custom implementations for the IPluginResolver, this seems to work now.