ConfidentialClientApplication.AcquireTokenForClient fails with Microsoft.Identity.Client.MsalClientException: MSAL V3 Deserialization failed

[fahd-ms]

Logs and network traces
Without logs or traces, it is unlikely that the team can investigate your issue. Capturing logs and network traces is described in Logging wiki.

Which version of MSAL.NET are you using?
MSAL.NET 4.40.0.0

Platform
.NET 4.7

What authentication flow has the issue?

• Desktop / Mobile
  • Interactive
  • Integrated Windows Authentication
  • Username Password
  • Device code flow (browserless)
• Web app
  • Authorization code
  • On-Behalf-Of
• Daemon app
  • Service to Service calls

Other?
Using token cache serialization from Microsoft.Identity.Web.TokenCache 1.22.1

• CCA is configured to use DistributedCache using the AddDistributedTokenCache method
• MsalDistributedTokenCacheAdapterOptions has Encrypt=true
• Keys are protected using ProtectKeysWithCertificate and shared via Redis
• The flow was working till the DataProtectionCertificate used ^^ was rotated. (bug -> [Bug] configuring multiple decryption certificates when rotating a cert microsoft-identity-web#1575)

Looking at the stack trace (attached file) and the code, it looks like DeserializeMsalV3 fails when UnprotectBytes is unable to decrypt the token in Microsoft.Identity.Web.TokenCache, it still tries to deserialize encrypted token and fails as that is not a valid json string. Link to code:

• https://github.com/AzureAD/microsoft-identity-web/blob/d26e79ddf9fdd2e1fa740711ec652cb2a221fc1f/src/Microsoft.Identity.Web.TokenCache/MsalAbstractTokenCacheProvider.cs#L101

• https://github.com/AzureAD/microsoft-identity-web/blob/d26e79ddf9fdd2e1fa740711ec652cb2a221fc1f/src/Microsoft.Identity.Web.TokenCache/MsalAbstractTokenCacheProvider.cs#L105-L121

• stack trace

Is this a new or existing app?
a. The app is in production, and I have upgraded to a new version of MSAL and the DataProtectionCertificate was rotated.

Repro

• Enable Distributed cache with encryption using Redis.
• Rotate the certificate used to protect token encryption keys.
• Code for building/configuring ConfidentialClientApplication:

IConfidentialClientApplication app = ConfidentialClientApplicationBuilder.Create(appId)
				.WithCertificate(clientCertificate)
				.WithAuthority(authorityUri)
				.WithAzureRegion()
				.WithCacheSynchronization(false)
				.WithLegacyCacheCompatibility(false)
				.Build();
app.AddDistributedTokenCache(services =>
{
	// Adding data protections helps secure the keys that are used to encrypt the token cache.
	services.AddDataProtection(o =>
	{
		o.ApplicationDiscriminator = DataProtectionApplicationDiscriminator;
	})
	// Key material needs to be shared so it can be available to different machines/processes.
	// Using Redis to store the keys and then encrypt these keys with the X509 data encryption certificate.
	.PersistKeysToStackExchangeRedis(
	() => dataProtection.Cache.GetDatabase(),
	  dataProtection.KeyName)
	.ProtectKeysWithCertificate(dataProtection.EncryptionCertificate);

	services.Configure<MsalDistributedTokenCacheAdapterOptions>(o =>
	{
		o.Encrypt = true;
		o.OnL2CacheFailure = (e) =>
		{
			// Retry cache operation? false - no, true - yes.
			return true;
		};
	})
	.AddStackExchangeRedisCache(o =>
	{
		o.ConfigurationOptions = dataProtection.Cache.ConfigurationOptions;
	});
});

Expected behavior
AcquireTokenForClient should not fail on L1/L2 cache read and/or write failure and should return access token directly from AAD.

Actual behavior
AcquireTokenForClient fails with MSALClientException MSAL V3 Deserialization failed.
stack trace

[trwalke]

Understood, so it looks like we should try to acquire a new token if we are unable to read the bytes from the cache since they key used may have been rotated. Any idea what the implications of doing this will be @bgavrilMS, @pmaytak?

[bgavrilMS]

We will need to look at this in more detail. In the L1/L2 cache, the L2 cache operations are already set to fail silently. However, the encryption occurs in this base class, no tin the L1/L2 class.

I also wonder if the ASP.NET Core data protection infrastructure doesn't take this case into account.

[bgavrilMS]

CC @jennyf19 for an opinion

[pmaytak]

@fahd-ms Specifically for the issue related to key encryption certificates, Data Protection API has UnprotectKeysWithAnyCertificate method that can accept new and old certs. Have you tried this?
For a more general issue of deserialize failing, we're investigating a better way of handling the exception.

[fahd-ms]

Thanks @pmaytak - I'll try this out.

[pmaytak]

@fahd-ms Were you able to try it out, did it help?

[pmaytak]

Closing, please reopen if issue is not resolved.

Not much can be done on MSAL.NET side; encryption must be correctly set up user-side. Microsoft Identity Web will have a better exception and logging in 1.23.1. Scenario documented in https://aka.ms/ms-id-web/token-cache-troubleshooting.

[jennyf19]

released in 1.23.1