Consider updating release.json format to add another level of indexing

[richlander]

The release.json files grow without bound (other than EOL). That's not a great architecture.

This file: https://github.com/dotnet/core/blob/main/release-notes/9.0/releases.json

Could look more like: https://github.com/dotnet/core/blob/main/release-notes/releases-index.json

And then have a release.json file in a directory like: https://github.com/dotnet/core/tree/main/release-notes/8.0/8.0.1

For most use cases, this approach would dramatically reduce the number of bytes downloaded, at least by a naive JSON reader like a serializer. It just occurred to me now, but much of the (example) premise of my Convenience of System.Text.Json post could be underminded by doing this.

[mairaw]

This sort of change would probably warrant a breaking-change announcement given that other people rely on that file for many things I'm not even aware of. Does Azure for example consume that sort of information?

The website is a given but I'm sure there are others.

[richlander]

For sure. We can do better. Also, if we were to come up with a short-link scheme, now would be the time to merge that in.

Another point is that almost no one benefits from 7 Previews and two RCs being included in these big files. That "weight" ends up being a substantial part of a release, particularly for the first year.

6.0 is at 6.0.27. That's 28 releases + 9 pre-release ones. That's 36, so pre-release ones are still 1/4 of the overall file. That's a waste, not to mention folks that only care about the latest release, and many of those likely only care about a very small subset of links.

[Falco20019]

I would love this change as I'm also a fan of smaller linked files over big files. It's definitely a breaking change that at least affects https://github.com/actions/setup-dotnet and most probably also the Azure pipeline.

I would also love if we use the breaking change to extend the JSON schema to include some more information that most probably benefit others like information on operating system support that is currently only maintained in the os-support.md in a non-parsable format.

[Falco20019]

@richlander I did it testwise for .NET 8: Falco20019@c80f409

.NET 9 already started to add the preview1 folder below preview while .NET 8 and prior only had all markdown files in there directly and a named subfolder in preview/api-diff. I therefore would propose to clean this up as well. This also means that the release-notes for previews will change, but since they are linked in the updated releases.json, this should be fine.

My proposal would only leave release-date, release-version, security and release-notes in the releases.json of the channel and move all other (and duplicate) everything that was previously an element of the releases array. I added a release.json (attention: singular) short-link to those elements instead.

[richlander]

One way to avoid the breaking change is to create a peer to releases-index.json, with two options:

• Stop adding new releases to file at .NET 8 (remove .NET 9 from it).
• Continue publishing information to both flavors forever.

I don't think picking a date where we break the format is a good idea. Those plans never work.

[leecow]

Interesting ideas. A per-patch release file (e.g. /8.0.1/release.json) feels unappealing though that may be because I'm not understanding something.

@richlander, can you expand on what you mean by a 'peer' of releases-index.json?

To avoid breaking existing users (and there's a lot of them), perhaps we could create the release.json (latest patch version only) alongside the releases.json (cumulative, maintaining the major release history).

[richlander]

Backing up, the large files we produce seem like an anti-pattern to me, in two ways:

• Almost no one ever needs all (or most) of the data in our releas.json files.
• Large JSON files should generally be avoided

As I said, I wrote a whole blog post on making reading these files efficient. It took a lot of effort. We shouldn't require our users to do that.

[leecow]

I don't disagree with any of that, though we have used the historical information in the 'complete' releases.json on a number of occasions. The clarifying question is along the lines of mitigating the breaking change while making this better for consumers. Something like:

• Implement a single, channel-located (e.g. https://github.com/dotnet/core/blob/main/release-notes/9.0/release.json) containing only the latest channel release information. Same schema as releases.json but only the latest release data.
• releases.json remains as-is but all documentation is updated to direct folks to the single file.
• Decide if and when the releases-index.json should be updated to point to the singleton providing a functional migration point to the new data for consumers that rely on the index for discovery.

[richlander]

we have used the historical information in the 'complete' releases.json on a number of occasions

I don't see why this would ever be a problem. If there was a 9.0 index, you'd be able to get all the same information.

[richlander]

This probably isn't that hard. We just need to create another link here:

core/release-notes/releases-index.json

Line 14 in f424bc4

"releases.json": "https://dotnetcli.blob.core.windows.net/dotnet/release-metadata/9.0/releases.json"

Like:

        {
            "channel-version": "9.0",
            "latest-release": "9.0.0-preview.1",
            "latest-release-date": "2024-02-13",
            "security": false,
            "latest-runtime": "9.0.0-preview.1.24080.9",
            "latest-sdk": "9.0.100-preview.1.24101.2",
            "product": ".NET",
            "release-type" : "sts",
            "support-phase": "preview",
            "latest-release": "https://dotnetcli.blob.core.windows.net/dotnet/release-metadata/9.0/9.0.1/release.json",
            "releases.json": "https://dotnetcli.blob.core.windows.net/dotnet/release-metadata/9.0/releases.json"
        }

[leecow]

Precisely what I was suggesting with regards to the new file. The additional index link fills in the picture.

[Falco20019]

We can also add another releases-index.json per channel and link them in the main index. Than it's up to the caller if they follow the releases.json or releases-index.json link. No breaks involved.

I mean in addition to the proposed latest-release link.

[richlander]

I was also thinking about that. I think that's a good idea. We can then position the big releases.json as a legacy option.

[Falco20019]

One example I prepared a couple days ago:

Falco20019@d6bb627

Just think of the 9.0/releases.json to be named 9.0/releases-index.json instead and the 9.0/releases.json to look unchanged.

Will create a full draft including all touched files tomorrow since I'm out of office today.

Do you know if anyone relies on the structure of https://github.com/dotnet/core/tree/main/release-notes/8.0/preview and sub-folders? Because if not, we could align it with how .NET 9 uses a sub-folder for the preview1 to make it cleaner and have the release.json in there next to the markdown per version.

[richlander]

Just think of the 9.0/releases.json to be named 9.0/releases-index.json instead and the 9.0/releases.json to look unchanged.

Do you have a typo there? You mention 9.0/releases.json twice.

We'd be very hesitant to change the 8.0 preview content. It is now in the past and putting effort into that or breaking people isn't desirable. We want to focus our efforts on forward-looking improvements.

[Falco20019]

No typo, maybe just written confusing. This was in reference to the linked commit. The change listed says 9.0/releases.json but should be 9.0/releases-index.json. I prepared it before that was discussed.

And the second mention meant that beside the name, the original one is kept as discussed here.

About .NET 8, we would still need to split it up at least for the versions under support in my opinion to make the index useful. But we can definitely do it non-breaking. Just wanted to make sure where the limits are 👍

[richlander]

I understand your wording now.

My thinking is that we'd start this scheme with (for example) 8.0.3 and just move forward. Backporting the scheme would be nice, but it isn't clear that it is needed.

[Falco20019]

I can have a look since I already did it test wise for the full .NET 8 and 9 releases. I think having .NET 6 through 9 should be enough, but as it just feels incomplete, I might try to generate the rest as well for completion sake.

Sorry if my wording is sometimes confusing. I’m no native speaker.

[richlander]

Sorry if my wording is sometimes confusing. I’m no native speaker.

Hearing you loud and clear.

Let's validate we have a good model for the in-support versions and then if you want to backfill date, you can.

[Falco20019]

I created a draft PR for discussion (only for .NET 8 and .NET 9, fully non-breaking). I also extracted the CVE part and introduced a separate index for it (open for discussion).

[martincostello]

Throwing my hat into the ring as a consumer of this data.

I maintain a GitHub Action that consumes this data for automating patches: martincostello/update-dotnet-sdk.

I have various other things built on top of this data for automation purposes as well ("Is your internal application running the latest patch version of .NET?", "Download the latest two versions of the .NET SDK", "Is this version of .NET still in support?").

If things are going to be changed in a breaking manner, then for myself at least, I'd need a few weeks' notice of the final format (and ideally a side-by-side deployment period) to update all these things to the new data format.

[Falco20019]

@martincostello We will make sure to only introduce the new format in a non-breaking manner. Any existing file must not be changed in a way not covered/permitted by JSON schemas. Adding fields might require an updated JSON schema, but should not break anyone.

Most changes will be in new files and should be introduced in parallel (maybe even marked as experimental for now) to gather input. That was at least already suggested by @leecow in a previous comment and I do like that approach.

[leecow]

Another point to mention is that the existing assets are machine generated and maintained, and the new ones will be as well. While we can iterate over ideas in 'experimental' PRs, the final PR implementing what we decide upon will be the result of tooling updates.

[Falco20019]

I extended my draft #9210 to include os-support and cve information as new data as this is something that I would love to have in a form that would help me generating some requirements for our customer documents. Basically making information that was previously only available in markdown files machine readable.

It would be great to hear from others if they miss something or have more input.

[martincostello]

Being able to determine which repo(s) a CVE applies to would definitely have value. For example, a CVE related to WPF wouldn't be relevant for an ASP.NET Core app.

[richlander]

Now that we have fancy markdown releases notes in this repo, I'm wondering if we should move our CVEs from dotnet/announcements to a cves directory (peer to 9.0), with years as children. We'd then rely on our digest posts in dotnet/announcements to document the CVEs as a group. If folks really want us to keep on posting CVEs to dotnet/announcements, we can do that too.

We document CVEs as markdown, but don't share the markdown with you. That's not intentional, but a function of the way GitHub issues works. We publish markdown and you get HTML. A major goal of this new effort is to share markdown since it is a lot more flexible for re-use than HTML.

We're currently working on a cve.json format as part of this project. Each CVE object would then link to the markdown file (with a regular GH URL, but switching that to the raw variant should be trivial should that be desired).

Good idea?

I also took the opportunity to clarify the license of all of this content. I hope that's favorable.

• Markdown release notes: https://github.com/dotnet/core/tree/main/release-notes/9.0/preview/preview3
• CVE post: Microsoft Security Advisory CVE-2024-21409 | .NET Elevation of Privilege Vulnerability announcements#303
• Digest post: .NET April 2024 Updates announcements#302
• License: https://github.com/dotnet/core/blob/main/release-notes/license-information.md

[richlander]

This issue is too long and some of the conversation (my fault) is confusing. I'm going to create more narrow issues going forward on this general topic.