Type Stripping in node_modules/*/*.ts

[karlhorky]

Copied from original discussion in nodejs/loaders#217 (comment)

---

I know that as a user I won't have much sway here, but I think I'm not alone in thinking that files in node_modules/*/*.ts should also be allowed --experimental-strip-types type stripping.

A few thoughts:

1) Node.js Alone Won't Prevent TS npm Publishing

There is a lot of movement in the "execute TypeScript, don't build it" ecosystem already, for example with the advent of many ecosystem projects such as:

• TypeScript execution CLIs like ts-node, tsm, tsx
• TypeScript-native runtimes like Bun and Deno
• modern TS-native registries like JSR
• somewhat related: ECMAScript proposal Type Annotations

Publishing .ts files to npm and other registries is already a growing pattern, which drastically reduces the effort and complexity of publishing a TypeScript package.

I think as these projects evolve and more people get to know about this possibility, there will be a greatly increased interest in this possibility.

It's my opinion that this banning of node_modules/*/*.ts in Node.js alone won't be able to hold back the energy in the ecosystem, especially with working alternatives. (not to mention users circumventing this by using any of the other projects, which do not have this limitation)

2) User Experience

My first experience with node --experimental-strip-types was creating an internal checking tool which was its own package, without a build step.

I immediately fell flat on my face with this, because of the banning of node_modules/*/*.ts files.

Even though I was already somewhat aware of the discussions and objections behind this (eg. the nodejs/TSC meeting notes from 2024-07-24), for my use case it felt like an arbitrary limitation imposed on my project.

I imagine that as type stripping becomes more widespread, this will be a common complaint from users.

3) Node.js Features Available Everywhere

@GeoffreyBooth wrote about supporting a Node.js feature everywhere:

If we support a file type, we support it everywhere. Just as module detection needs to apply within node_modules so that a package that works locally continues to work when published to npm, the same applies to TypeScript. We shouldn't be limiting our functionality to try to encourage best practices.

This resonates with me - it seems like supporting a Node.js feature in as many places as possible will be the best for users and cause the least amount of support complaints.

Banning node_modules/*/*.ts to try to avoid a pattern which the ecosystem may adopt seems like a weak reason. (not to mention that the ecosystem may end up adopting this pattern anyway by finding cowpaths around this)

4) Identifying and Addressing Downsides

Downsides as mentioned by Ryan and Daniel on the TypeScript team and vocal node_modules/*/*.ts opponents such as Matteo are important and should be considered.

But it feels like also:

1. Each problem should be carefully thought through, examined, clearly enumerated and documented (also asked for by Geoffrey)
2. Once clearly enumerated and documented, solutions should be also thought through, with each solution's tradeoffs being also enumerated and documented

This being done with a dispassionate, neutral stance, not trying to prove any one viewpoint.

In reading through the publicly-available documents, I don't think this has been done yet.

5) At Least Allow Users to Choose

Even if careful enumeration of problems and solutions leads to the final conclusion that the tradeoff is not worth it for Node.js, I think it would be good to allow users to choose for themselves and disable this node_modules/*/*.ts banning with a flag, eg:

node --experimental-strip-types-in-dependencies

6) Similarities to .ts Rewriting Limitation

Oh one more thing, more related of a feeling than fact-based:

This node_modules/*/*.ts banning feels like it has a similar vibe to the .ts -> .js rewriting decision in TypeScript, which was overturned after years of doubling-down on .js imports (ironically heavily inspired by Node.js type stripping).

[marco-ippolito]

Adding a flag to allow type stripping in node_modules is trivial.
For a real solution the issue is how does node recognize if its published npm package or a monorepo type project?
If there was a marker in the package.json that would be possible.
Immagine an entry in the package.json "local_node_modules": true that prevented publishing to npm (when you try npm publish it would throw) but if node finds it, it allows type stripping even in the node_modules.

[JakobJingleheimer]

If there was a marker in package.json that would be possible.

Isn't there? The namespace

---

I'm in favour of allowing publication of ts in general (as I'm also in favour of publishing jsx) because it allows the consumer to decide the output and avoids make-work steps and a game of telephone.

[aduh95]

At Least Allow Users to Choose

I think that's possible, with loader hooks, Marco have been working on exposing the type stripping utilities for this exact use-case IIRC.

[karlhorky]

how does node recognize if its published npm package or a monorepo type project?

@marco-ippolito I think maybe first needs to be identified:

1. What is the purpose of banning published npm packages with .ts? (What are all of the enumerated downsides and their symptoms?)
2. Should these downsides / banning also apply to packages published to other registries? (eg. private registries or TS-native registries?) If yes/no, why?
3. What are possible solutions or improvements to those downsides?

It seems like trying to ban users from publishing .ts files is maybe the wrong thing to be putting effort into?

Maybe better would be 4) Identifying and Addressing Downsides, listing out the real downsides which occur as a result of publishing .ts files, and then try to work on those problems / symptoms?

[JakobJingleheimer]

It is possible with a loader hook: https://github.com/jakobjingleheimer/nodejs-loaders#jsx--tsx

(This is about to be split into a monorepo and published separately as @nodejs-loaders/tsx)

[marco-ippolito]

Btw as @aduh95 mentioned, if parser.stripTypeScriptTypes nodejs/node#55282 lands, its possible to use an external loader that supports type_stripping in node_modules with the same output as node. amaro (should) already cover this use case when used as a loader

[mcollina]

I agree that we should document the downsides of not supporting .ts files inside downloaded npm packages are not documented, and it would be great if @DanielRosenwasser and the rest of the typescript team voiced their concerns. They are the same as mine, just expressed in a much more precise form. I could take a stab at writing those concerns if they prefer (for them to review). This is important.

Summing them up are:

1. TypeScript is designed so that only .d.ts files are shared across projects. They are interface files.
2. Assuming a significant portion of the npm ecosystem will be replaces by ts-only modules, it would slow down massively our IDEs, because the numbers of parsed files would increase (10-100x?).
3. Allowing only-TS dependencies will force a single tsconfig.json in the ecosystem.

[hverlin]

modern TS-native registries like JSR

jsr allows ts files, but has some restrictions (https://jsr.io/docs/about-slow-types#typescript-restrictions).
Following them might help to address the performance issues. (There is also the isolatedDeclarations option)

[jakebailey]

My first experience with node --experimental-strip-types was creating an internal checking tool which was its own package, without a build step.

I immediately fell flat on my face with this, because of the banning of node_modules/*/*.ts files.

Theoretically, this shouldn't happen for plain monorepo use; the determination of "is in node_modules" considers the realpath of a file. But if you aren't using a monorepo tool with symlinks, I can see how this would fail.

Publishing .ts files to npm and other registries is already a growing pattern, which drastically reduces the effort and complexity of publishing a TypeScript package.

The reduction in complexity here is from the tooling, though; something like jsr handles the downleveling and d.ts emit for you behind the scenes. Just publishing TS code itself doesn't reduce much complexity in a world which really does need to have/run .js files... (I think it has the potential to actually increase complexity, without some careful consideration.)

jsr allows ts files, but has some restrictions (jsr.io/docs/about-slow-types#typescript-restrictions).
Following them might help to address the performance issues. (There is also the isolatedDeclarations option)

These kinds of restrictions do not help published TS source; TypeScript will load all of the source code given to it, including function/class bodies, expressions, etc. Publishing and loading .d.ts files strips away all of those interior bits and allows TypeScript to do less work.

"Slow types" is a bit of a misnomer becuase it's talking about "slow" in terms of generating declaration files from the perspective that the "slow types" require running tsc to typecheck. isolatedDeclarations encodes a list of rules that prevent needing type checking, and is therefore "fast", but the speed at which declaration files are emitted does not change the fact that loading d.ts files is faster than actual source code. It'd be nice if JSR's docs were modified to not call them "slow types" anymore.

There are other concerns here as well:

• If users don't set --skipLibCheck (which, they arguably shouldn't in order to understand when their deps are broken), then they're going to be checking all of that extra source code too, including fully-annotated code.
• Additionally, a package's tsconfig may not have the same settings as the user; depending on the differences, new errors may appear that wouldn't be there if .d.ts files are consumed instead (e.g., exactOptionalPropertyTypes, noUncheckedIndexAccess, etc, in the consuming package). This is also why transforming someone else's TS code is dangerous; there are options that some other package may have set which would affect emit, e.g. decorator use (including uses that are not exported!), useDefineForClassFields, etc.
• Publishing TypeScript source may limit downstream users abiliy to not use the same TS version as you. Today, one runs tsc (or similar), leaving behind only JS (not read by tsc), and .d.ts without anything from expresion space. This means that TS is free to introduce a feature like satisfies and it'll be backward compatible for users of your package as the satisfies keyword does not appear in d.ts files. If you publish source, the version of TS you use to typecheck your code impacts downstream users.
  • This is technically no different than adding ES features, except that basically all TS syntax is erasable.
  • TS may also eventually remove something or be forced to change the emit of something altogether, e.g. to deal with enum, a new module keyword in JS, and so on.

All of these problems are handled by transforming your code before publishing it.

(My list is not exhaustive either; a proper combined list would of course be good.)

[marco-ippolito]

What if we supported running in node_modules only if the package.json containts the "private": true property https://docs.npmjs.com/cli/v10/configuring-npm/package-json#private which prevents a package being published.
This way you still use monorepos but wont disrupt the ecosystem

[karlhorky]

Downsides of Publishing .ts to npm

So if I may try to make a simplified list of problems:

1. Loading .ts source code is slower than loading d.ts files
2. Checking .ts source code in node_modules without --skipLibCheck is slow
3. Publishing newer / different TS syntax than the consumer tsconfig.json / TS version causes failures (similar to publishing newer ES syntax than consumer supports)

(happy to edit this or for someone to take over with a new list, just trying to make a simplified list)

Metrics

It would be cool to get some symptoms (metrics) on these things.

For example, with a small sized project that has 10 direct dependencies (and a number of transitive dependencies), and considering that 100% of the dependency graph converted to TypeScript, what would the concrete impacts be in latest VS Code on a 2020 mid-range computer?

By metrics I mean something like these fake numbers:

1. Loading .ts source code is slower than loading d.ts files
   • VS Code startup time: +1s (+80%)
   • TypeScript extension startup time: +2s (+20%)
   • RAM: +200MB (+10%)
   • Auto-completion response time: +0.5s (+15%)
2. Checking .ts source code without --skipLibCheck is slow
   • Full project build time: +2s (+15%)
   • Type checking for 10 dependencies: +1s (+10%)
   • RAM usage during type checking: +50MB (+5%)
   • Incremental builds time increase: +0.5s (+5%)

Update: @andrewbranch did some research and produced some similar metrics here:

.d.ts / .ts

Files:           268 / 317
LOC:             49k / 62k
Instantiations:  11k / 46k ❗
Memory:          92 kB / 118 kB
Parse time:      .76 s  / .91 s
Check time:      1.48 s / 3.02 s ❗
Total time:      2.58 s / 4.43 s ❗

• https://x.com/atcb/status/1705675339626909911

[karlhorky]

With clear metrics, it will be more clear what the tradeoffs are and what could be solved, optimized or worked around to minimize those downsides.

Some projects may decide that the tradeoff is worth it:

• 👎 decreased performance, compat considerations
• 👍 no build setup, configuration, maintenance, CI step

[electrovir]

What if we supported running in node_modules only if the package.json containts the "private": true property

I personally have, and I've seen many other projects have, mono-repos with npm workspaces (packages) that are not private, they are published as individual packages. The mono-repo's top-level package.json does set "private": true though.

[marco-ippolito]

What if we supported running in node_modules only if the package.json containts the "private": true property

I personally have, and I've seen many other projects have, mono-repos with npm workspaces (packages) that are not private, they are published as individual packages. The mono-repo's top-level package.json does set "private": true though.

Its a trade-off. If you want to run .ts in your node_modules, then you should do it locally.
private ensures that the package cannot be published.
This will work for some use case (projects that do not need publishing on NPM).

[targos]

There's another problem with publishing .ts source code: It must be compatible with the tsconfig flags of the consumer (strict, verbatimModuleSyntax, etc.).

[karlhorky]

@targos yeah, that was mentioned above a couple of times, including my list above:

3. Publishing newer / different TS syntax than the project tsconfig.json / TS version (similar to publishing newer ES syntax) causes failures

I'll reword to be more clear

[robpalme]

On the list of reasons @jakebailey stated for why redistributable TS on npm is a cause for concern, I will add my agreement (and expand) on a couple of specific points raised regarding future compatibility of TS syntax.

TS may also eventually remove something or be forced to change the emit of something altogether, e.g. to deal with enum, a new module keyword in JS, and so on.

• enum might become a JS keyword if it comes back for more discussion at TC39. It was only blocked last time on a technicality - the inability to frame the problem statement - rather than any lack of a desire for the functionality. At the meeting last week I spoke to one of the folk who previously pitched it for Stage 1 and they remain interested. There are active discussions about permitting more syntax sugar in the JS language that might ease the ability to progress such features in future.

• The legacy module keyword for TS namespaces is under consideration for future deprecation and several steps have already been taken towards dissuading further usage. There are a bunch of module-related JS language features in progress and at least one might use this keyword.

These are just possibilities. It's hard to say whether they will play out. Regardless, it's very cheap for Node to be conservative and avoid complicating future evolution. We just need to avoid third-party redistributable code (i.e. npm deps) depending on Node supporting that syntax. Happily --experimental-strip-types already side-steps both of these cases 👍

[jakebailey]

What if we supported running in node_modules only if the package.json containts the "private": true property docs.npmjs.com/cli/v10/configuring-npm/package-json#private which prevents a package being published.
This way you still use monorepos but wont disrupt the ecosystem

I don't understand why this helps; if the package is private, then it can't have come from a registry. If you're in a monorepo and doing this, then the package can only have been local and could be symlinked and not need anything to work today.

[jakebailey]

It would be cool to get some symptoms (metrics) on these things.

@andrewbranch did the analysis and shared his results here about a year ago: https://x.com/atcb/status/1705675335814271157

[karlhorky]

@jakebailey thanks! Updated my post above with those numbers.

[benjamingr]

With numbers, I'm -1 on transpiling/stripping in node_modules, I'm fine with having an opt-in flag in the package.json if we can reliably prevent published packages from transpiling but since we have a workaround (a loader + exposing parser.stripTypeScriptTypes) I'd personally prefer a better default (not transpiling node_modules) and having users opt in with the loader.

[robpalme]

To be clear, Andrew's numbers are comparing the cost of checking TS vs DTS. It seems fair to conclude we should continue encouraging distribution of DTS files in packages for faster checking of dependencies. That will help keep tsc and IDE usage fast.

This could be considered somewhat independent of whether the executable code in the package is TS or JS.

[anthonyshew]

(@karlhorky messaged me wondering if I had an opinion so just going to jot it down here.)

TL;DR: I pretty much agree with what TS core team is saying above. The way I'm building monorepos today, I wouldn't need this flag.

My high-level thought these days on internal packaging for monorepos is: Compile like you're shipping to npm*, even though you're not.

I find this simplifies the mental model. Your package consumers are going to look into node_modules to find the symlinks for the internal package's source code anyway**, so we can treat that boundary the same way, whether the package happens to be coming from npm or your monorepo. You then get to lean into the assumptions TypeScript makes about node_modules, the way Node.js approaches resolution, etc., instead of working against the ecosystem. While its convenient to export TypeScript from a package sometimes, I've been starting to change my tune about some of the tradeoffs.

As far as what's proposed in this issue goes, I admittedly don't have much of an opinion. There are a lot of different ways to build a monorepo today, many of which I'm not fond of. I try to teach folks my little happy path that I've carved out, but there are still plenty of valid, wonky things you can do, in the true spirit of JavaScript. If one were to follow the way that I happen to teach Workspace-building, this flag would be unnecessary, because there would be .js and .d.ts in each package, ready to be executed as if they came from npm.

*: There are exceptions to this idea, like dual-publishing. In your monorepo, you're in control, so you don't necessarily have to cater to the unknown matrix of consumers of a public npm package. Example: you know your monorepo is committed to using ESM, so you don't have to dual-publish.

**: I'm working from the assumption that you're not using compilerOptions#paths to share your packages amongst your workspace, properly using your package manager instead.

[marco-ippolito]

Id add a flag to enable typescript in node_modules after the feature gets unflagged. Id also want to check if we can make it compatible with jsr 🌝

[DanielRosenwasser]

This is on the agenda for tomorrow's meeting, so I'll try to summarize the official TypeScript stance on why we are currently against running TypeScript source files within node_modules.

The TL;DR is:

1. Node.js users and maintainers are going to have a difficult time with versioning, configuration, and compatibility.
2. Even if collectively had a satisfactory way to handle those issues, the TypeScript project is nowhere near ready with a complementary solution that has a good developer experience.

Advantages

There are a few main advantages we see in running TypeScript within node_modules.

First, such a feature can simplify the publishing process for library authors because they will not need a build step before publishing to the registry. These developers can ship their code "as-is".

The other advantage is that it can make it easier for users to debug library code, since sourcemaps would not be necessary (and sourcemaps are often either incorrect or tools may have issues in their sourcemap support).

There is also another possible advantage, which is that it can make it easier for users to explore the implementations of their dependencies. Whereas today TypeScript will prefer jumping to .d.ts files in the absence of a declaration source map, one could imagine that TypeScript would always jump to the source definition.

Disadvantages

Unfortunately, we believe the disadvantages outweigh the advantages.

The biggest disadvantage we see is that this feature would cause versioning headaches around TypeScript feature usage. If a library author decides to ship TypeScript syntax that is not understood by a specific version of Node.js or TypeScript, this makes the situation more complicated for library consumers. While TypeScript declaration files occasionally still have this issue, it does not affect Node.js users directly, it is much rarer, and users can apply a post-processing step to "downlevel" the .d.ts files. By encouraging library authors to ship JavaScript files and declaration files, these libraries become more broadly compatible. We believe that whatever TypeScript-specific features a library authors uses in their function bodies should remain an implementation detail.

A similar, but not entirely related issue, is that there is not a consistent strategy for how to handle any arbitrary TypeScript files in the absence of a tsconfig.json. Source files are not necessarily fully annotated, meaning that checking against a dependency also requires type-checking the implementations of functions and variables. It is very questionable whether users would want to, or even could, type-check their dependencies' source code with type-checking options from their own tsconfig.json. For example, if a user wants to type-check their code with all --strict options, but a library author wants to disable --strictClassInitialization, the library code could have type-checking errors that are out of the user's control.

Finally, there is the issue of more code to analyze, along with the need to check the implementation of source code. We've seen a few people look into the size delta between source files and declaration files (here and here). Furthermore, as mentioned above, source files typically would need to have their implementations type-checked. So shipping TypeScript source files would probably require one of two strategies for the TypeScript type-checker: either increase the size of all code analyzed by TypeScript all at once, or perform an up-front analysis of all dependencies ahead of time and cache the results as .d.ts files in the background. The latter is not undoable, but fairly complex given that today TSServer (the TypeScript language server) does not persist intermediate results, and watching node_modules for changes is a common source of issues. Either way, one of the biggest complaints from TypeScript users is performance and resource usage, so it seems undesirable to rebuild all dependencies at least once per install. The current constraints mean that library authors must do this once prior to publish, and everyone else benefits from a faster experience after installing.

Conclusion

While it is possible that some of these disadvantages could be mitigated, we believe that the disadvantages are significant enough that we would discourage this feature. We are open to hearing more feedback on this topic and revisiting, but we believe that the current strategy of shipping JavaScript and declaration files is the best strategy for the ecosystem as a whole.

[shinebayar-g]

I think it's fair to say that nobody expects any type checking in the node_modules directory. Since it’s out of the end user's control, it’s meaningless. TypeScript can continue ignoring node_modules as it currently does.

If this feature proposal is ever implemented, I’d expect only one functionality from it:
.ts files in the node_modules directory should behave the same as any other .js files in the node_modules directory. No special treatment is needed there. NodeJS has its own ECMAScript compatibility, so type stripped .ts files in node_modules could target that.

I’m not sure if it’s too much to ask for TypeScript to read the tsconfig.json file's target and module settings and attempt to downlevel on the fly according to those values. If that's unrealistic, then whatever javascript code left after the type stripping is probably fine.

After all, your target users are nodejs backend developers, who would never need any special browser bundling in the first place. If a library is expected to run in the browser, then it should absolutely be shipping .js files.

[jakebailey]

I think it's fair to say that nobody expects any type checking in the node_modules directory. Since it’s out of the end user's control, it’s meaningless. TypeScript can continue ignoring node_modules as it currently does.

I don't think that's entirely fair to say (many projects do not enable skipLibCheck), and even without checking enabled, performance is negatively hit as all of the internals will still be parsed, bound, and potentially still incur type checking for inference or other checks the compiler still does even if no errors are shown.

I’m not sure if it’s too much to ask for TypeScript to read the tsconfig.json file's target and module settings and attempt to downlevel on the fly according to those values.

I believe it is unrealistic to attempt to read any tsconfigs but your own; tsconfigs can extend configs from devDeps (e.g. the @tsconfig/* packages), or may not even be in the same tree at all (monorepos with a base config at the root used by packages in subdirs).

[karlhorky]

Has anyone considered usage of fields like engines or devEngines (discussed in nodejs/node#51888, among other places) to express the TS features and version of a library to consumers?

I'm not sure this would completely mitigate many of the concerns with TS features compatibility, but maybe it could be one piece of a starting point for published TypeScript source files.

[JakobJingleheimer]

Has anyone considered usage of fields like engines or devEngines

I was just thinking this 🙂

[marco-ippolito]

We had consensus in this meeting #19, that for now there is no good reason to implement support in node_modules, since most of the monorepo that have been tested should be working fine.
Once the feature has stabilized we might want to re-evaluate.

[electrovir]

Just my small 2¢:

this feature would cause versioning headaches around TypeScript feature usage

There is already a similar problem with Node version mismatch that package installers must keep track of. Most of the popular npm packages support an old enough Node version that it's not a problem, but taking advantage of any new features will still break packages. (Personally I run into this frequently as I'm usually eager to start using experimental Node features.)