Code

Move ESM loaders off-thread#44710

Code

Merged

aduh95 merged 126 commits intonodejs:mainnodejs/node:mainfrom JakobJingleheimer:feat/esm_off-thread-loadersJakobJingleheimer/node:feat/esm_off-thread-loadersCopy head branch name to clipboardApr 13, 2023

+997-457

Commits

Commits on Apr 12, 2023

• esm: move hook execution to separate thread

  Show description for 03f6bb4
  

  4 peoplecommitted
  03f6bb4

  Copy full SHA for 03f6bb4

• esm: resolve optionally returns import assertions

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  a0bb205

  Copy full SHA for a0bb205

• port updates to worker_thread

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  cd16ccd

  Copy full SHA for cd16ccd

• move HooksProxy to hooks.js

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  01f1191

  Copy full SHA for 01f1191

• ESMLoader::addCustomLoaders → ESMLoader::registerCustomLoaders

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  5f02297

  Copy full SHA for 5f02297

• fix startup issues in hooks worker

  Show description for 820aa55
  

  JakobJingleheimer

  authored and

  aduh95

  committed
  820aa55

  Copy full SHA for 820aa55

• WIP: move importing custom loaders back to main thread

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  231b964

  Copy full SHA for 231b964

• 🎉 /test-esm-example-loader.mjs is passing!!

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  ed412d5

  Copy full SHA for ed412d5

• Fix tests that are loading modules/esm/worker.js on the main thread

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  4776a6d

  Copy full SHA for 4776a6d

• WIP: convert resolve to sync

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  2aed49a

  Copy full SHA for 2aed49a

• Fix test

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  78a2889

  Copy full SHA for 78a2889

• Restore parentURL validation to defaultResolve (and therefore also to import.meta.resolve)

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  5746d17

  Copy full SHA for 5746d17

• Improve deserialization error (sometimes)

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  c1afb3d

  Copy full SHA for c1afb3d

• Rename #hooks to #chains

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  666ee10

  Copy full SHA for 666ee10

• Subclass customized module loader

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  8198a13

  Copy full SHA for 8198a13

• Fix import.meta being undefined (WIP)

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  2d39e13

  Copy full SHA for 2d39e13

• Remove undocumented globalPreload setImportMetaCallback (WIP)

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  dd6a5d3

  Copy full SHA for dd6a5d3

• fs.write(1, message) works at least in Mac and Windows for printing from inside a worker

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  1415d6e

  Copy full SHA for 1415d6e

• Standardize names, fix lint

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  a374bd1

  Copy full SHA for a374bd1

• Exceptions thrown inside loader hooks should be returned to the main thread and rethrown there so that they're printed in stdout

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  98b5512

  Copy full SHA for 98b5512

• Rewrite test-esm-loader-hooks test to spawn child process and inspect stdout

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  7be92f3

  Copy full SHA for 7be92f3

• Send loader initialization errors up to the main thread so they can be thrown and printed

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  382753a

  Copy full SHA for 382753a

• Remove debugging code (fixes 3 tests!)

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  2b3e6a5

  Copy full SHA for 2b3e6a5

• Catch another worker thread exception so that the process doesn't hang

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  c73001f

  Copy full SHA for c73001f

• Add error handling around hooks method responses being undefined or hooks method responses being too large when serialized

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  b4cccd4

  Copy full SHA for b4cccd4

• separate worker data & lock buffers

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  b272d31

  Copy full SHA for b272d31

• chunks working?? (code is very rough!)

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  0ec518a

  Copy full SHA for 0ec518a

• Fix import.meta.resolve

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  11c1471

  Copy full SHA for 11c1471

• More carefully pass errors back to the main thread

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  0dd5114

  Copy full SHA for 0dd5114

• Some errors span multiple chunks, apparently

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  d15b425

  Copy full SHA for d15b425

• Fix chunking by separating metadata into separate buffers; fix error handling

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  b796c02

  Copy full SHA for b796c02

• unlock worker thread

  Show description for 599a616
  

  targos

  authored and

  aduh95

  committed
  599a616

  Copy full SHA for 599a616

• use syncCommPort to exchange data too

  

  targos

  authored and

  aduh95

  committed
  42d485e

  Copy full SHA for 42d485e

• fix --import

  

  targos

  authored and

  aduh95

  committed
  8e3e0c6

  Copy full SHA for 8e3e0c6

• fix doclint

  

  targos

  authored and

  aduh95

  committed
  76e23d8

  Copy full SHA for 76e23d8

• remove internal check

  

  targos

  authored and

  aduh95

  committed
  75fe62b

  Copy full SHA for 75fe62b

• refactor worker exception handling

  Show description for 350351f
  

  targos

  authored and

  aduh95

  committed
  350351f

  Copy full SHA for 350351f

• errors during load should just be rethrown

  

  targos

  authored and

  aduh95

  committed
  f3b2ed9

  Copy full SHA for f3b2ed9

• run global preload scripts in the main thread

  

  targos

  authored and

  aduh95

  committed
  7e79b2f

  Copy full SHA for 7e79b2f

• nits

  aduh95

  committed
  c0142d6

  Copy full SHA for c0142d6

• nits

  aduh95

  committed
  5d7e3b4

  Copy full SHA for 5d7e3b4

• fix linter errors

  aduh95

  committed
  685590c

  Copy full SHA for 685590c

• Add HookProxy.prototype.makeAsyncRequest

  aduh95

  committed
  49a4088

  Copy full SHA for 49a4088

• Wrap messages passed from worker to main thread so that we can carry along and restore the extra properties of Error objects

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  d40834d

  Copy full SHA for d40834d

• Emit experimental warning only once

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  47bd0bf

  Copy full SHA for 47bd0bf

• Don't re-throw uncaught exceptions and then re-catch them ad infinitum

  

  GeoffreyBooth

  authored and

  aduh95

  committed
  17ff524

  Copy full SHA for 17ff524

• fix linter errors

  aduh95

  committed
  2065222

  Copy full SHA for 2065222

• transfer ArrayBuffers

  aduh95

  committed
  b0cc21f

  Copy full SHA for b0cc21f

• improve exception serialization

  aduh95

  committed
  29d019f

  Copy full SHA for 29d019f

• chaining loader hooks

  aduh95

  committed
  8706cbd

  Copy full SHA for 8706cbd

• make things more consistent

  aduh95

  committed
  04ebc5c

  Copy full SHA for 04ebc5c

• remove unnecessary parenthesis

  aduh95

  committed
  8aae17e

  Copy full SHA for 8aae17e

• simplify error cloning

  aduh95

  committed
  b74dcaa

  Copy full SHA for b74dcaa

• small optimisation: read error properties only once when cloning

  aduh95

  committed
  3312b1c

  Copy full SHA for 3312b1c

• skip already cloned properties

  aduh95

  committed
  7e8be7c

  Copy full SHA for 7e8be7c

• DRY

  aduh95

  committed
  7b7e5be

  Copy full SHA for 7b7e5be

• fix test-esm-loader-chaining

  aduh95

  committed
  3f4c9bf

  Copy full SHA for 3f4c9bf

• update docs

  aduh95

  committed
  140c03b

  Copy full SHA for 140c03b

• fix waitAsync usage

  aduh95

  committed
  020b9d7

  Copy full SHA for 020b9d7

• simplify ArrayBuffer transfer

  aduh95

  committed
  6899203

  Copy full SHA for 6899203

• do not die waiting on worker to respond,

  aduh95

  committed
  9d392ef

  Copy full SHA for 9d392ef

• import() should not block the main thread, even for the resolve step

  aduh95

  committed
  1c7dd71

  Copy full SHA for 1c7dd71

• rework docs

  aduh95

  committed
  acfa8d9

  Copy full SHA for acfa8d9

• do not keep the worker alive when awaiting hooks to resolve

  aduh95

  committed
  1821301

  Copy full SHA for 1821301

• keep the worker thread alive longer

  aduh95

  committed
  b1dcccc

  Copy full SHA for b1dcccc

• let uncatcught exceptinnon be uncaught exceptions

  aduh95

  committed
  0cfc683

  Copy full SHA for 0cfc683

• use serializeError instead of custom logic

  aduh95

  committed
  7fd6fa1

  Copy full SHA for 7fd6fa1

• fix unserializable errors handling

  aduh95

  committed
  a3982ee

  Copy full SHA for a3982ee

• only count async messages

  aduh95

  committed
  47b4d83

  Copy full SHA for 47b4d83

• remove global error handler only at the end of the sync messages

  aduh95

  committed
  d518e53

  Copy full SHA for d518e53

• use 'beforeExit' event to detect when threads are ready to die

  aduh95

  committed
  531c9a3

  Copy full SHA for 531c9a3

• ensure there are no pending messages before letting worker die

  aduh95

  committed
  3bcf18c

  Copy full SHA for 3bcf18c

• linter fixes

  aduh95

  committed
  b071a89

  Copy full SHA for b071a89

• get rid of the tick counter timeout

  aduh95

  committed
  0911d15

  Copy full SHA for 0911d15

• Use Atomics.load for better results

  aduh95

  committed
  0b1c22b

  Copy full SHA for 0b1c22b

• add shared_constants internal module to make code clearer

  aduh95

  committed
  88686f5

  Copy full SHA for 88686f5

• rename ModuleLoader -> createModuleLoader

  aduh95

  committed
  1b1583e

  Copy full SHA for 1b1583e

• lazy load esmLoader

  aduh95

  committed
  3f4934a

  Copy full SHA for 3f4934a

• nits

  aduh95

  committed
  800d151

  Copy full SHA for 800d151

• fix test-esm-loader

  aduh95

  committed
  a5a798b

  Copy full SHA for a5a798b

• unref worker more often

  aduh95

  committed
  f9371ae

  Copy full SHA for f9371ae

• add more comments

  aduh95

  committed
  0b79809

  Copy full SHA for 0b79809

• fix main thread refusing to die waiting on a response that will never come

  aduh95

  committed
  0cfec42

  Copy full SHA for 0cfec42

• remove unrelated/unnecessary changes in test/

  aduh95

  committed
  281e943

  Copy full SHA for 281e943

• remove unrelated change in src/

  aduh95

  committed
  3e31a1c

  Copy full SHA for 3e31a1c

• restore changes related to sync resolve

  aduh95

  committed
  595a95f

  Copy full SHA for 595a95f

• Revert "unref worker more often"

  Show description for 10bf7b8
  aduh95

  committed
  10bf7b8

  Copy full SHA for 10bf7b8

• nits

  aduh95

  committed
  106b400

  Copy full SHA for 106b400

• use console.log instead fs.writeSync

  aduh95

  committed
  cc9373a

  Copy full SHA for cc9373a

• fix lint

  aduh95

  committed
  a67af11

  Copy full SHA for a67af11

• add comment to describe the shared memory organisation

  aduh95

  committed
  aab57d7

  Copy full SHA for aab57d7

• Remove unnecessary Atomics.store calls

  aduh95

  committed
  638b497

  Copy full SHA for 638b497

• s/BORED/IDLE/

  aduh95

  committed
  e340d24

  Copy full SHA for e340d24

• add SHARED_MEMORY_BYTE_LENGTH to shared_constants

  aduh95

  committed
  018ab18

  Copy full SHA for 018ab18

• add definition for idleness

  aduh95

  committed
  80cd74c

  Copy full SHA for 80cd74c

• Update doc/api/esm.md

  Show description for 817d0e8
  

  aduh95

  and

  targos

  committed
  817d0e8

  Copy full SHA for 817d0e8

• Update lib/internal/modules/esm/shared_constants.js

  Show description for 0590df4
  

  aduh95

  and

  JakobJingleheimer

  committed
  0590df4

  Copy full SHA for 0590df4

• add/fix comments + nits

  aduh95

  committed
  cee03fa

  Copy full SHA for cee03fa

• for..of -> classic for

  aduh95

  committed
  258cf8a

  Copy full SHA for 258cf8a

• s/NUMBER_OF_INCOMING_MESSAGES/NUMBER_OF_MESSAGES_IN_TRANSIT/

  Show description for 8863cb9
  

  aduh95

  and

  JakobJingleheimer

  committed
  8863cb9

  Copy full SHA for 8863cb9

• add explanation for how process.cwd() can fail

  Show description for 13f3d8d
  

  JakobJingleheimer

  and

  aduh95

  committed
  13f3d8d

  Copy full SHA for 13f3d8d

• Apply suggestions from code review

  Show description for 4731362
  

  aduh95

  and

  JakobJingleheimer

  committed
  4731362

  Copy full SHA for 4731362

• fix worker permissions for internal workers

  aduh95

  committed
  12717f7

  Copy full SHA for 12717f7

• fix dead lock

  aduh95

  committed
  61ad2a5

  Copy full SHA for 61ad2a5

• Apply suggestions from code review

  Show description for fa29a70
  

  aduh95

  and

  ljharb

  committed
  fa29a70

  Copy full SHA for fa29a70

• Apply suggestions from code review

  aduh95

  committed
  159ca6f

  Copy full SHA for 159ca6f

• git mv lib/internal/modules/esm/worker.js lib/internal/main/loader_thread.js

  aduh95

  committed
  b4a8a2f

  Copy full SHA for b4a8a2f

• add comment

  aduh95

  committed
  6177afa

  Copy full SHA for 6177afa

• revert git mv lib/internal/modules/esm/worker.js lib/internal/main/loader_thread.js

  aduh95

  committed
  a85288d

  Copy full SHA for a85288d

• never-settling import.meta.resolve

  aduh95

  committed
  be8090e

  Copy full SHA for be8090e

• reuse the public port instead of creating an additional communication channel

  aduh95

  committed
  f6679bc

  Copy full SHA for f6679bc

• fix: disable test's case concurrency to avoid deadlock ~1/10 times

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  722fd0e

  Copy full SHA for 722fd0e

• fix rare deadlock

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  544bf89

  Copy full SHA for 544bf89

• update Hooks entry in to note use of threads and caveat for async ops

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  88f9042

  Copy full SHA for 88f9042

• chore: de-lint md

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  6d8bfc6

  Copy full SHA for 6d8bfc6

• switch test fixtures from console.log → writeSync(1, …)

  

  JakobJingleheimer

  authored and

  aduh95

  committed
  31e7d5d

  Copy full SHA for 31e7d5d

• fix loaders with node:fs

  

  targos

  authored and

  aduh95

  committed
  372065e

  Copy full SHA for 372065e

• use more writeSync

  

  targos

  authored and

  aduh95

  committed
  244f916

  Copy full SHA for 244f916

• remove remaining await in front of import.meta.resolve

  aduh95

  committed
  bc8e970

  Copy full SHA for bc8e970

• resolve TODOs for edge cases

  aduh95

  committed
  f44a246

  Copy full SHA for f44a246

• fix race condition that could cause a dead lock when dealing with never settling hooks

  aduh95

  committed
  221f2e2

  Copy full SHA for 221f2e2

• reduce the size of the sab and introduce a never-setlle message status

  aduh95

  committed
  9436c97

  Copy full SHA for 9436c97

• fix race condition?

  aduh95

  committed
  ad73db6

  Copy full SHA for ad73db6

• tidy code, comments, and docs

  JakobJingleheimer

  committed
  95052c6

  Copy full SHA for 95052c6

• Update lib/internal/worker.js

  aduh95

  authored
  acc0900

  Copy full SHA for acc0900

• Update lib/internal/worker.js

  aduh95

  authored
  f02012f

  Copy full SHA for f02012f