feat: trace resource requests to allow fetching dynamic resources from pyodide process without a page refresh

[hydrosquall]

Motivation

• Improve on the design introduced in feature: proof of concept for supporting plugins that load custom JS + CSS + server-side responses (JSON) #4, thanks to the observation by @gvergnaud that the browser respondWith API supports Promises, and I can use a lightweight timeout function to bail on waiting if the web-worker can't respond in time.
• Solves the 3rd party assets part of the plugins problem outlined in Get JavaScript working (table.js, plugins and more) simonw/datasette-lite#8

Terminology

• pyodide is project that lets us run Python in web browsers via WebAssembly (WASM).
• An asset is a file generated by the Datasette server running in Pyodide.
  • This could be CSS from a static folder, JS from Datasette, rendered text content, or JS from a plugin
• serviceworker is a JS script that intercepts requests. It can't reach the webworker directly.
• webworker is a JS script that hosts pyodide. It isn't strictly required, but we do it because the file is big + potentially slow. Code in the webworker doesn't interrupt the main rendering thread. It can't reach the serviceworker directly.
• "HTML page" - this refers to the page that contains the DOM that the user sees. It also acts as a bridge between the webworker and serviceworkers.

Changes

In #4, assets were only available on the second fetch. I wasn't sure how to block the service-worker process while it was waiting for the webworker to respond. This PR removes that hurdle by using JS Promises . Instead of checking for the datasette response in a cache, the service-worker uses a requestId as a form of "bookkeeping" for the server to include in its response, ensuring that response can be matched to the request with a message-passing rather than callback-based dataflow.

• When the host HTML page requests an asset server-worker.js intercepts the request, and decides whether to check for it in the local cache, pass it through to the public internet via fetch, or ask the web-worker for it.
• If it needs help from the webworker, we add an entry to the registry with a unique requestId to handle an async message from the webworker, and wait for up to TIMEOUT seconds. A message is sent to the parent HTML page that can be forwarded to the webworker, including a requestId and a path.

Meanwhile, in the webworker, we await messages from the other process. The Pyodide process uses the path to generate an appropriate response (text, json, etc), and returns it (including a proper MIME Type) along with the requestId.

Finally, back in the serviceworker: when we receive a message with a requestId, we return the result to the host HTML page. To cleanup / free up memory, we delete the "pending" request from the response registry.

Testing

• This can also be tested with cloudflare: https://d2ae7b89.datasette-lite.pages.dev/

• http://localhost:3000/?install=datasette-nteract-data-explorer#/content/pypi_packages?_nocol=description

• http://localhost:3000/?install=datasette-nteract-data-explorer#/content/pypi_packages?_nocol=keywords&_nocol=description

• Here's a sample graph: https://a.cl.ly/Z4uDgerK

  • Reproduce with this: https://d2ae7b89.datasette-lite.pages.dev/?install=datasette-nteract-data-explorer#/content/datasette_repos?_filter_column_1=closedIssueCount&_filter_op_1=lt&_filter_value_1=100&_filter_column=&_filter_op=exact&_filter_value=&_sort=id

Notes

• Faceting in the gear doesn't work, hiding column in URL doesn't work either, due to assumptions that the table.js code made about the structure of the parent page's URL. However, if you construct the URLs manually (like in my testing section), it will work.
• Pyodide assets are cached, so this should work offline after the first load Moving this part to a separate PR

Illustrated Explainer

Here's a cartoon of the problem that this PR solves.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Pages

Latest commit:	b859411
Status:	✅  Deploy successful!
Preview URL:	https://d2ae7b89.datasette-lite.pages.dev
Branch Preview URL:	https://cameron-yick-feature-trace-r.datasette-lite.pages.dev

View logs