feat: Helia fetch for dApps

[SgtPooki]

this task is intended to cover all of the actions necessary to get a library, lets call it "helia fetch" working for dApps.

• Create library that accepts HTTP Request and returns HTTP Responses
• do unix-fs-exporter things (sub resource loading)
• bring in learnings from https://github.com/ipfs-shipyard/helia-service-worker-gateway/blob/main/src/lib/heliaFetch.ts & https://github.com/ipfs/helia-http-gateway

ping ed from liquity to see what he's working on

Ecosystem Needs

(lidel) One way to look at the body of work ahead is to identify needs of different stakeholder, and focus on delivering useful software for the wider audience first. Below if my first stab in surfacing needs and related scope of work, feel free to adjust as we go.

Food and Shelter for Web developers and public gateways

Use cases:

• web dev new to content-addressing: wants to "add IPFS to their website" and load image/video by CID, but does not want to spend hours learning about our stack. they are familiar with how web works, are comfortable with Fetch API and Service Worker
• website owner: being able to offload and verify fetch og content-addressed subresources without depending on a single gateway
• ipfs.io: we want to enable public good gateways to start throttling websites which hotlink images and videos, and return an error message which encourages leeching websites to switch to our helia-fetch library (mutual benefit, which removes their dependency on a single gateway).

Needs:

• Turn-key NPM package developers can include in their projects
• API modelled after Fetch API: HTTP Request → HTTP Response
  • Prior art: https://github.com/RangerMauve/js-ipfs-fetch
  • Good abstraction, can be reused in both Web Apps and HTTP gateway implementations
  • UX: suppport both ipfs:// and /ipfs/ paths
• Support codecs and hashes of defined in Web Pathing Spec Draft (dag-pb/raw/dag-cbor/dag-json, UnixFS HAMTs)
• Minimal subset of path gateways required for hosting of subresource (file) hosting.
  • No need for _redirects, dir listing, or CAR responses.
  • Range requests are required.
  • (optional) IPNS can be added later (focus on immutables first)
• Ability to configure source(s) of blocks
  • HTTP trustless gateways (on/off + urls of trustless gateways to use)
  • libp2p bitswap (on/off + urls of delegated /routing/v1 endpoints)
    • (future, out of scope for now) trustless gateway over libp2p: https://specs.ipfs.tech/http-gateways/libp2p-gateway/
• A code example which uses this library in ServiceWorker for handling requests to /ipfs/ paths on one's website.
  • This will already be very useful to many web devs ad Dapp developers

PWA and Dapps

Use cases:

• pwa owner: wants to publish their PWA to IPFS and it to work the same in all web contexts
• dapp maintainer: wants to NOT be responsible for hosting dapp instances, but also give end users a gradient of security options where they control how much trust is delegated to third-parties

Needs:

• Ability to bootstrap the top level document via Service Worker and have proper Origin isolation
  • This will be mostly done once we have path gateway, only additional feature is to support fetch equivalent of Host header for subdomain functionality. Checking Host header, and also window.location.origin should be good enough.
  • This enables service worker versions of subdomain/dnslink gateways, loading top level document via Service Worker instead of from backend server. This does not solve end-to-end verification as the server returning SW code still needs to be trusted, but the server is no longer responsible for transferring bytes, which is a step in the right direction.
• _redirects support for PWA and parity with Web Gateways / Brave
• Ability to configure how mutable pointer (DNSLink and IPNS) are resolved
  • IPNS can be resolved by asking trustless gateway's /ipns/name for application/vnd.ipfs.ipns-record (or delegated routing endpoint at /routing/v1/ipns)
  • DoH endpoint(s) for DNSLink resolution
    • DNSLink of ENS name requires special DoH endpoint that supports .eth (eg. curl -s -H "accept: application/dns-json" "https://resolver.cloudflare-eth.com/dns-query?name=_dnslink.vitalik.eth&type=TXT" | jq)
      • bare minimum is to allow override/fallback by providing alternative DoH URLs, perhaps eth.limo could host one (they already run resolver).
      • (future) at some point web3 dapp developers may create trustless client for resolving ENS without intermediaries like Cloudflare
• Service Worker code example which bootstraps website's Origin root (/) from a CID

[SgtPooki]

• the exporter doesn't write to the blockstore.. we could do a blackhole blockstore with helia-fetch, because returning a response will allow the browser to cache the content, and then subsequent requests will use the browsers in-built cache.

[lidel]

I've added my understanding of "Ecosystem Needs" to the description of this issue.

Tried to split needs into groups that then could be fullfilled by delivering sensible milestones and so far I got two (food and shelter and PWA/DApps).

Hope it is useful for planing work once we are back in January.
Feel free to edit/adjust as we go. Also ok if I should move this to https://github.com/ipfs/in-web-browsers/issues (but felt more useful to have it all in a single place here)

[hannahhoward]

have you looked at https://github.com/filecoin-saturn/js-client and https://github.com/filecoin-saturn/browser-client? There is probably lots of code to crib there. We do essentially this -- intercepting requests with service worker, making trustless HTTP requests, using unixfs-exporter to verify the car and produce the flat file, and serving it to the user if it works.

BTW, beware that ServiceWorker doesn't run on first load (at least until it's installed and registered, usually after sub resource URLs are fetched).

[BigLep]

2024-01-09 conversation:

Concerning API modelled after Fetch API: HTTP Request → HTTP Response

Location:

• ipfs/helia/helia-fetch: will it be discoverable?

We did conversation in https://hackmd.io/gXm6nzJlTDmhFJ8ZL2CI_w

Next steps:
@SgtPooki is going to create the corresponding issue for this work (which has a design step up front)

[SgtPooki]

lidel mentioned in conversation: delegated routing request for https://delegated-ipfs.dev/routing/v1/providers/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi points to "/dns4/dag.w3s.link/tcp/443/https"

and we should see what it would take to prioritize fetching over https

[SgtPooki]

Created https://github.com/ipfs/web3-fetch to host some of the design decisions as proposed by @lidel

[2color]

@SgtPooki Did we decide if this is going to be a package inside this repo?

[SgtPooki]

@2color it was either or, but I figure starting out the with the design in a separate repo will make changes easier for now, instead of mucking with Helia versioning

[SgtPooki]

Moved to #392

[SgtPooki]

we may want to re-open this or open new issues with some of the items above.