unist-changeset

[tumidi]

I searched the different discussions and issues extensively, but couldn't find anything similar.

Context

We are working on innodoc which is a CMS for educational content. Currently we are migrating to a data structure based on mdast. Later we want to add collaborative real-time online editing.

Idea

Collaborative real-time editing is implemented in Etherpad using Changesets. Here those changesets work on source text directly. We imagine something similar for syntax trees.

graph BT
    A["[empty]"] -->|Changeset 1| B
    B[Rev. A] --> |Changeset 2| C
    C[Rev. B] --> |Changeset ...| D
    D["HEAD"]

Loading

A document (HEAD) can be described using a series of transformations (changesets). Those changesets can be stored in a database and transferred over the network to sync' document state across editors. At the same time they are the version history for a document.

Proposal

Create a library (eg. unist-changeset). It should provide a spec and functions to operate on changesets.

• specify a data structure for changesets
• apply changeset to an AST
• diff two ASTs and emit changeset (similar to eugeneware/changeset)
• merge changesets
• (...)

See also Etherpad and EasySync Technical Manual.

[ChristianMurphy]

Welcome @tumidi!

Thanks for striking up a conversation on realtime collaboration and editor experiences!

Taking a few steps back before diving in.

Questions

1. Are you tied specifically to Etherpad? Or could any realtime collaborative editor work?

2. Could you expand a bit more on why the collaborative editing would work better at the AST level for you and your project?
   I ask because in the space of collaborative editing, there is a lot of well established prior art, which we may be able to leverage instead of creating something new.
   A few levels to this:

   1. Could you expand on why you want to do which wouldn't be possible with plain text markdown? (text is the most established space in collaborative editing, and markdown is also plain text)
   2. If the answer to i is yes, could you expand on what a unist specific structure would provide over a more generic JSON based solution? (Have you run into a specific problem that a more generalized solution could not handle?)
   3. If the answer to ii is yes, would unist have enough context to solve the issue you are seeing? Would you need something mdast or hast based?

Prior art

There are two common technical solutions to collaborative editing:

• Operational Transform: see https://en.wikipedia.org/wiki/Operational_transformation and https://github.com/topics/operational-transformation
  This is centralized and uses more computation, but saves storage space.
• Conflict-free Replicated Data Types (CRDT): See https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type and https://github.com/topics/crdt
  This is distributed and lighter on compute, but uses more storage to track history.

Either of these can be explored depending on needs.

[tumidi]

For supporting plain-text and WYSIWYG, that means the client already does need to have a parser running in browser, correct?

Well, yes and no. I simplified my response because I wanted it to be brief.

Yes, because the editor part of the application needs all the parsing and AST processing. But then, no, because 99.9% of the users are viewers and wouldn't receive the big bundle that has all the AST processing included.

It would mean to do parsing on the client

No, you can still do full server side rendering if you want.

Well, also here I left out the full story for brevity. To be more specific this should probably read parsing on-the-fly while rendering or something.

In fact, we're doing server-side rendering for the initial page load. All subsequent navigation happens client-side. The application is completely API driven. At first we stored Markdown in the database and did the parsing in the API request handler, sending JSON over the wire. Later we switched to storing mdast JSON in the database which saves us the parsing/processing on every API request.

We can switch back to storing Markdown text anytime, but still struggle on making a reasonable decision on that matter.

I mean solutions like:

• https://github.com/streamich/json-joy
• https://automerge.org/docs/types/values/
• https://docs.yjs.dev/api/subdocuments
• among others

Which support collaborative editing on JSON documents (through CRDT or OT). unist is a JSON document.

This is extremely helpful and might be exactly what we need.

As a default for most users looking to do collaborative editing in markdown. I'd recommend using markdown plain text, for both the medium that CRDT/OT work with, and as the storage format server side. It's flexible, it allows for client side or server side rendering, it is highly compressible for data efficiency, and parsing is generally fast enough for most real time preview scenarios.

That's not to say that you cannot create a new unist based solution, you absolutely can. But, there are also some really excellent existing solutions for collaborative editing out there which can already be integrated with unist/mdast/remark/rehype.

Why re-invent the wheel if you don't need to?

Since the beginning of the rewrite of the application we're thinking about how to store the documents (plain-text or JSON). And, no, we don't want to re-invent the wheel. We try to keep things as simple as possible and using a solid existing solution would be the best.

What would be the go-to options when using plain-text? I know about ot.js but it is unmaintained.

[ChristianMurphy]

In fact, we're doing server-side rendering for the initial page load. All subsequent navigation happens client-side. The application is completely API driven. At first we stored Markdown in the database and did the parsing in the API request handler, sending JSON over the wire. Later we switched to storing mdast JSON in the database which saves us the parsing/processing on every API request.

APIs don't have to return JSON, they can return plain HTML. 🙂
If you have mixed content in the return, JSON with HTML in a property is also valid.

What would be the go-to options when using plain-text?

Depends on what language(s) you are comfortable with, the GitHub topic on OT: https://github.com/topics/operational-transformation has quite a few which are actively maintained.

[tumidi]

APIs don't have to return JSON, they can return plain HTML. slightly_smiling_face If you have mixed content in the return, JSON with HTML in a property is also valid.

I agree that HTML can be returned by an API. But I'm afraid I fail to see the point in stating this.

What would be the go-to options when using plain-text?

Depends on what language(s) you are comfortable with, the GitHub topic on OT: https://github.com/topics/operational-transformation has quite a few which are actively maintained.

Yes, thank you. I browsed the list but didn't find any well-known project which is maintained in TypeScript/JavaScript. Only [ot.js](https://github.com/Operational-Transformation/ot.js], as I said, which is unmaintained for three years.

[ChristianMurphy]

I agree that HTML can be returned by an API. But I'm afraid I fail to see the point in stating this.

That you can return fully rendered html snippets, you don't need to send mdast or hast across the wire.

Yes, thank you. I browsed the list but didn't find any well-known project which is maintained in TypeScript/JavaScript.

Yes, Apache Wave was probably the best known open source OT implementation, many of the solutions today are in Java, Go, or Rust. And are inspired by Wave.
There are some actively maintained TypeScript implementations, like:

• https://github.com/0xTheProDev/Operational-Transformation
• https://github.com/BrotherJing/scalable-ot

[tumidi]

Thank you for taking the time and all the suggestions. 🙏