Migrate from callbacks to async/await

[muxator]

This PR represents the line of work that refactors Etherpad to use async/await calls instead of callbacks, vastly simplifying the code, as per #3540.

This represents a vast change, and by its nature requires a lot of testing, hence its WIP status. It is currently deployed on https://beta.etherpad.org. It will be merged as soon as deemed sufficiently stable.

It has the priority over other work: present and future PRs, in case of conflicting changes, will need to rebase themselves over this work.

The development is by contributor @raybellis.
The branch will be periodically force-pushed if needed, please update often.

[muxator]

Hi, the PR is almost ready to be integrated. I just need a review from @ray.

While writing async, some boolean expressions where made stricter (e.g. == -> ===, != -> !==). This is generally a very good thing, but on a code base with lots of technical debt like ours, this can resurface latent bugs (this happened at least once).

1. For this reason, I have tried to extract & centralize as many such changes as possible in a single commit, namely d520e28. These changes were all present in Ray's original work.

2. While doing this, I noticed that in some files this transformation was not symmetrical (some places were made stricter, others were not). I've done another commit (8ecd0dd) which tentatively makes the situation even. None of these changes were part of Ray's work, yet I think we could integrate them.

I need a review on points 1 and 2. If Ray recognizes 1 as his beloved brainchild, and thinks that 2 may complete it, I'll fold the two changes and merge the PR.

[raybellis]

I think your author === null check may result in a similar to the bug that I caused inadvertently with the SessionManager.

If you like, I'm happy to audit all calls to db.get* and ensure that they work with either undefined or null values, given that we can't be sure which will be returned by ueberDB.

I was going to suggest that just testing for if (!value) would suffice, but on reflection that may cause issues if the result value can legally be 0 or "", so an explicit check for val === null || val === undefined may be preferred. The alternative is to explicitly require that loose checks are annotated to indicate that they are intentionally loose.

[muxator]

That's true: using author === null could introduce regressions.

However, in its current status on async (async-PR closely tracks it) AuthorManager.js has only strict checks (X !== null or X === null) except for the one in mapAuthorWithDBKey().

The main pain point is probably ueberDB, but I do not want to digress.

On commenting lax comparison: in async-PR, I agree, indeed I had already put such a comment in SessionManager.js:

/*
   * In some cases, the db layer could return "undefined" as well as "null".
   * Thus, it is not possible to perform strict null checks on group2sessions.
   * In a previous version of this code, a strict check broke session
   * management.
   *
   * See: https://github.com/ether/etherpad-lite/issues/3567#issuecomment-468613960
   */
if (!group2sessions || !group2sessions.sessionIDs) {

I think we have to decide what to do with AuthorManager, as well. All the original comparisons were lax, but if you have a look at d520e28 (which, I remember, just extracts changes made on async), all but one were made strict. Could this introduce latent bugs, as well?

If you like, I'm happy to audit all calls to db.get* and ensure that they work with either undefined or null values, given that we can't be sure which will be returned by ueberDB.

That would be welcome: in that case we could reach a more coherent situation.

If you do not have time, or if we want to postpone this, I can also make a ugly proposal in the name of pragmatism:

• put lax comparisons wherever the old code base was already lax. This would bring us nearer to bug-for-bug compatibility
• put comments explaining our reasoning
• merge the PR so we can start moving forward

If you think it makes sense, I can compare develop with the asynchronous version, and worsen its purity in the name of compatibility.

[raybellis]

I've got time to do that audit. How (and where) would you prefer to see any relevant commits made?

[raybellis]

For reference, here's a comparison of ueberDB2's behaviour with dirty vs mysql:

DirtyDB

get undefined (random) key
-> undefined
set explicit value and verify
-> pass
set value to undefined
-> undefined
set value to null
-> undefined

MySQL

get undefined (random) key
-> null
set explicit value and verify
-> pass
set value to undefined and get
-> undefined
set value to null and get
-> null

[raybellis]

I found an alternate solution - see 32cebc4

[muxator]

Nice. Pulled in with ac7663c.

[muxator]

It's done.

I think it's worth citing from the commit message:

Merge pull request #3559 from raybellis/async-PR

With this commit, that closes #3540, we pay the first big slice of our technical debt. In this line of work we streamlined the code base, reducing its size by 15-20% and making it more understandable at the same time.

The changes were audited and tested collaboratively and are deemed sufficiently stable for being merged.

Known issues:

• plugin compatibility is still not perfect
• the error handling path needs to be improved

This is an important day for Etherpad: thanks, Ray!