UDT implementation

[Kleissner]

Keeping track of important links to UDP implementations.

• Native library, although status unclear (2016) https://github.com/oxtoacart/go-udt
• Fork with more recent commits (2020): https://github.com/odysseus654/go-udt
• Wrapper around C implementation fork (2015): https://github.com/jbenet/go-udtwrapper
• Other fork with more recent changes (2018): https://github.com/cloudedcat/go-udtwrapper
• Original UDT spec (2010): https://udt.sourceforge.io/

Specs via https://udt.sourceforge.io/doc.html:

• https://udt.sourceforge.io/doc/udt-ccc-sc05-v10.pdf
• https://udt.sourceforge.io/doc/udtv4-gridnets08.ppt

Protocol of choice: UDP

UDP seems to fit the use case (file transfer) squarely. QUIC might be an overkill and uTP might have undesired side effects (quote from the spec "This effectively makes uTP yield to any TCP traffic").

The trick will be to fork the implementation from odysseus654 and instead of using new UDP connections, using the existing Peernet protocol as transport layer.

Before starting with the actual implementation, a new file transfer message must be defined in Peernet with a small header. The header must include:

• Protocol: 0 = UDP
• Control: 0 = Request start, 1 = Terminate hash not found, 2 = Terminate protocol not supported, 3 = Terminate
• File hash and size
• Offset and Limit

[Kleissner]

Also relevant to look at - uTP. Documented here https://www.bittorrent.org/beps/bep_0029.html

• Wrapper https://github.com/anacrolix/go-libutp
• Native implementation (2020?) https://github.com/h2so5/utp
• Another native implementation (2018) https://github.com/anacrolix/utp
• Wrapper https://github.com/anacrolix/go-libutp

Long discussion about uTP and UDT libraries and their status: ipfs/kubo#58
Updated discussion here: ipfs/kubo#6481

[Kleissner]

A third option would be QUIC.

• Native implementation https://github.com/lucas-clemente/quic-go
• Another one https://github.com/goburrow/quic

QUIC seems to be an overkill for this use case. Simplicity is winning here.

[Kleissner]

Since embedding the actual payload data in regular Peernet packets is very expensive, a new concept of "lite packets" is established. They have a small header which bypass all regular checks. They are identified via session IDs.

This fixes the CPU bottleneck that appeared due to signing (and decrypting) the transfer packets. Before, TransferMaxEmbedSize was set to 1121 bytes.

const TransferMaxEmbedSize = internetSafeMTU - PacketLengthMin - transferPayloadHeaderSize

For lite packets it is 1214 bytes, an 8% increase.

[Kleissner]

Just implemented lite packets in f305e5b. Stats still need to be recorded upon receiving in Networks.packetWorkerLite.