Implement UDP as a swarm protocol

[whyrusleeping]

Having UDP as a protocol for swarm to use will make ipfs MUCH more flexible.

Some considerations will need to be taken for packet loss though, in some cases lost packets are not okay. Implementing a basic packet-wise ACK system might be useful.

[jbenet]

uTP or SCTP/UDP might be the right thing to do. We don't need streams, but maybe easier than segregating our packets and writing our own reliable transport for things that really matter.—
Sent from Mailbox

On Fri, Sep 12, 2014 at 2:23 PM, Jeromy Johnson [email protected]
wrote:

Having UDP as a protocol for swarm to use will make ipfs MUCH more flexible.

Some considerations will need to be taken for packet loss though, in some cases lost packets are not okay. Implementing a basic packet-wise ACK system might be useful.

Reply to this email directly or view it on GitHub:
#58

[jbenet]

I found https://github.com/h2so5/utp/ It has an alpha warning, and is currently slow (see https://github.com/h2so5/utp/pull/3) but a great start.

[jbenet]

Also, two years ago, @cyberroadie implemented SCTP at https://bitbucket.org/cyberroadie/go-sctp -- it should be a library, as it probably won't go into golang's stdlib net. We can see about extracting it.

[whyrusleeping]

Cool,I got myself more familiar with UDP in Go last night, so maybe I could take a look at whats making the utp package slow, and do some pull requests

[cyberroadie]

Hi, my SCTP addition to the Go net lib is a bit dated, don't think it will work with newer versions of Go. One thing to keep in mind is that it is dependent on the implementation of SCTP of the OS it is running on. My version only works on FreeBSD and OS X.

[jbenet]

@cyberroadie ah, i see. So not actually an implementation in Go, but a binding to the kernel implementation? Have you found any of SCTP, uTP, UTD, DCCP, ... in Go?

[jbenet]

• http://udt.sourceforge.net/ (and https://github.com/oxtoacart/go-udt)

[cyberroadie]

Yes indeed a binding. TCP and UDP do exactly the same in Go. The binding is abstracted away with a thin layer of assembly which you can use to call the different system calls: https://code.google.com/p/go/source/browse/#hg%2Fsrc%2Fsyscall

[cyberroadie]

Didn't know about UDT, interesting. I'm currently implementing a VoiP solution integrated in a game engine and I'm looking for a faster and more reliable solution then UDP/TCP. I definitely will have a look at this.

[jbenet]

@cyberroadie is this in Go?

[cyberroadie]

The server side is. The client side will be in C# and C++.

[jbenet]

@cyberroadie if you go for UDT, we should all collaborate on a Go implementation.

[cyberroadie]

@jbenet yes, that would be a cool project.

[jbenet]

"Oh, I'll just make quick bindings for this C++ UDT lib. how hard can it be?"

12 hours later... Okay, I just finished making Go bindings for C++ UDT: https://github.com/jbenet/go-udtwrapper (can test it with go test, cd udtcat; make test -- warning: annoying installation due to the (currently) dylib).

Would love to see real benchmarks because (a) not clear how much faster the protocol is, (b) not clear how fast this c++ impl is, (c) not clear i didnt screw it all up and make it much slower than its supposed to be. (small benchmarks would be a plus).

Anyway, we can use this while the nascent utp implementation grows.

[whyrusleeping]

👍

[jbenet]

Update: #488 revealed that h2so5/utp (old "working" commit) was breaking as a transport. #490 removed it. We will have to get another udp reliable transport for nat:

• utp - https://github.com/h2so5/utp is not yet ready
• udt - https://github.com/jbenet/go-udtwrapper does not have required assembly for arm.
• sctp - maybe i should wrap sctp. hopefully it doesn't have assembly. wont be quick though.

---

@oxtoacart are you guys using go-udtwrapper*? (something else for udp reliability in go?) @whyrusleeping had trouble running it in his ARM chromebook, we determined it was some missing ARM assembly in the c++ udt lib. I haven't looked into this-- will soon-- but just thought you should know.

[oxtoacart]

@jbenet Thanks for the heads up! We're not actually using go-udtwrapper yet. First we have to release a build that includes go-natty and collects some stats on how well nat-traversal is working. Actually proxying traffic over UDP will hopefully follow not too long thereafter.

[jbenet]

@oxtoacart sounds good. I'm about to work on bundling go-natty into go-ipfs (at some point soon we should discuss making natty strip out only the relevant c++ from webrtc and either building a tiny binary, or link it in using cgo. currently natty adds ~10MB (or is it 40+MB if you bundle multiple archs together?))

[oxtoacart]

@jbenet The natty version for each architecture is compiled into the go build for that architecture, so you're looking at just one natty executable as overhead.

[oxtoacart]

BTW, I know that @atavism wants to work on decreasing the size of the binary, but he has his plate full with other stuff at the moment.

[jbenet]

Ah okay, thanks! And no worries, this is lower priority stuff :)

[oxtoacart]

@jbenet Thank you! Hope you had a nice holiday season.

[jbenet]

thanks! same to you all! btw, feel free to hang out with us in #ipfs on freenode

[whyrusleeping]

we have UDT that works just needs to be wired in, I think this can be closed now

[jbenet]

Update on uTP. anacrolix wrote to the go-nuts mailing list:

I ended up writing https://github.com/anacrolix/utp for this. I found h2so5/utp to be unstable and hard to work with. anacrolix/utp is at least very stable, and has been used in production for 8 months now. Its performance could be better, and some things in the spec could be clarified, but if you want to use uTP, this will get the job done.
From https://groups.google.com/d/msg/golang-nuts/Co8vQ0KkrLU/qeTcEgHvR9oJ

perhaps we should try it out :) -- i would be more comfortable if we had a large cross-internet test case.

[xaionaro]

Well:

• go-utp-transport -- doesn't work (doesn't compile)
• go-udp-transport -- not implemented
• go-libp2p-quic-transport -- doesn't work with ED25519 (QUIC transport constructor fails with ed25519 keys libp2p/go-libp2p-quic-transport#60). It works in my sandbox, I've prepared a PR and waiting for a codereview. But still in upstream there's still no support of UDP (at all) for an ED25519-based node.

But the ticket is closed :). Can we reopen the ticket?

Reference: https://github.com/libp2p?utf8=%E2%9C%93&q=transport&type=&language=go

[Stebalien]

This issue has a lot of outdated information. I've created a new UDP meta issue: #6481.