Switch to async/await for bin/server and tests

[jtgeibel]

This PR series converts the future combinations to async/await syntax in the server binary (when USE_HYPER=1 is set) and the recording HTTP proxy used in tests.

This change requires the update of several dependencies at once (see the list below). It is not strictly necessary to update reqwest at this time, but I'm doing so in this series to avoid the build time regression of pulling in hyper 0.12, hyper 0.13 and their associated transitive dependencies.

Blockers:

• async/await is stable in 1.39.0 🎉
• Bump the rust release in RustConfig
• Stable release of underlying stack
  • futures 0.3.1
  • tokio 0.2.9
  • hyper 0.13.1 and hyper-tls 0.4.1
  • reqwest 0.10.1
  • conduit-hyper 0.2.0

[rust-highfive]

r? @sgrif

(rust_highfive has picked a reviewer for you, use r? to override)

[bors]

☔ The latest upstream changes (presumably #1876) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #1899) made this pull request unmergeable. Please resolve the merge conflicts.

[jtgeibel]

🎉 I've rebased this branch and now consider it ready for review. 🎉

There is currently a missing feature in conduit-hyper which should be resolved before we try rolling out USE_HYPER=1 to production again. That can be handled in a future PR and I think we can land this now for the improvements to the test harness.

r? @carols10cents

[bors]

☔ The latest upstream changes (presumably #2072) made this pull request unmergeable. Please resolve the merge conflicts.

[jtgeibel]

Rebased to resolve merge conflicts, and added a commit to switch from git master to the now released reqwest v0.10.0.

[bors]

☔ The latest upstream changes (presumably #2066) made this pull request unmergeable. Please resolve the merge conflicts.

[jtgeibel]

I'd like to merge this PR before bumping any more dependencies in Cargo.toml, to avoid messy rebasing conflicts.

I've just pushed a new commit that resolves the remaining known issue. The server will now limit the max number of background threads. If the max thread count is exceeded by a new request, that request will be rejected with a 503 Service Unavailable response. This ensures that we don't end up with an effectively unbounded queue of threads waiting to obtain a database connection. Of course, hyper is still only used when USE_HYPER=1 is set.

[jtgeibel]

The build failure is a regression on nightly and is already reported as rust-lang/rust#68264.

[carols10cents]

Still looking and need to test but wanted to save the one comment I have so far :)

[carols10cents]

I'm having trouble signing in locally when I'm on this branch rebased on master at 7072305 (and I verified that when I'm on 7072305 locally I can sign in), even without USE_HYPER=1.

It appears to be failing when requesting the user info from GitHub; the request is returning 403. I'm able to print out the Authorization token value that I'm trying to use locally (println!("auth secret = {}", auth.secret()); as in here), and use it in a curl request to GitHub that returns successfully:

curl -H "Accept: application/vnd.github.v3+json" -H "Authorization: token [redacted]" https://api.github.com/user

Soooo 🤷‍♀ ???

[jtgeibel]

@carols10cents excellent catch! So it turns out reqwest no longer sets a default user-agent header. I've rebased and then added a commit which sets the user-agent to our URL and updates the tests to always check for this value.

I'm pushing the changes to staging now, and will also set USE_HYPER=1 there.

[bors]

☔ The latest upstream changes (presumably #2186) made this pull request unmergeable. Please resolve the merge conflicts.

[carols10cents]

2 issues we investigated tonight:

• The default number of threads used in development mode needs to be increased to make the site usable locally (I tried 4 and it seemed to work ok)
• Using thread::sleep in a controller function appeared to be blocking other requests, and this wasn't the expected behavior when using tokio::task::spawn_blocking

[carols10cents]

If those two issues get tracked somewhere and block turning on hyper for everyone everywhere, feel free to r=me :)

[jtgeibel]

I've opened #2204 as a tracking issue for any remaining blockers for enabling USE_HYPER=1 in production.

@bors r=carols10cents

[bors]

📌 Commit f724f05 has been approved by carols10cents

[bors]

⌛ Testing commit f724f05 with merge 28216b3...

[bors]

☀️ Test successful - checks-travis
Approved by: carols10cents
Pushing 28216b3 to master...