changes to get ray serve responding on REST API calls when distribute…

[venkatkalluru]

…d tracing is enabled

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[venkatkalluru]

FastAPIInstrumentor instrumentation was not working and was throwing errors so I added to add this.
This is taken from https://github.com/anyscale/tracing-example/blob/main/fp.py

[venkatkalluru]

Also, we have to do this injection on all the serve deployments or else we get the same error described in the issue, where it complains with the following error.

TypeError: got an unexpected keyword argument '_ray_trace_ctx'

[venkatkalluru]

Added this example to show that on all the serve deployments we have to do tracing injection and see the comment below.

to deploy and test do the following..

ray start '--resources={"no_gpu": 100}' --head --port 7990 --tracing-startup-hook ray.util.tracing.setup_tempo_tracing:setup_tracing

and then

serve run test_tracing_with_deployment:translator_app

and send traffic as shown in the below example.

curl -X POST -H "Content-Type: application/json" -d '"Hello, how are you?"' http://127.0.0.1:8000/