Cudnn dropout

[szha]

Description

Use dropout in CuDNN

Tested on p3.2x (V100). Test case:

import mxnet as mx
a = mx.nd.ones((10, 200, 300, 500), ctx=mx.gpu(0))
a.attach_grad()
mx.autograd.set_recording(True)
%timeit mx.nd.Dropout(a, 0.5, mode='always').wait_to_read()

	fwd	fwd+bwd
cudnn GPU	46ms 4.3ms	48ms 15ms
prev GPU	305ms	321ms

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For new C++ functions in header files, their functionalities and arguments are documented.
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• CuDNN Dropout.
• skip resource request for parallel random when using cudnn forward.
• extend cpptest to support stateful operators, resourcerequestex, ResourceRequest::kCuDNNDropoutDesc.
• fix bug where the backward pass on an inference-mode dropout forward didn't act as identity.
• add kCuDNNDropoutDesc resource for sharing dropout state across operators.
• replace dropout with identity when it's not dropping anything.

Comments

• cudnnSetDropoutDescriptor is an expensive call due to initialization on each of the stream multiprocessor on a GPU. Since cudnn 7, cudnnRestoreDropoutDescriptor becomes available so that the initialized space can be cached. This descriptor is currently used in both Dropout op and RNN op. We need a mechanism for caching this so that initialization on each stream happens only once, as the same desc can be shared among operators.

[stu1130]

@mxnet-label-bot add [pr-work-in-progress]
Thanks for your contribution @szha

[eric-haibin-lin]

shall we also add a cudnn_off flag to this op?

[eric-haibin-lin]

@TaoLv @pengzhao-intel @ptrendx @DickJC123 could you guys help review?

[szha]

Thanks for the review, @eric-haibin-lin @TaoLv.

@pengzhao-intel @ptrendx @DickJC123 I'm holding onto updating the PR until you get a chance to review this PR.

[pengzhao-intel]

@szha sorry I am on the vocation and don't have enough time to look into the details.

@TaoLv took the review so please go ahead to move forward for this PR.

Happy Chinese New Year @szha @eric-haibin-lin @TaoLv :)

[TaoLv]

Thank you. My comments are addressed.

[roywei]

I m not able to get the speed in the test case, see #13825 (comment)

[szha]

@roywei by default cudnn_off is turned on. You need to turn it off to benefit from cudnn dropout.