[MKLDNN] Enable signed int8 support for convolution.

[ZhennanQin]

Description

Major changes:

• Integrate mkldnn signed int8 convolution.
• Introduce new quantization flow out_type auto, which automatically determines the out_type according to calibration information. If negative value is detected from calibration min, out_type will be int8, otherwise, out_type will be uint8.
• Introduce new operator quantize_v2, which only has 1 input data and will self-calculate min/max in runtime or use calibration information set by attrs. Then calibration informations are all recorded in model file instead of part in params file. User can easily modify them without touching params file. This operator also used to implement out_type auto.
• Add new parameter monitor_all for monitor module to allow monitoring both input & output of operators. This is used for collecting calibration information of in_data to support quantize first layer of graph.

@pengzhao-intel @TaoLv @zheng-da @reminisce @azai91

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

[ZhennanQin]

@marcoabreu Looks like CI has trouble to build this PR. Some builds failed to compile this line
https://github.com/apache/incubator-mxnet/blob/84131807d67bb6a256b78cd75bb12a274c22347b/include/mxnet/tensor_blob.h#L290
with comparison between signed and unsigned integer expressions. But this PR doesn't touch that part,and looking into the code, both sides are returning size_t, which should be all same. Another prove is, clang can build pass with -Wsign-compare, which indicated that the gcc build has some problem. Can you have a look? Thanks.

Update: I found the reason, the right side returns from mshadow::Shape::Size(void), which is index_t.
https://github.com/dmlc/mshadow/blob/696803bd7723ade8230af878460d96c68a550fbc/mshadow/tensor.h#L126
So comparing size_t with index_t causing build failed. Question: why our CI didn't report this previously, and even not from other PRs sent at same time? Does ccache hide this issue?

[pengzhao-intel]

@KellenSunderland could you help take a look for the CI issue?

[pengzhao-intel]

FYI @yoel-shapiro

[yoel-shapiro]

Hi, I'm still getting an issue when using "int8"

setup:
I rebuilt MKLDNN master
cd mxnet-incubator, git pull origin pull/13697/head
verified that the script works with "uint8"

I am able to run calibration but when I use the quantized output model I get the following error, specifically when trying to read the model's predictions:

Traceback (most recent call last):
File "inference_dsalite.py", line 201, in
prob = pred[k_head][k_im].asnumpy()
File "/home/local/ANT/yoelsh/work/mxcode/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 1987, in asnumpy
ctypes.c_size_t(data.size)))
File "/home/local/ANT/yoelsh/work/mxcode/incubator-mxnet/python/mxnet/base.py", line 252, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [14:12:29] src/operator/nn/mkldnn/mkldnn_base.cc:326: Unknown MKLDNN format for 4 dimensions: 47

[ZhennanQin]

@yoel-shapiro Thanks for trying MKLDNN int8 solution. Did you apply this PR? This PR removed line 346 check, and did lots of changes to support int8. Please try int8 with this PR.

[ZhennanQin]

@yoel-shapiro PR is updated to address the issue you mentioned. Please try again. Thanks for reporting this.

[yoel-shapiro]

fixed!
thank you!

[pengzhao-intel]

@marcoabreu Looks like CI has trouble to build this PR. Some builds failed to compile this line
......
So comparing size_t with index_t causing build failed. Question: why our CI didn't report this previously, and even not from other PRs sent at same time? Does ccache hide this issue?

@marcoabreu Sorry to bother you. Currently, the PR is blocked by CI failure as @ZhennanQin mentioned above.
Would you help take a look for the CI issue after the holiday? Thanks in advance.

[szha]

The API change that @sergeykolychev pointed out needs to be addressed first. Requesting change to make it clear.

[pengzhao-intel]

Thanks for the suggestions and comments @szha @reminisce @sergeykolychev
We are on the Chinese holiday now and will address all comments when coming back to the office.

API concern is addressed. Thanks for the quick turnaround.

[ZhennanQin]

@reminisce As you have concern about entropy change, I removed the parts that for uint8 fix, as it's not the major task of this PR. Other entropy change are re-applying the fix of entropy which removed by mistake. Please review again. Thanks.

[xinyu-intel]

@reminisce
I tested resnet152 and inception-bn on Tesla V100 and resnet152 looks good. However, there is a quantization bug with inception-bn because #13297 enabled quantized_concat on CPU side. It seems that #14060 is fixing this bug. Below are accuracy numbers of these two models(after apply #14060 ).

MODE	ResNet152	Inception-bn
FP32	77.18%/93.00%	72.38%/90.61%
online	75.46%/92.24%	72.08%/90.31%
5 batch naive	75.41%/92.10%	71.81%/90.29%

[reminisce]

@xinyu-intel Thanks for the verification. I have approved PR #14060, but I cannot merge that PR because it hasn't passed CI test.

[reminisce]

PR #14060 is merged. Please rebase to trigger CI again. Thanks.

[pengzhao-intel]

@szha @reminisce @KellenSunderland @TaoLv all comments are resolved now.
Could you help take a look again and merge the PR in case no other new comment?