HIVE-28673: Fix issues in JSON SerDe implementations related to Decimal #5584
Conversation
|
| @@ -131,7 +143,7 @@ POSTHOOK: type: QUERY | |||
| POSTHOOK: Input: default@json_serde3_2 | |||
| #### A masked pattern was here #### | |||
| -2 FALSE NULL TRUE 1.23E45 value | |||
| -2 false NULL true 1.23E45 value | |||
| -2 false NULL true 1.23E+45 value | |||
There was a problem hiding this comment.
Why would this expression change? Is it a valid and expected change?
There was a problem hiding this comment.
@okumin Thank you so much for the review!
This was not an expected change but definitely a valid one. It is a java issue as we are now using java.math.BigDecimal for float values. BigDecimal parses scientific notations and adds a '+' to our number.
| @@ -172,7 +173,7 @@ public HiveJsonReader(ObjectInspector oi) { | |||
| public HiveJsonReader(ObjectInspector oi, TimestampParser tsParser) { | |||
| this.tsParser = tsParser; | |||
| this.oi = oi; | |||
| this.objectMapper = new ObjectMapper(); | |||
| this.objectMapper = new ObjectMapper().enable(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS); | |||
There was a problem hiding this comment.
can this change be made applicable only for Primitive decimal type ?
There was a problem hiding this comment.
@Indhumathi27 Thank you so much for your review!
The changes are required for all decimal types as jackson is not using BigDecimal even for big numbers. Hence the numbers are getting rounded off.
What changes were proposed in this pull request?
Fixing jackson parser code to increase precision of decimal values during deserialisation.
Why are the changes needed?
Jackson tree parser incorrectly parses decimal values, that leads to data issues. [Please see jira to reproduce steps and see the wrong values]
Does this PR introduce any user-facing change?
Yes
Is the change a dependency upgrade?
No
How was this patch tested?
My test case and current testclasses