Martin_T
(Martin T)
November 12, 2020, 9:00pm
1
I see different results after parsing a JSON file (using ParseJSON
) with following double values:
Original:
{"pgSz":{"w":597.60000000000002274,"h":842.39999999999997726}}
Parsed:
{"pgSz":{"w":597.5999755859375,"h":842.4000244140625}}
Can anyone tell if this is a bug or if it is automatically rounded due to the length of the decimal point? Are the double values too long after the decimal point?
DerkJ
(DerkJ)
November 12, 2020, 9:11pm
2
Double is by definition not precise so this could happen. Perhaps an engineer could let you know if this could be a bug or not.
But note that with doubles you won’t get the exact values you expect mostly.
In fact for paper sizes, you may as well be storing these as integer sizes, that’s how they are defined.
A1 paper is 594 x 841
I’d say it was a bug, the original double seems to be being changed into something shorter.
597.60000000000002274
0100000010000010101011001100110011001100110011001100110011001101
597.5999755859375
0100000010000010101011001100110011000000000000000000000000000000
2 Likes
Do they crop it to 32-bit float instead of 64-bit double?
2 Likes
It looks like its being converted to a single.
1 Like
Works JSONMBS class in with MBS Xojo Util Plugin :
and the ToString outputs it like this:
{
"pgSz": {
"w": 597.60000000000002274,
"h": 842.39999999999997726
}
}
As you see we preserve the digits.
Martin_T
(Martin T)
November 13, 2020, 10:21am
8
Sounds logical. I‘ll create a feedback case.
system
(system)
Closed
May 14, 2021, 10:21pm
9
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.