Rounding dilemma

VFP has one numeric type, called, not surprisingly, numeric. You should not assume that its internal representation is IEEE floating point. It probably is decimal, given the need to digest database numerics which are not true floating point but a defined / known # of decimal places with n digits of precision. I would imagine they are themselves decimal types, generally.

I understand that IEEE floats offend your sense of decorum – it does me, too, which is why I don’t use them. But where they shine is in matrix algebra and the like where speed and memory efficiency are paramount, and the developer understands the tradeoffs and is willing to work around them.

They are only as popular as they are because of the slowness and memory constraints of early computer systems. IMO today they are anachronistic at least for line of business applications.

1 Like

As another illustration, according to what I can find online, it happens that 16,777,216 (2 to the 24th power) is the largest integer that a single precision float can accurately represent.

The largest integer that can be represented is 39 digits long but the next lowest number is 2^104 less than that. These gaps get smaller as the number gets smaller, until it disappears at 16,777,216.

I know devs who don’t have this background concerning floating point types loose their s__t over things like this but don’t shoot the messenger. It arguably “shouldn’t” work like that but it manifestly does.

The real problem is why, in a product that will often be used for line of business apps, after 20 years, isn’t there a native decimal data type to properly address this issue?

Var originalDouble As Double = 32.175

Var asCurrency As Currency = Round(originalDouble*10000)/10000

Var as2digitsCurrency As Currency = Round(asCurrency*100)/100


And this could be wrapped in a function to insure the expected results – but at the expense of several floating point multiplication and division operations. In practice, probably an acceptable tradeoff but you have to know to do it. It would be better if there were a native Decimal type that Just Works, such that everyone who got an incorrect on a 3rd grade math test and isn’t inclined to give the framework a free pass when they didn’t get one, can just have things the way they are “supposed” to be.

I do like the idea of an arbitrary precision number type pairing with that found in SQL types NUMERIC or DECIMAL found in PostgreSQL, MySQL and etc.

1 Like

And how are you doing the rounding in Xojo?

What happens if you use Single rather than Double as the type? It is likely what FoxPro was using.

Machines in the past have had decimal hardware. The Xerox Sigma 7 that I worked on in the 1970s could be purchased with such hardware, although the unit I worked on didn’t have it. The decimal hardware operated on decimal strings up to 31 digits + sign, and the four basic operations (add, subtract, multiply, divide) could be done. This would effectively be integer arithmetic, you decide for yourself where any decimal point resides.

Looks like IBM can sell you a machine with such hardware.

Yes and it’s not that hard to come up with a storage format either. Convert the digits to hexadecimal and add a sign nibble and you get nearly 50% compression for storage. The client, as you point out, has to “know” where the implied decimal point goes, and you’d need some kind of basic math operations support. So 12345.67 becomes, typically, in hex:

12 34 56 7C (the C becomes D if negative)

1 Like

Dedicated hardware is not really needed today, as evidenced by the Decimal types in many languages, but yes, hardware support even today “should” be faster. I just doubt it’s fast enough to be a selling point in typical business apps. Maybe in something really intense like matrix operations.

And it looks like IBM can sell you a machine with decimal floating-point hardware, too. Using numbers up to 128 bits long, and in the same format as used by Xerox (at least in terms of how the digits are represented). Still of limited precision though, just like binary floating-point. Perhaps IBM already had such hardware on their 360/370 models and Zerox just copied the format.

Yes, two decimal digits per byte with the sign in the right-most nibble. (this is big-endian, of course).


Every programmer or engineer must adapt their formulas to obtain the correct result. Some languages do it better than others.

What surprises me a lot, speaking exclusively of companies and businesses, is that they do not help to facilitate the treatment of numbers in financial calculations.

With the use of WEB2, I already had an unpleasant experience with the development of a POS system. The explorer opened with different interpretations in the format of the comma or point, depending on the country.

As with everyone else, I assume they have rolled their own rounding function. Here is the one we have been using up until now, but we have a new one now that seems to work.

Two parameters passed, the number and the number of decimals.

Dim RoundedAmt,ABSNumber As Double
RoundedAmt = 0

ABSNumber = ABS(Number)

If Decimals = 0 Then
RoundedAmt = Round(ABSNumber)
RoundedAmt = Round(ABSNumber * (10 ^ Decimals)) / (10 ^ Decimals)

If Number < 0 Then
RoundedAmt = RoundedAmt * -1

Return RoundedAmt

1 Like


Actually, VFP has Numeric, Double, Float, Integer, and Currency for their data types in their table, but when you declare a numeric value in a variable, it simply shows it as “N” for a numeric type, but it does show the correct precision as shown below in this screen snippet.

Yes I was referring to there only being the one type of numeric variable (or memory variable, in Xbase parlance). My guess was always that if such a variable was assigned from a table field, it would use what is known about that field type, # of decimals defined for it, etc., to configure the variable to properly handle that. I don’t recall that there was any sort of programmatic access to the internals of numeric variables though.

My memory could be faulty because I haven’t touched VFP since about 2007.

Knowing what a crack team was working on the product before M$FT screwed the pooch, and what a great performer VFP code was, I would not be surprised to find that there was a “hidden” internal integer type as well, such that if you had a for loop variable, it would just assume it was an integer.

There are multiple definitions of rounding (see the Wikipedia article Excel has 6 functions for rounding - ROUND, MROUND, ROUNDUP, ROUNDDOWN, EVEN, and ODD.

Putting that aside and using the most common definition of rounding, I agree that rounding 32.175 to two decimals results in 32.18, however rounding 32.174999999 to two decimals correctly results in 32.17 as Xojo does.

The issue, as many have pointed out, is that any arbitrary Real number cannot necessarily be stored correctly in any computer using any system. The Real number line is uncountably infinite. For any pair of Real numbers on the Real number line, no matter how close, there will be an infinite number of Real numbers between them.

It is up to the programmer to understand the requirements of the domain they are programming for (finance, engineering, general math, etc.) as well as the limitations of the software they are using. Floating point numbers, while appearing simple, are plagued with many gotchas.



that used floating point numbers with 6 digits of precision and it handled floating point numbers correctly

Then they weren’t using Doubles. Unfortunately, you have chosen the wrong data type in your conversion. It is well known that you don’t use Doubles in a financial application. The closest Xojo has is Currency. Since that doesn’t meet your needs, you’re going to have to roll your own or use a third party solution like Delaney’s.

There is no mystery to the fact that not all decimal numbers between .000 and .999 can be represented precisely in binary computers. Money is decimal; computers are binary. That is not a flaw, it’s a mathematical fact. While Xojo provides a type (Currency) for programmers, the traditional method is to make all numbers integer then use some type of built-in or home-built display function to get the decimal point in the right place. Most COBOL programs I have come in contact with over the years use this type of system.

Currency type in Xojo is a binary fixed type (4 decimal digits) also able to conversion failures as

COBOL have some real precise decimal ones. BCD ones, so a

MYVAR PIC 9(15)V9999 COMP-3

or bigger can receive a value as 9999999999999.9876 without distortions