Issue with math: float calculation, ToStr()

Archived
1

Try this:
? 1.1* 9
console prints 9.9

But try this:
? str(1.1 * 9)
console prints 9.900001

I assume this is some sort of floating point rounding error having to do with binary math and how computers work, etc etc. But it’s a problem for me, because in my game when the player is holding 9 spider abdomens in his or her inventory, on the list the weight is showing up at 9.900001, when everything else is just one decimal point, not six!

Any way to force a float to only be one decimal point accurate?

Is my only recourse to always check the final string and count characters after the “.” ?

2

I just implemented this to brute force solve it:


FUNCTION checkStrFloat(Str AS STRING) AS STRING ' trig by getItemDscr(1), c2Inventory(1)	' Search for a . in the string, then check if the string is longer than one character past the . and if so, trim off the rest
	StrLen = Str.Len()
	FOR i = 1 TO StrLen
		chr = Str.Mid(i -1, 1)								' String.Mid() is 0 based, not 1 based, so need to -1
		IF chr = "."																				' Ex: "9.900001", i will be 2
			IF StrLen > i +1																		' If there is more than just one character after the decimal point
				StrNEW = Left(Str, i+1)						' Left() is 1 based		' Ex: i +1 will be 3, thus first three characters of "9.900001" will be "9.9", one character past the "."
				? "checkStrFloat() found long float Str, orig " Str ", shortened " StrNew
				RETURN StrNEW
			END IF
		END IF
	END FOR
	RETURN Str
END FUNCTION
3

Due to the way floating point numbers are implemented, your number could be “9.899999”, in which case your function will return “9.8” when it should return “9.9”.

I’d do something like this (assuming your numbers are stored internally as Doubles, but you want to print them to one significant decimal place):


function convert(f as double) as string
  a = CInt(f * 10)
  n = a \ 10
  m = abs(a) mod 10
  return n.toStr() + "." + m.toStr()
end function
4

“Komag” wrote:
Try this:
? 1.1* 9
console prints 9.9

But try this:
? str(1.1 * 9)
console prints 9.900001

A couple of options:


x=1.1 * 9

' 1) use Fix to try to force 1 digit of precision.  may or may not work?
? Str(Fix(x * 10)/10)

' 2) use printf-style conversion
? x.ToStr("%3.1f")

' Note that Str prefixes a blank on non-negative numbers, the above ToStr does not.
' if you want that, use x.ToStr("% 3.1f")
5

My advice is to only use integer math unless there’s a compelling reason not to (which there never is). If you only want one decimal point of precision then multiply everything by ten (9 spiders at 11 tenths per abdomen is 99 tenths). There’s any number of ways to display 99 tenths as 9.9.

-JT

6

“renojim” wrote:
My advice is to only use integer math unless there’s a compelling reason not to (which there never is).
Animation might be an exception.

7

For some reason I had the impression that x.ToStr() was only for integers, not floats. Because that actually prints correctly:

? str(1.1*9)
 9.900001
? (1.1*9).tostr()
9.9

Thanks for the suggestions. I like the idea of always doing all math with integers, but at this point I don’t want to go back and rework everything, so I’m going to go with “laundering” all my floats before converting to strings, with:


FUNCTION washFloat(dirtyFloat)
 Int10 = Cint(dirtyFloat *10)
 cleanFloat = Int10 / 10
 RETURN cleanFloat
END FUNCTION

And then I’ll use float.ToStr() for double measure (unless there are Roku models or firmware on which that won’t work?)

8

“Komag” wrote:

And then I’ll use float.ToStr() for double measure (unless there are Roku models or firmware on which that won’t work?)

Default ToStr() on intrinsic types (numeric, Boolean, and string) goes back to firmware 7.0.
ToStr and string Format with printf format specifiers goes back to firmware 7.6.