This forum uses cookies
This forum makes use of cookies to store your login information if you are registered, and your last visit if you are not. Cookies are small text documents stored on your computer; the cookies set by this forum can only be used on this website and pose no security risk. Cookies on this forum also track the specific topics you have read and when you last read them. Please confirm that you accept these cookies being set.

rounding error
#4
If you try writing 103 with ETS you will get 103.04. The thing is that it's not possible to encode 103 as a 2-byte float value, you either get 102.96 or 103.04. If you need better precision then use 4-byte float if your devices support it.
Reply


Messages In This Thread
rounding error - by rocfusion - 22.07.2017, 21:22
RE: rounding error - by admin - 24.07.2017, 06:01
RE: rounding error - by rocfusion - 24.07.2017, 08:39
RE: rounding error - by admin - 24.07.2017, 08:52
RE: rounding error - by buuuudzik - 24.07.2017, 09:49

Forum Jump: