27.08.2021, 12:20
float16 has limited precision so you can't get 5000 precisely. If possible use float32 or int16.
set value 5000 on float16 becomes 4999.68
|
« Next Oldest | Next Newest »
|
Messages In This Thread |
set value 5000 on float16 becomes 4999.68 - by spyder - 27.08.2021, 12:11
RE: set value 5000 on float16 becomes 4999.68 - by admin - 27.08.2021, 12:20
RE: set value 5000 on float16 becomes 4999.68 - by spyder - 28.08.2021, 09:50
|