Options

Possible Rounding issue in NAV

jordi79jordi79 Member Posts: 273
Hi,

Code to perform a rounding as as follows:

Dec1 := 14.32455;

Precision := 0.01;

MESSAGE(FORMAT(ROUND(Dec1, Precision, '=')));


Once code is executed, the result is: 14.32

I would have expected the output to be 14.33. Is my assumption correct? This code is executed in NAV ver 5Sp1. The same result is observed in ver NAV 3.10.

Comments

  • Options
    BeliasBelias Member Posts: 2,998
    no, the result is mathematically correct, if you round to the second dec, you have to take into account only the third decimal. example:
    2,444444444444444444445 rounded to the second dec is 2,44 :wink:
    -Mirko-
    "Never memorize what you can easily find in a book".....Or Mibuso
    My Blog
  • Options
    jordi79jordi79 Member Posts: 273
    Hi Belias,

    You are correct. The result IS mathematically correct. What was I thinking! Must be all the hours spent staring at the computer have fried my brains.
  • Options
    BeliasBelias Member Posts: 2,998
    Honestly, after reading your post, i have had 10 seconds of doubt about it, too :mrgreen:
    -Mirko-
    "Never memorize what you can easily find in a book".....Or Mibuso
    My Blog
Sign In or Register to comment.