High Memory usage in NAV 2013 R2

engsiong75

Hi

I encountered a very puzzling problem with NAV 2013 R2. I accidentally ran a report which ran into an infinite loop. I killed the report after 5mins when I realized something was wrong. Suddenly, I found that I had difficulty login into Navision. Even when I logged in, running any reports would take a long time to run if it does not hang the client. After troubleshooting, I found that the Application Server Nav Server was using a high 12-14 GD of RAM out of a total 16GB.

The high memory usage on the application server was affecting my login and subsequent performance issue. This part I can understand. What I could not understand was that the memory usage remain high even though there was no users using the server. (It is a NAV Server is used for testing.) Secondly, assuming that the infinite loop caused the high memory usage, it should release the memory when I terminated my session.

Things to test out:
1) Restart the NAV server service on the application server.
2) Patch it with the latest rollup.

Does anyone have any other suggestion?

Thank you,

geordie

How did you kill the report, simply forcedly closing the client? I think the session remained active and running, consuming resources as much as the application was able to allocate.

engsiong75

Yes. I killed the session from task manger. I will also checked that the session was terminated on the server.

The problem was not occurred after I restarted the NAV server. I am not able to recreated by running the report with the infinite loop.

sylvainroy

Sharing a similar experience, hoping we will eventually find an explanation.

We were also having performance issues with a 2013 R2 instance. Opening up a simple page would take seconds, one user being logged in. I checked the SQL and NST server (same machine) and found out that SQL was consuming 25-35% of total CPU. Investigating further I found that the process within SQL that was pumping out the CPU was RESOURCE MONITOR. You can use the following query to find that out:

select * 
from sys.dm_exec_requests
order by cpu_time desc

Doing some additional research, I found out this blog : http://danieladeniji.wordpress.com/2013/09/21/technical-microsoft-sql-server-v2012-high-cpu-resource-monitor/ and found out that the Navision Server instance was taking over 90% of all the memory available on the server, which I think was causing the high CPU of RESOURCE MONITOR.

I restarted this specific instance and everything turned out suddenly OK... CPU usage went down and memory usage from NAV server came back to a normal.

This must be one of the new features of R2!

Torben_Wind_Meyhoff[MSFT]

What kind of report was it?
The server pushes replies to the client per 100 rows, so if the client stops responding it should also stop execution on the server.
Did you try the Sessions page? Try to attach the debugger to the session next time to see where the thread is running.
BR
Torben

Phenno

Today we had exactly the same issue on NAV server, high memory usage even after we killed all sessions (via powershell). NAV consumed all available memory leaving nothing for OS and SQL server (on the same server) thus almost total system hang.

After little digging we found a cause of memory leak. On one batch-report there was an infinite loop with message box in it. I suppose, NAV pushes messages to the stack, and for infinite loop that pumped memory to its limits.

You can easily reproduce this situation by executing the code

WHILE 1 = 1 DO
  MESSAGE('test');

Keep in mind that memory will grow very fast, in few seconds several Gigs.

Of course, this is not a normal code to make and execute but the problem is that memory stays reserved for process even after session is killed, the only solution is to restart NAV instance. Maybe I didn't wait long enoungh for process to release memory by itself.

krzychub83

I will add my 2 cents to this conversation.

Yes, this is a feature of NAV 2013 and it is very well advertised! As a result you need to be very careful with your code now, as NAV Server is 64 bit application and can eat a lot more memory now!

I’ve seen a lot of examples where incorrect .NET code was puling data from NAV Server without any ‘proper’ filters. So in a result it was trying to pull all data from some tables. In case of websites, IIS was terminating browser request with timeout error. So the user was getting a page like 'Your request was timed out'. However NAV Server was still trying to deliver requested data by IIS. So it was growing and growing. Before NAV 2013 era, NAV Servers were 32 bit applications, thus they were breaking with out of memory error when they were reaching limit of 2GB. However now they are 64 bit applications, thus they can consume much more memory.
I do not know exactly how much memory can be consumed by NAV Server. I read somewhere that it was 120GB, but then I read somewhere else that it was 250GB.

So how to protect yourself? There is no perfect solution for high memory consumption, but I would suggest to:
1. Write proper code. We are all humans, and even the best tests and testers might not notice code bugs, thus read it more as 'be aware of the risks and minimize them'. Add tests related to processing large amount of data. Monitor how memory consumption is behaving. Etc.
2. I would suggest not to use NAV Server and SQL Server on the same servers. Create dedicated NAV Servers for tasks like Web Server, NAV Server, etc. (as NAV Installation Manuals suggests too)
3. Lookup Windows System Resource Manager, as it can be used to kill NAV Server before it will try to kill whole server. As you know when you will consume whole memory then page file is being consumed. After page file is fully consumed then funny things start to happen to the servers due to no memory left. Though I don’t remember now if NAV Server (2013) was fully killing server or not, as it is a while when I was testing it and it was never my primary task to do. I am preparing quite a large database now to be upgraded from NAV 2009 (classic client) to NAV 2013 R2 between all other normal tasks. So there is still on my list to check how to limit risks related to memory consumption from different sources (like how reports will behave when a user will select a very large data set in it…).
4. Identify where you might reach memory limits, and move those users/tasks to dedicated servers. Even with smaller hardware configuration, taking them out of the scope will result in better performance for rest of the team.

If only there was somebody from MS looking at this thread… It would be a nice feature to have upper memory limit, which can be consumed by NAV Server, in NAV Service’s configuration files. The same way how you can limit SQL Server’s memory consumption in SQL Server itself. Still Windows System Resource Manager is a better solutions, as setting limits there gives you more flexibility.