Performance weirdness

jwilder

I am no stranger to NAV and performance. I have recently come across something that is about as weird as it gets. I know when it comes to performance problems there are tons of variables but I thought I would put this post out there in case anyone has come across this before.

We are running NAV 2009 R2 Classic. Basically I am moving some of our NAS's from one server to another. One of these NAS's is a job scheduler that records start and end times for the jobs it runs. I noticed on the new server that jobs were take 2 or even 3 times the length to run. The new server is a virtual server and is superior in every way to the old server. Newer, more memory, faster cpu's, etc...

So my next step was to take NAS out of the picture and just try with a NAV client. So I created a form with 3 jobs of different lengths that record the start and end time. These jobs are reading, modifying and deleting records, nothing special. Same problem. The new server was much slower. I have now tried this form on many machines and most of them are the same as the newer machine (slow). We have found 3 servers that run things fast and everything else we have tried is slow. The 3 fast servers are different operating systems and one is 32 bit and the others are 64bit.

Since we are running the exact same code on each machine it is not a NAV code problem or index problem.

We have ruled out the network by taking the connections from the fast machine and plugging into a slow machine. The slow machine was still slow, no difference. That seemed like a pretty good test to rule out the switch, network segments and who knows what else.

Is there anything with network protocols being enabled or disabled? Anything with the nettype value I setup in NAV (typically set to DEFAULT)? Objectcache is set to 40000, seems reasonable. Anything with certain builds of NAV that might cause problems (we are running 6.00.32519 on most machines)? Is there different versions of .NET or SQL Native Client that might cause a problem?

I will take any and all thoughts of tests I could run to help figure this out.

davmac1

have you tried doing a Windows copy of a large group of files from the NAV server to the PC? I am not a network guru, but there are things like packet size and latency. If you can test the performance outside of NAV, you can see if there are other issues.

kriki

Are you sure it is the NAS or network?

It might also be that the VM/host has powersaving on. This should be turned off. The powermanagement should be in high performance mode. To be checked in Windows and the BIOS of both host and VM.

jwilder

I have tried file copies and they generally take the same amount of time. This again leads towards taking the network out of it.

kriki

The Powersetting just got higher up on the suspect list!

PS: you can test it out. What I did once I had the problem and suspected this problem is this:
-I created a small codeunit that just looped from 1 to 2.000.000.000 and then restarted from 1. I ran a classic client with this codeunit for each processor in the VM. With Processexplorer I changed the priority of those clients to background (=4). That means that all CPU's will be going at 100% but if some other process kicks in, these clients will run slower to let the other process have all the CPU it needs while keeping the CPU's at 100%. Doing that at my customer made everything go fast!

Keeping the CPU's at 100% (or close) avoids that the powersaving kicks in.

What happens when powersaving kicks in?
If you CPU normally runs at 3 Ghz, with power saving it goes down to under 1 Ghz to save electricity (electricity savings are only up to 25%, so you lose 66% of performance to save 25% of electricity. BAD deal!).
Once the CPU is occupied (generally it is only a peak), the CPU has to be cranked up to 3 Ghz. This takes time. When it finally is at 3 Ghz, the peak has generally finished, so the CPU goes down again. In that moment all is slow because you are basically running a 1 Ghz server instead of on a 3 Ghz server!

JoergR

Hey, you can try to log your performance counters using perfmon.exe and add important indicators like network wait queue, cpu time, hard disk wait queue, page fault rate and so on.
Maybe you can get an insight whats going on.