Bad performance native NAV 5.1 on VMware 3.5i

vdstroet

We are working for 3 weeks with several specialists on a weird problem:

The performance of NAV 5.1 using a database server on a virtual VMware machine is very bad. The test is very simple: we try to import one million records using a dataport. A messagebox appears after import displaying the runtime.

The results:

Import runtime using NAV 5.1 running on a database server (VMware virtual machine): 580 seconds
Import runtime using NAV 5.1 running local (VMware virtual machine): 25 seconds
Import runtime using NAV 5.1 running on a database server (physical machine): 40 seconds

The second test with a runtime of 25 seconds shows that the hardware is running fine. A few specialists from Dell and VMware have confirmed this.

So it seems that running the database server service slows down the system.

Anybody who has a possible solution for this problem?

matttrax

Well first of all, why are you running your database server on a virtual machine? I mean I know you can, but it's definitely not best practice. Certainly not recommended by Microsoft.

Any reason it can't be a physical machine?

vdstroet

Moving our physical environment to a VMware environment has many advantages in terms of higher availability, costs reduction and centralized data storage.

The question is why the database server in a virtual environment results in such a low performance. There must be a reason why doing the same test on the local database is 23 times faster than a database which is called via the database server service.

Parameters like commit cache, cache etc. are correctly set and are the same as on the physical machine.

kriki

You file to be imported was on a local drive?
If it is on a remote disk, it slows down a lot.

vdstroet

Yes, it's on a local drive.

Poltergeist

Apparently, a local database is always faster compared to a server database (running through a service). It seems that the database service adds a lot of overhead (which is understandable). I've also conducted some tests, which show that it can lead to a 50% perfomanceloss. I think that if you run the same test on the physical machine using a database service, you will see a decline in performance (but not as much as you do now).

But I do not understand how your virtual server is set up: Is the database itself inside the virtual machine (so, on the virtual disk), or is it on some external drive? If it's on the virtual disk, it's understandable there is a performance degredation. you should never run a database from a guest os drive. If it's not, how fast is the connection to the database? (1Gb, 100 Mbit, fiber, copper)

Then, are you running antivirus software inside the virtual machine? If so: disable it, or at least exclude the databasefiles. Are you running a virusscanner on the host? Disable it.

DenSter

Without knowing the details of the installation, which make a HUGE difference when working with VM's, there's not a lot we can say about it.

A VM is nothing more than a file that acts like a computer. If you are running your VM on your laptop, it's a computer within a computer that uses the system resources of the host machine. So if you have 1GB of memory, now you have two operating systems competing for it, and there's your performance loss right there.

We'd have to know a LOT more about your setup, because you can't say anything generic about this.

vdstroet

The virtual server has two virtual disks:
The first disk is the one with the OS running on it, is based on RAID5 on our ISCSI SAN.
The second one with the database is based on RAID1 and also on the ISCSI SAN.

The server and the database are connected via dedicated ISCSI (1 Gb) connections to the SAN. There is no virus software installed.

The tests are done on the virtual server itself. We started a client which is installed on the virtual server and performed the tests.


Setup of the system:

Two ESX 3.5i VMware hosts (Dell R805i servers with dual quad core AMD Opteron processors 2.4 GHz).
Each host has 32 GB RAM. VMotion and HA are enabled.

The virtual server inside the ESX host is running on Windows 2003 SP1. 4 GB memory and 2 processors. We also tested with 2 GB memory and 1 CPU. We also did some tests using Windows 2008 as OS.

The Navision database size is 25 GB. 75% of the database space is in use.

The virtual machines are stored on a Dell MD3000i ISCSI SAN.
At this moment we have no other machines running in this VMware environment.

ajhvdb

Wow, i'd love to see a fix for this. I'm just about setting up something like this.

Are there any other hints why you started this test? (i.e. what else was slow?).

DenSter

vdstroet wrote:

The virtual server has two virtual disks

But the VM itself, where does that exist? You can have a super SAN set up virtually in your VM, but if it's all one file on a laptop you are going to be limited to what the laptop can do.

I really don't know if it is a good idea to have a virtual RAID 5 by the way. Having a NAV database on a physical RAID 5 is already not recommended, and virtualizing would only make that worse.

Having said all of that, I really don't know much about virtualization. I'll send you a PM with a contact that I know in Holland that does.

<edit> ok you put in more details than I read at first never mind my critique </edit>

vdstroet

@ajhvdb:
3 Weeks ago we planned to move our Navision database from an old physical server to a virtual machine. Before making this move, we did some workload tests and discovered that it took much longer to perform these actions on our brand new hardware than doing it with the old server.

ajhvdb

Let's hope someone with a simular vmware setup reads this.

Did you try it with just a simple harddisk (no raid, san). I've this running and it is a little bit slower then in a normal PC setup but that is to be expected. (3 vm's on 1 PC, esxi)

vdstroet

We tried it also without a SAN. We created a virtual machine and a Navision database running on the local harddisks of the ESX host. The performance was almost the same.

vdstroet

I got an answer from Microsoft support:

"With the Virtual Environment the communication between the OS and the disk there is add the disk driver from Virtual Software, which for the Microsoft Dynamics NAV Database Server will slow performance completely down."

This sounds as "no solution possible". But there are several native NAV 5 servers running in a VMware environment at other companies which are running fine. So IMHO there must be a solution.

We discovered something strange yesterday. When running the tests, the processes on the virtual machine showed next CPU usage:

System 69%
Fin.exe 20%
Server.exe 8%
Slave.exe 0%

It seems that some system activities increase enormously when doing these database transactions.

Poltergeist

SQL Server runs fine in virtual environments, so you might consider using that as your Database platform...

vdstroet

Changing to SQL is no option at this moment. Our Navision system is too much customized to make do a quick conversion to SQL.
Getting this virtual machine performing is our first goal.

nromijn

Hello,

Does anyone has a solutions for this problem?

Please let me know.

TonyH

Have you run a SQLIO check on the RAID Controllers? (Search for the SQLIO app, thinks its on the MS website)

Back in the day we did some comparison work on disk controllers and found that HP out performed DELL by 4x!!!!
But that was a while back.

Kick the RAID 5, adopt RAID1+0 for your data drive and make sure its on a separate spindle. Place the logs where you like but not on the data spindle.

To me your issue is your setup, I have customers who run VMWare and we run it internally, as well as VirtualPC and I cannot report the same issues you are having....

t

<ignore the "logs anywhere else" since you won't have one>

themave

vdstroet wrote:

Moving our physical environment to a VMware environment has many advantages in terms of higher availability, costs reduction and centralized data storage.

The question is why the database server in a virtual environment results in such a low performance. There must be a reason why doing the same test on the local database is 23 times faster than a database which is called via the database server service.

Parameters like commit cache, cache etc. are correctly set and are the same as on the physical machine.

Saving are not magic, something has to give, in this case, if you want to save and run it on a virtual server, then you will pay the price in performance. Just my opinion.

rdebath

This should work fine, but you still haven't given us much information, still, I'll try.

Call ESX server VMHOST.
Call the Windows 2003 server that's running the Navision database server inside the virtual machine on the ESX server VMNAVISION.
Call the Client machine running XP/Vista on a physical machine CLIENT1
Call another Windows 2003 server running a Navision database server on a physical machine PYNAVISION

Your three tests at the start:

1) SLOW: Navision client running on CLIENT1 connects to database server running on VMNAVISION
The problem.
2) VFAST: Navision client running on VMNAVISION opens database file directly.
This test means that the disk subsystem is performing fine. (Despite the RAID5, iSCSI and Microsoft's reply)
Do check that the disk is idle after the test to be sure that you haven't got some sort of commit problem too.
3) FAST: Navision client running on CLIENT1 connects to database server running on PYNAVISION
Seems to say that the client machine is fine.

Please try:

Navision client running on VMNAVISION connects the database server on VMNAVISION. Do not open file directly. This tests the setup of the Navision database server on VMNAVISION.

Navision client running on VMNAVISION connects to the database server on PYNAVISION.
This tests the physical and virtual network pieces that are not tested in your test (3).

Other items.

Please check VMNAVISION has correct drivers for it's virtual hard drives and network adaptor.
Please check that the database drives are preallocated and not shared on the SAN (Test 2 seems to say this is OK)
Were are you running the iSCSI initiator for the database drives; VMNAVISION or VMHOST? Does it make a difference for you?

vdstroet

Thanks for your answer Robert,

Your description of our tests is OK except the fact that we tested with the client on the machine which was running the Navision database server. This to exclude network issues.
Running the client on another machine slows down the performance a little but not much.

We checked the drivers, this is OK.
The database drives are not shared.
We tested with Microsoft ISCSI initiator and with the VMware ISCSI adapter. Also no big difference.

Mark Brummel who is specialized in performance problems checked also our setup. Mark also tested our database is his own testing environment. Almost the same results.

We discovered that a Nav 5 database server running on VMware server (the free VMware version) performs much better than Nav 5 running on a ESX host.

A week ago, we asked VMware to investigate what's going wrong with Nav 5 on ESX.

I'll keep you all informed.

Joeri

Hi all,

i know that this is an old topic, but i'm having the exact same problem with a customer.
They had a 3.70 database (native) and migrated it to a 5 SP1 database (native) on a Windows 2008 server that runs on VMWare. The performance is really terrible.

When I open the fdb directly (local) i can import a million records < 1 minute, but when i try it using the native db server it takes forever...

Have you found a solution for this problem already?
Any help would be greatly appreciated.

rdebath

IIRC the only real difference between 3.70 native and 5.00 native is the use of tcps. As the TCPS uses an encrypted connection it may be slower.

If you setup the connection to the database server to use TCP rather than TCPS does it make a difference?

kriki

-Is your W2008 64bit or 32bit. For native server, it is best 32bit because NAV native is 32 bit and 32bit programs run slower on 64bit Windows.
-Are you sure you have COMMIT-cache enabled?
-How much memory have you given to your DB-cache? And how big is your DB (not the file but the data in it)? And now we talk about it:how big is your file (you need at least 20% free space for good performance)?
-How much memory has your VM? It should be +- 512MB for 32bit or 1GB for 64bit + 1GB for the native server.
-Are you sure you are not overcommiting memory on VMWARE (meaning for example you have 10 VM's with 1 GB of mem and fysically you have only 8GB)?
-How are the disks of your VM? They should be REAL disks, not diskfiles. How are they configured (NO RAID5!)?

mrQQ

has anyone ever solved this slow native server on vmware thing?