NAV2009R2 Webservice Performance Issues - Multiple calls

ScottJones · 2011-06-27

We are having a few performance issues when calling middle tier web services repeatedly at very close intervals and I wanted to raise this to see if any other peeps out there have stumbled on this before we use one of our valuable Microsoft support incidents. I have developed a very simple web service Page to demonstrate this in DynamicsNAV2009R2 see function below:

//Codeunit exposing WebService function
FncAnswerToUniverse() : Integer
// Called by ASP.NET to test latency communication with middle tier
EXIT(42)

using System;
using System.Collections.Generic;
using System.Text;

namespace PerformanceChecker
{
    // Import newly generated Web service proxy.
    using DemoCallWSCustomerPage.WebService;

    class Program
    {
        static void Main(string[] args)
        {
            int IntTheAnswerIs = 0;

            // Create instance of service and set credentials.
            SvcSTPGlobalFunctions GlobalFunctions = new SvcSTPGlobalFunctions();
            GlobalFunctions.Url = "http://SOV-NAV-T-APP2:7047/DynamicsNAV/WS/Kelway2009ST/Codeunit/SvcSTPGlobalFunctions";
            GlobalFunctions.UseDefaultCredentials = true;

            int numberOfIterations = 50;
            for (int i = 0; i < numberOfIterations; i++)
            {
                DateTime start = DateTime.Now;
                IntTheAnswerIs = GlobalFunctions.FncAnswerToUniverse();
                DateTime end = DateTime.Now;
                TimeSpan runTime = end - start;
                
                Console.WriteLine("The answer to the universe is: {0} and it took {1} m/s to compute!", IntTheAnswerIs, runTime.Milliseconds.ToString());
                
                //Comment out and run full speed and compare time to compute
                System.Threading.Thread.Sleep(300);
            }

            Console.WriteLine("Press [ENTER] to exit program!");
            Console.ReadLine();
        }

    }
}

When we repeatedly call this from an ASP.NET test harness we get timings of around 200ms for each round trip to the web service (Running on clients mega servers)

If we place a delay of 300m/s between each call (Uncomment line in sample attached) to the web service function we get responses round trip responses of arounf 8-10ms

It seems to indicate there is something within the middle tier service that is throttling responses back one it has replied to a request.

We have also written a test harness to send the request in sequence to 4 separate middle tiers services. This handles the load a lot better, but we still get the 200ms delay on each middle tier service before we can re-use it (see tier 0 timings) but because the requests are spread across multiple middle tiers it handles the load better. See screen shot below from our development server:

Has anyone else experienced performance issues with Webservice calls like this??? Any help/ideas would be appreciated.

Thank you!

Belias · 2011-06-27

works like a charm @full speed on my standalone installation (nst/sql and your console application on same pc).
If you find something, please post it here, cause i'm also doing some work about WS

ScottJones · 2011-06-27

Oh that's strange!!! Errr, is that on three separate servers/tiers??? I've tried it on clients servers and our servers and get same slow results when bombarding middle tier with requests... We are running on R2 build 32259 are you on a Newer version?

Bergamo?? I was in Northers Italy (Passo del Bocco) last month for Giro del Italia & Mille Miglia !! Absoulutly Fantastic

ara3n · 2011-06-27

I believe the issue is authentication. I was reading a while back that delegation was slowing down the response time.

ScottJones · 2011-06-27

What as in the configuration of Kerberos and the tickets passed between the three server tiers? This might explain why it's not apparent on a single machine. I'm sure it's not my test harness and after trying it on two different server locations (Our servers and clients mega blade servers running VMware) and setups i'm sure it's not something specific to our develpment servers.

I find it quite amazing it is exaclty 200ms!!! If I put a sleep of 100ms it still takes 100ms for web service call adding upto a total of 200ms give or take the odd ms!!!!

Belias · 2011-06-27

My build is navr2 32275, and of course i did not set delegation for my installation (it's standalone, as i said).
Now that rashed tell it, i've also heard about slowness due to delegation one year ago in MS italy, but my memory is ethereal

...i think i've a test environment <> from my pc in our company, i'll try it!

ScottJones wrote:

Bergamo?? I was in Northers Italy (Passo del Bocco) last month for Giro del Italia & Mille Miglia !! Absoulutly Fantastic

I'm glad you liked it! One day i hope to travel in US and feed you back as well

ScottJones · 2011-06-27

Belias, If you could try it on your company servers that would be fantastic. I am concerned that 200ms for each Webservice request is a long time on a busy web portal. I don't want to raise the support incident with Microsoft if other people have experienced this issue and have a fix or my test harness is 'Cane Merda'!

Our portal login procedure executes 4 webservice calls, this is taking around 1000ms because of this issue and we need to get this as fast as possible. Data binding to Telerik controls and page load are taking 50-100ms.

Thank you again

Scott

kine · 2011-06-27

Have you tried when you have some antivirus/firewall installed? Once I had problem with quick network and communication with servers when using Avast - it silently blocked the packets when they were send "too quickly" - and it made problems on the 1GB LAN... Disabling it was not enough, I needed to uninstall it....

Belias · 2011-06-28

Just wondering: what does it happen if you call 2 different webservices? what's the timespan between the two? something like (pseudo code, hope it's clear):

for i
time
answer1 = webservice1.function1;
time
answer2 = webservice2.function2;
time

i don't think anything different will happen, but who knows?

ScottJones · 2011-06-28

Belias,

I tried calling two different Webservices as detailed in ur Pseudo (This is what our portal does FncCheckCompany, FncLogin, FncValidateCustomerDetails, so exepected the same issue) and yes I receive the same 200ms delay between each call

If I run the test application in multiple times in separate threads I still get the delay but there is no drop in speed even with 10-20 copys running...

Antivirus... I have disabled Symantic on both tiers but difficult to remove without speaking to the gods... It's not made any difference disabling it...

My thought now is as suggested to move away from Kerberos and Delegation if this is possible???? I'm going to look into 'WebServicesUseNTLMAuthentication' setting in the CustomSetting.Config for the middle tier. Our web portal has it's own security engine and there is no need for pass through authentication/delegation and all the overheads.

Cheers

ScottJones · 2011-06-28

After countless test and late nights we are sure this is as a result of the Nagle algorithm RFC 896 6th Jan 1984 http://www.faqs.org/rfcs/rfc896.html. If the soap packet sent from the client to the middle tier is under the MTU size (Block Size) of 1460 characters. The server tier delays returning the acknowlegement to the packet as it expect further packets and doesn't want to flood the network with ACK responses. This is an issue for all development languages sending small TCP/IP requests.

If we add dummy parameters (with values) to the exposed web service function FncAnswerToUniverse(TxtPadding1 : Text[1024];TxtPadding2 : Text[1024]) so that the SOAP packet the ASP.NET client generates is > 1460 character the response is immediate. ie. around 15ms per request instead of 200ms!!!!

There are registery hacks for this on the internet but if we implement these it will effect the speed of large multi-packet responses (Streaming order history data). We are unsure what to do at this stage. Ideally there needs to be some why to disable the Nagle algorithm for each server request. Reading RFC's for WinSock this protocol was modified to allow this.

I now wonder if there is some way the middle tier can be insturcted to not use the Nagle algorithm in certain requests... If anyone out there has encountered this it would be appreciated...

Belias · 2011-06-28

amazing :shock:
you should mark this as Good posting, and talk with ms about it..
i don't know anything about "Nagle algorithm RFC 896", i have to study it, i'm really curious now.
thanks for sharing! :thumbsup:

ScottJones · 2011-06-28

All new to us too!!! Just fired an e-mail off to Lars at Microsoft see if he can help.

By the way for those who havent read 'Hitchhiker's Guide to the Galaxy'... http://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#Answer_to_the_Ultimate_Question_of_Life.2C_the_Universe.2C_and_Everything_.2842.29

Thanks again for for your comments thus far and if anyone else has experience of this issue it would be good to know...

kfuglsang · 2011-07-24

Scott, would you be so kind to provide us with an update if you and MS finds any solutions?

At the moment I don't have access to a test setup, but you could try these two things:

1) Changing the client:
You can create your own proxy around the generated webservice bindings.
For example like this class:

public class JobWebservice : Job_Binding
{
        public JobWebservice() : base() { }

        protected override System.Net.WebRequest GetWebRequest(Uri uri)
        {
            var request = base.GetWebRequest(uri);
            
            if (request is HttpWebRequest)
            {
                (request as HttpWebRequest).AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
            }
            
            ServicePointManager.Expect100Continue = false;
            ServicePointManager.UseNagleAlgorithm = false;

            var servicePoint = ServicePointManager.FindServicePoint(uri);
            servicePoint.Expect100Continue = false;
            servicePoint.UseNagleAlgorithm = false;

            return request;
        }
    }

2) Changing the server:
In the Microsoft.Dynamics.NAV.Server.exe.config file, you could try to add the following

<system.net>
  <settings>
    <servicePointManager expect100Continue="false" useNagleAlgorithm="false"/>
  </settings>
</system.net>

Best Regards
Kenneth

ScottJones · 2011-07-25

Kenneth,

Editing the .config file and restarting the middle tier did not make a difference BUT settiing the two properties on the ServicePointManager prior to the first connection worked \:D/ !!!!!

No news from Microsoft as yet, I was told it's holiday time and not to expect a quick response... We've currently edited all Webservice calls to add two dummy text parameters of 700 characters and changed all Reads to ReadMultiple so that we can pass a filter string that is repeated many times so that it increases the packet size to over 1460 characters!! It will take us some time to change all these back and test but excellent news as it works on the test harness!

Thank you again

kfuglsang · 2011-07-25

Excellent to hear that it worked for you!

I will make sure to try and implement it in one of our test environments when I am back in the office in a few days, to verify that it also works there.

If only I had a blog.... :-)

Houmark · 2013-10-08

Hi Kenneth

I know this thread has been inactive for some time, but we found good use of your advise regarding disabling the Nagle algorithm.
We are getting data fore our webshop from a NAV2013 and experienced some slow response times from the web services.

So we started to investigate on the Nagle issue and got it to work thanks to your super advice.

Our test code just calls "DisableNagleAlgorithm" and voila - the response is now down to 25-35 ms instead of 150-170ms

		public static NavSoap_PostCodes.PostCodes[] GetPostCodesSoap()
		{
			DisableNagleAlgorithm();
			List<PostCodes_Filter> filters = new List<PostCodes_Filter>();
			var filter = new PostCodes_Filter();
			filter.Field = PostCodes_Fields.Code;
			filter.Criteria = "9900";
			filters.Add(filter);

			var postcodes = PostCodesService.ReadMultiple(filters.ToArray(), null, Int32.MaxValue);
			return postcodes;
		}

		private static void DisableNagleAlgorithm()
		{
			System.Net.ServicePointManager.Expect100Continue = false;
			System.Net.ServicePointManager.UseNagleAlgorithm = false;
		}

But we still have some issues:
- the first call to the web service is still very slow, like 700 ms. And if the web server is inactive for some time the high "first call" response time is seen again.
- the other issue we have is when we request data from a oData web service (using a NAV 2013 page object). We do this using LINQ, but can't seem to get to disable the Nagle algorithm.

If there is anyone who have solved these issue or just have some ideas on how to fix this, we'll be very glad to get some feedback.

Best regards,
Allan Houmark Persson

kfuglsang · 2013-10-08

Hi Allan

Great to know that you had use of it :-)

I have not yet tried to work with the ODATA services, but I have had experience with the other issue. Quite simply, this happens to all .NET-based websites in the process of saving resources on the webserver.
Microsoft created a tool for IIS that can be used to solve the issue. Depending on which version of Windows Server you are running the tool has different names. The latest is IIS Auto-Start. Read more about it here: https://www.simple-talk.com/blogs/2013/ ... t-feature/

The NAV webservice is affected by the same issue, but the MS tool cannot be used outside of IIS. If your website calls the NAV webservice on any request the IIS Auto-Start will indirectly solve this as well. If not, you can create a simple scheduled job to keep the NAV webservice alive.

Hope that helps.

Howdy, Stranger!

Quick Links

Categories

NAV2009R2 Webservice Performance Issues - Multiple calls

Comments