xmlPort and Webservice performance & encoding problems

deV.ch

Hi there i have a problem with a webservice. This webservice produces a lot of data (500kb in plain xml-text is normal).

At the first place we uesed the xmlport as a parameter in the webservice Function, but we noticed that with larger calls the response time gets realy slow (ca 1 minute).

Then i started to test a bit around and i manged to reduce the repsonse time to 30 sec. by using a bigtext var as the reference and expose the xml as text.

but here's the problem. we have special characters like ä ü ö and when this text is readed in the bigText var it's all messed up!
why is this? when i stream the same xmlport-data in a stream it's totaly right, encoding is correct, and the special chars are correct. but as soon as i load this in the bigtext var it gets messed up.

the only way i got it to work was when i use UTF-16 encoding in the xmlport and then read trough the whole stream and convert all with the ansi2ascii method! this not realy makes sense, because when i use utf-8 as encoding in and use the same function, it gets messed again.

and the big problem with the utf-16 and convert from ansi2ascii -thing is that its not performant! with this method i end up with the same slow webservice method (ca. 60sec).

any suggestions?

jwilder

I would stick with UTF-8, I had all found UTF-16 would add some weird characters at the end of the bigtext var. Can you post your NAV code that is called by the web service?

Here is some of my code that does what you are trying to do (I have used for our website integration using NAV 2009 SPI) but we generally do not have special characters so this may be no help:
Name DataType Subtype
StreamOut OutStream
TempBlob Record TempBlob (temporary)
Customer Record Customer
CustComment Record Comment Line
CalledFromWebService Boolean (This is true in the below code)

IF Customer.FINDFIRST THEN BEGIN
IF CalledFromWebService THEN
TempBlob.Blob.CREATEOUTSTREAM(StreamOut) // Create OutStream for XMLPort
CustComment.SETRANGE("Table Name",CustComment."Table Name"::Customer);
CustComment.SETRANGE("No.",Customer."No.");
XMLPORT.EXPORT(XMLPORT::"Customer Comment Read",StreamOut,CustComment); //Make sure XMLPort is UTF-8
END;

This streams the XMLPort to a Temp Blob which I then stream to a bigtext variable that gets passed back through web service.

deV.ch

What i do is exactly the same. i stream into the tempblob table and then into a bigtext var. I can stream out to a file and then stream into the bigtextvar too. but that results in the same problem. Special chars get messed up.

TempBlob.INIT;
TempBlob.Blob.CREATEOUTSTREAM(OStream);
xPort.SETDESTINATION(OStream);
xPort.Export;
IF NOT ISSERVICETIER THEN
  TempBlob.Blob.EXPORT('',TRUE); // Export Text as File for Testing
TempBlob.Blob.CREATEINSTREAM(IStream);
OutputXML.READ(IStream);

As i said with some code that Reads trough the stream and converts it from ansii2ascii (just works with UTF-16) all is fine but slow, here's the code i used and below you see some benchmark results of this...

Counter := 0;
REPEAT
  Counter += 1;
  IStream.READTEXT(TempText,MAXSTRLEN (TempText));
  TempText := GeneralMgt.Ansi2Ascii(TempText);

  // Search Beginning (UTF-16 related problem)
  IF Counter = 1 THEN
  TempText := COPYSTR(TempText,STRPOS(TempText,'<'));

  OutputXML.ADDTEXT(TempText,OutputXML.LENGTH+1);
UNTIL IStream.EOS;

I did another test jsut to show the difference of ServiceTier / Classic Speed Difference:
Starting the Function in classic client by RUN: 3sec
Startint the Function in RTC Page by action: 53sec

This is insane!

deV.ch

I managed to speed everything up. i know have a reaction time of 10sec on the WS (before i had 55-60)

the first problem was the keys on the xmlport. They weren't for best performacne. The weird thing is that this had no impact on classic client. which doesn't make sense to me. but ok, not the first strange behavior in NAV

the second and important thing is the way i write in the BigText var. Here is my code:

Counter := 0;
  REPEAT
    Counter += 1;
    TotalLen := 0;
    CLEAR(TempText);
    REPEAT
      ActLen := IStream.READ(TempText2,MAXSTRLEN(TempText));
      IF ActLen <> 0 THEN BEGIN
        TotalLen += ActLen;
        TempText += TempText2;
      END;
    UNTIL (ActLen = 0) OR (TotalLen = MAXSTRLEN(TempText));
    TempText := GeneralMgt.Ansi2Ascii(TempText);

    // Search Beginning (UTF-16 related)
    IF Counter = 1 THEN
      TempText := COPYSTR(TempText,STRPOS(TempText,'<'));

    OutputXML.ADDTEXT(TempText,OutputXML.LENGTH+1);
  UNTIL (ActLen = 0);

For understanding: if you choose the UTF-16 format (which you have to choose if you work with special characters), the read methods for the stream (READ & READTEXT) only return one character each time you read. To speed up everything you should add up to 1024 chars for converting and adding in the bigtext var.

There are some problems with the begining of the text too. to solve this you nedd to search the first '<' char.

This is a fast way to load the xml in a bigtext var.


For my solution i will not use the bigtext solution. because i was able to reduce the reaction time of the variant with the xmlport as parameter (ref) even more than then the bigtext one. I wasn't aware of the fact that you don't need to call the EXPORT Function! This cuts reaction time en half!

I hope this is written clearly so anybody running in an similar problem finds help!

eknraw

I ran into something similar with characters like: ä ü ö in XML. I also tried using a similar function to parse through text and convert characters using the GeneralMgt codeunit. I found this to be quite slow on large messages and it didn't convert/format every character correctly.

I use exposed bigtext variables to pass xml strings back and forth between the NST and a website. When I receive XML with those characters things wouldn't import correctly and I don't think the DOM object xmldomdoc.load() would even work. What I ended up doing was using an ADO stream automation to write the XML string to a temporary text file in UTF-8 format which I then imported using tempblob.blob.import which I then loaded into a XML DOM object. This was faster for me and I didn't have any formatting issues. However, those extended characters are the exception (very small amount) in my case so usually I'm able to pass streams and text around without having to write to disk.

lblobTemp.Blob.CREATEOUTSTREAM(OutStream);
bigtext.WRITE(OutStream);

lblobTemp.CALCFIELDS(Blob);
lblobTemp.Blob.CREATEINSTREAM(InStream);

CREATE(adoOut);
adoOut.Type := 2;
adoOut.Open;
adoOut.Charset('UTF-8');
WHILE NOT InStream.EOS DO BEGIN
  Instream.READTEXT(Text1024);
  adoOut.WriteText(Text1024);
END;
adoOut.SaveToFile(TEMPORARYPATH + ltxtFileName,2);
lblobTemp.Blob.IMPORT(TEMPORARYPATH + ltxtFileName);

This was one of those problems where the above worked for me and there was a time crunch, but I should probably revisit because this just seems like something that could be handled better.

This might help too: http://blogs.msdn.com/b/freddyk/archive/2008/11/10/nav-2009-and-unicode.aspx

Bill

deV.ch

Nice hint! Unfortunatly this doesn't work in webservices because i can only expose bigtext or the xmlport itself. My suggestion would be (for webservices) to stick with the exposed xmlport and if you have performance issues, try to tweak the port. You can do much wrong, which probably is fast on classic but not an NST...

i haven't tried it but as i understand with the link of freddy you could convert the chars in the consuming .NET application. If the code i posted is to slow this would be a nice solution, thanks! :thumbsup:

GoMaD

Following C/AL code in NAV 2009 R2 solved the special (Unicode) characters problem for me:

PROCEDURE fctConvertBigTextEncoding@1100084010(VAR pbtxXMLFileIn@1100084000 : BigText;VAR pbtxXMLFileOut@1100084001 : BigText);
    VAR
      ByteUTF8@1100084002 : DotNet "'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Array";
      ByteUTF16@1100084003 : DotNet "'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Array";
      EncodingUTF16@1100084005 : DotNet "'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Text.UnicodeEncoding";
      EncodingConvert@1100084006 : DotNet "'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Text.Encoding";
    BEGIN
      EncodingUTF16 := EncodingUTF16.UnicodeEncoding;

      ByteUTF16 := EncodingUTF16.GetBytes(pbtxXMLFileIn);
      ByteUTF8 := EncodingConvert.Convert(EncodingConvert.Unicode,EncodingConvert.UTF8,ByteUTF16);

      CLEAR(pbtxXMLFileOut);
      pbtxXMLFileOut.ADDTEXT(EncodingConvert.Default.GetString(ByteUTF8));
    END;

ps.: Unicode = UTF-16