xmlPort and Webservice performance & encoding problems

Answers

• jwilder Member Posts: 263

  2010-12-30

  I would stick with UTF-8, I had all found UTF-16 would add some weird characters at the end of the bigtext var. Can you post your NAV code that is called by the web service?
  
  Here is some of my code that does what you are trying to do (I have used for our website integration using NAV 2009 SPI) but we generally do not have special characters so this may be no help:
  Name DataType Subtype
  StreamOut OutStream
  TempBlob Record TempBlob (temporary)
  Customer Record Customer
  CustComment Record Comment Line
  CalledFromWebService Boolean (This is true in the below code)
  
  IF Customer.FINDFIRST THEN BEGIN
  IF CalledFromWebService THEN
  TempBlob.Blob.CREATEOUTSTREAM(StreamOut) // Create OutStream for XMLPort
  CustComment.SETRANGE("Table Name",CustComment."Table Name"::Customer);
  CustComment.SETRANGE("No.",Customer."No.");
  XMLPORT.EXPORT(XMLPORT::"Customer Comment Read",StreamOut,CustComment); //Make sure XMLPort is UTF-8
  END;
  
  This streams the XMLPort to a Temp Blob which I then stream to a bigtext variable that gets passed back through web service.

  Jason Wilder
  jwilder@stonewallkitchen.com

  0
• deV.ch Member Posts: 543

  2011-01-03

  What i do is exactly the same. i stream into the tempblob table and then into a bigtext var. I can stream out to a file and then stream into the bigtextvar too. but that results in the same problem. Special chars get messed up.
  

  TempBlob.INIT;
  TempBlob.Blob.CREATEOUTSTREAM(OStream);
  xPort.SETDESTINATION(OStream);
  xPort.Export;
  IF NOT ISSERVICETIER THEN
    TempBlob.Blob.EXPORT('',TRUE); // Export Text as File for Testing
  TempBlob.Blob.CREATEINSTREAM(IStream);
  OutputXML.READ(IStream);
  

  
  As i said with some code that Reads trough the stream and converts it from ansii2ascii (just works with UTF-16) all is fine but slow, here's the code i used and below you see some benchmark results of this...

  Counter := 0;
  REPEAT
    Counter += 1;
    IStream.READTEXT(TempText,MAXSTRLEN (TempText));
    TempText := GeneralMgt.Ansi2Ascii(TempText);
  
    // Search Beginning (UTF-16 related problem)
    IF Counter = 1 THEN
    TempText := COPYSTR(TempText,STRPOS(TempText,'<'));
  
    OutputXML.ADDTEXT(TempText,OutputXML.LENGTH+1);
  UNTIL IStream.EOS;
  

  
  I did another test jsut to show the difference of ServiceTier / Classic Speed Difference:
  Starting the Function in classic client by RUN: 3sec
  Startint the Function in RTC Page by action: 53sec
  
  This is insane!

  My Blog

  0
• deV.ch Member Posts: 543

  2011-01-03

  I managed to speed everything up. i know have a reaction time of 10sec on the WS (before i had 55-60)
  
  the first problem was the keys on the xmlport. They weren't for best performacne. The weird thing is that this had no impact on classic client. which doesn't make sense to me. but ok, not the first strange behavior in NAV
  
  the second and important thing is the way i write in the BigText var. Here is my code:
  

  Counter := 0;
    REPEAT
      Counter += 1;
      TotalLen := 0;
      CLEAR(TempText);
      REPEAT
        ActLen := IStream.READ(TempText2,MAXSTRLEN(TempText));
        IF ActLen <> 0 THEN BEGIN
          TotalLen += ActLen;
          TempText += TempText2;
        END;
      UNTIL (ActLen = 0) OR (TotalLen = MAXSTRLEN(TempText));
      TempText := GeneralMgt.Ansi2Ascii(TempText);
  
      // Search Beginning (UTF-16 related)
      IF Counter = 1 THEN
        TempText := COPYSTR(TempText,STRPOS(TempText,'<'));
  
      OutputXML.ADDTEXT(TempText,OutputXML.LENGTH+1);
    UNTIL (ActLen = 0);
  

  
  For understanding: if you choose the UTF-16 format (which you have to choose if you work with special characters), the read methods for the stream (READ & READTEXT) only return one character each time you read. To speed up everything you should add up to 1024 chars for converting and adding in the bigtext var.
  
  There are some problems with the begining of the text too. to solve this you nedd to search the first '<' char.
  
  This is a fast way to load the xml in a bigtext var.
  
  
  For my solution i will not use the bigtext solution. because i was able to reduce the reaction time of the variant with the xmlport as parameter (ref) even more than then the bigtext one. I wasn't aware of the fact that you don't need to call the EXPORT Function! This cuts reaction time en half!
  
  I hope this is written clearly so anybody running in an similar problem finds help!

  My Blog

  0
• eknraw Member Posts: 26

  2011-01-03

  I ran into something similar with characters like: ä ü ö in XML. I also tried using a similar function to parse through text and convert characters using the GeneralMgt codeunit. I found this to be quite slow on large messages and it didn't convert/format every character correctly.
  
  I use exposed bigtext variables to pass xml strings back and forth between the NST and a website. When I receive XML with those characters things wouldn't import correctly and I don't think the DOM object xmldomdoc.load() would even work. What I ended up doing was using an ADO stream automation to write the XML string to a temporary text file in UTF-8 format which I then imported using tempblob.blob.import which I then loaded into a XML DOM object. This was faster for me and I didn't have any formatting issues. However, those extended characters are the exception (very small amount) in my case so usually I'm able to pass streams and text around without having to write to disk.
  

  lblobTemp.Blob.CREATEOUTSTREAM(OutStream);
  bigtext.WRITE(OutStream);
  
  lblobTemp.CALCFIELDS(Blob);
  lblobTemp.Blob.CREATEINSTREAM(InStream);
  
  CREATE(adoOut);
  adoOut.Type := 2;
  adoOut.Open;
  adoOut.Charset('UTF-8');
  WHILE NOT InStream.EOS DO BEGIN
    Instream.READTEXT(Text1024);
    adoOut.WriteText(Text1024);
  END;
  adoOut.SaveToFile(TEMPORARYPATH + ltxtFileName,2);
  lblobTemp.Blob.IMPORT(TEMPORARYPATH + ltxtFileName);
  

  
  This was one of those problems where the above worked for me and there was a time crunch, but I should probably revisit because this just seems like something that could be handled better.
  
  This might help too: http://blogs.msdn.com/b/freddyk/archive/2008/11/10/nav-2009-and-unicode.aspx
  
  Bill

  0
• deV.ch Member Posts: 543

  2011-01-04

  Nice hint! Unfortunatly this doesn't work in webservices because i can only expose bigtext or the xmlport itself. My suggestion would be (for webservices) to stick with the exposed xmlport and if you have performance issues, try to tweak the port. You can do much wrong, which probably is fast on classic but not an NST...
  
  i haven't tried it but as i understand with the link of freddy you could convert the chars in the consuming .NET application. If the code i posted is to slow this would be a nice solution, thanks! :thumbsup:

  My Blog

  0
• GoMaD Member Posts: 313

  2013-10-07

  Following C/AL code in NAV 2009 R2 solved the special (Unicode) characters problem for me:
  

  PROCEDURE fctConvertBigTextEncoding@1100084010(VAR pbtxXMLFileIn@1100084000 : BigText;VAR pbtxXMLFileOut@1100084001 : BigText);
      VAR
        ByteUTF8@1100084002 : DotNet "'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Array";
        ByteUTF16@1100084003 : DotNet "'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Array";
        EncodingUTF16@1100084005 : DotNet "'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Text.UnicodeEncoding";
        EncodingConvert@1100084006 : DotNet "'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Text.Encoding";
      BEGIN
        EncodingUTF16 := EncodingUTF16.UnicodeEncoding;
  
        ByteUTF16 := EncodingUTF16.GetBytes(pbtxXMLFileIn);
        ByteUTF8 := EncodingConvert.Convert(EncodingConvert.Unicode,EncodingConvert.UTF8,ByteUTF16);
  
        CLEAR(pbtxXMLFileOut);
        pbtxXMLFileOut.ADDTEXT(EncodingConvert.Default.GetString(ByteUTF8));
      END;
  

  
  ps.: Unicode = UTF-16

  Now, let's see what we can see.
  ...
  Everybody on-line.
  ...
  Looking good!

  0