NAV 2009 RTC + Add-In problem: utf-8 encoded chars mixed-up

staub

Hello,
I create Add-In and configure to use in Navision like in the blog: "Add-ins for the RoleTailored client of Dynamics NAV 2009 SP1 (part2)"
http://blogs.msdn.com/b/cabeln/archive/ ... part2.aspx
I send string from BigText, data prepared by XMLport with UTF-8. After in add-in received data going to save in a file, now that has specific chars mixed-up:
(source char ī (c4 ab) became (c2 b8)+(c2 bd) )
What should I do to get correct charset?

Thank you,
Yuri

deV.ch

I don't know if it's exactly the same problem but i had errors with charsets and bigtext var too.
Here is my blogpost about it:
http://devch.wordpress.com/2011/12/22/clean-workaround-for-special-characters-in-bigtext-variables/

you need 2009 R2 to get it work!

ppavuk

You can encode your string on .NET side to Base64, and decode on NAV side. You can use standard xmldom methods to encode\decode Base64 string. The nature of problem is quite simple - NAV2009 does not support unicode.

BTW - say hello to Elva guys from me:)

edited:
Ok, didn't got, you sending a string from NAV - even simplier. Get rid of bigtext - use xmldom, and send entrie xmldom variable to your .NET application. So, stream out data from xml port, load to xmldom, send to .NET.

staub

Thank you Peter (ppavuk),

It is good to use xmldom object. But I choose other way, because of: no time to test new solution plus minimize changes on .Net part.
So, I use HTMLdecode/encode technique. It is good that I have nonASCII chars only at one direction from Nav to DLL. Algorithm is:
1. string stored in BigText I encode
2. then in .NET dll I decode received text
3. then general XML processing left the same – that mean in c# code only one additional statement: _xml = System.Web.HttpUtility.HtmlDecode(value);

The biggest problem is on Nav side, because as it was mentioned before by ppavuk: Navision works with ANSI charset - not utf-8. So it is challenge to encode utf-8 text (need to know the structure of that unicode), hereby in attachment is code for those who are interested.

P.S. special HELLO to Peter from Raimond.

UTF8ToHTML(VAR BTextSrc : BigText;VAR BTextDest : BigText) : Integer
ParseStrLen := MAXSTRLEN(ParseStr);
DestStrLen := ParseStrLen;
CLEAR(BTextDest);
BTextSrcLen := BTextSrc.LENGTH;
IF BTextSrcLen > 0 THEN BEGIN
  BTextSrcCurrPos := 1;
  WHILE BTextSrcCurrPos < BTextSrcLen DO BEGIN
    BTextSrc.GETSUBTEXT(ParseStr, BTextSrcCurrPos, ParseStrLen);
    ParseStrCurrLen := STRLEN(ParseStr);
    FOR I := 1 TO ParseStrCurrLen DO BEGIN
      CurrChar := ParseStr[I];
      DestStrPart := '';
      CASE CurrChar OF
        0..127: //ASCII
          BEGIN
            DestStrPart := COPYSTR(ParseStr, I, 1);
            BytesUsedForUnicode := 1;
            BytesUsedForUnicodeCurr := 0;
            CLEAR(ByteArr);
            //UnicodeStr := '';
          END;
        128..191: //LAST BYTE for unicode
          BEGIN
            //UnicodeStr += Char2Hex(CurrChar);
            BytesUsedForUnicodeCurr += 1;
            Char2Byte(CurrChar, ByteArr[BytesUsedForUnicodeCurr]);

            IF BytesUsedForUnicodeCurr >= BytesUsedForUnicode THEN BEGIN
              UnicodeInt := 0;
              FOR j := BytesUsedForUnicode DOWNTO 1 DO BEGIN
                ByteArr[1, 8 - (BytesUsedForUnicode-j)] := FALSE;
                ByteArr[j, 8] := FALSE;
                UnicodeInt += Byte2BigInt(ByteArr[j], ((BytesUsedForUnicode-j)*64));
              END;

              DestStrPart := '&#' + FORMAT(UnicodeInt) + ';';
            END;
            BytesUsedForUnicode := 1;
            BytesUsedForUnicodeCurr := 0;
            CLEAR(ByteArr);
            //DestStrPart := '&#x' + FORMAT(UnicodeStr) + ';';
            //UnicodeStr := '';
          END;
        ELSE
          BEGIN //first byte for unicode
            //UnicodeStr += Hex();
            CLEAR(ByteArr);
            BytesUsedForUnicodeCurr := 1;
            Char2Byte(CurrChar, ByteArr[BytesUsedForUnicodeCurr]);
            CASE CurrChar OF
              192..223:
                BytesUsedForUnicode := 2;
              224..239:
                BytesUsedForUnicode := 3;
              240..247:
                BytesUsedForUnicode := 4;
              248..251:
                BytesUsedForUnicode := 5;
              252..253:
                BytesUsedForUnicode := 6;
              ELSE
                BytesUsedForUnicode := 1;
            END;
          END;
      END;
      DestStrPartCurrLen := STRLEN(DestStrPart);
      IF CurrChar = '9' THEN
        CurrChar := '9';
      IF DestStrPartCurrLen > 0 THEN BEGIN
        IF (DestStrPartCurrLen + DestStrCurrLen) > DestStrLen THEN BEGIN
          BTextDest.ADDTEXT(DestStr);
          DestStr := DestStrPart;
          DestStrCurrLen := DestStrPartCurrLen;
        END ELSE BEGIN
          DestStr += DestStrPart;
          DestStrCurrLen += DestStrPartCurrLen;
        END;
      END;
    END;
    BTextSrcCurrPos += ParseStrLen;
  END;
  BTextDest.ADDTEXT(DestStr);
END ELSE
  EXIT(1);
EXIT(0);

Char2Byte(VAR CharPar : Char;VAR Byte : ARRAY [8] OF Boolean)
//1st bit (index 1) means 1st bit from the right
CharInt := CharPar;
Oper := 128;
FOR i := 8 DOWNTO 1 DO BEGIN
  IF CharInt >= Oper THEN BEGIN
    Byte[i] := TRUE;
    CharInt -= Oper;
  END;
  Oper := Oper DIV 2;
END;

Char2Hex(VAR CharPar : Char) ResultHex : Text[2]
//In Navision Char is coded by one byte so return value should be 2char only
Char2Byte(CharPar, Byte);
CLEAR(ByteHalf);
FOR i := 1 TO 4 DO
  ByteHalf[i] := Byte[i+4];
Int := Byte2Int(ByteHalf);
ResultHex := TinyInt2Hex(Int);
CLEAR(ByteHalf);
FOR i := 1 TO 4 DO
  ByteHalf[i] := Byte[i];
Int := Byte2Int(ByteHalf);
ResultHex += TinyInt2Hex(Int);

EXIT(ResultHex);

Byte2Int(Byte : ARRAY [8] OF Boolean) IntResult : Integer
//1st bit (index 0) means 1st bit from the right
IntResult := 0;
Oper := 1;
FOR i := 1 TO 8 DO BEGIN
  IF Byte[i] THEN
    IntResult += Oper;
  Oper := Oper*2;
END;
EXIT(IntResult);

Byte2BigInt(Byte : ARRAY [8] OF Boolean;Oper : BigInteger) IntResult : BigInteger
//1st bit (index 0) means 1st bit from the right
IntResult := 0;
IF Oper = 0 THEN
  Oper := 1;
FOR i := 1 TO 8 DO BEGIN
  IF Byte[i] THEN
    IntResult += Oper;
  Oper := Oper*2;
END;
EXIT(IntResult);

TinyInt2Hex(Int : Integer) Result : Text[1]
//Int supposed to be 0..15
CASE Int OF
  10:
    Result := 'A';
  11:
    Result := 'B';
  12:
    Result := 'C';
  13:
    Result := 'D';
  14:
    Result := 'E';
  15:
    Result := 'F';
  ELSE
    Result := FORMAT(Int);
END;

mdPartnerNL

thx of course for the code. Am I correct in how it works.

You only have a problem sending data to C#

NAV
1.
fillup Bigtext with text

2.
encode with UTF8ToHTML

3.
Send result back to webservice

C#
1.
decode to html

(Just edited the steps)

ppavuk

Why not to use 'bin.base64', which is just a 1 line of code? And of course you can do the same on your .NET application.


base64 obviosly do not contain any utf-8 char, and all encode\decode will be transparent, and moreover you can get rid of lots of code lines.

Let's have a look to pseudo-code below.

    encode := '';
    objXmlNode := objXmlDom.createElement("tmp");
    objXmlNode.datatype := "bin.base64";
    objXmlNode.nodeTypedvalue := 'your value';
    encode : = objXmlNode.Text;
    

    decode = ''
    objXmlNode := objXmlDom.createElement("tmp");
    objXmlNode.datatype := "bin.base64";
    objXmlNode.Text := base64Code;
    decode := objXmlNode.nodeTypedvalue;
    

Say hello to Raimonds too

mdPartnerNL

I like the idea but this way we need to iterate through all the nodes to encode it?

deV.ch

All these NAV convert functions are realy slow and don't work well with large data...
please read my blogpost (i already posted as 1. reply...) where i discribe how to use Dot Net StreamReader to solve the problem, it's simple & fast.

ppavuk

Actually streamreader is nice for r2, but will not work in older versions. So, xml for old versions, streamreader for R2 +

And yes, all text parsing/converting code in C/AL is extremely slow! We have to avoid this if possible.

ppavuk

mdPartnerNL wrote:

I like the idea but this way we need to iterate through all the nodes to encode it?

Actually not, you already iterating to create nodes. just add base64 property to node type. That's all.

mdPartnerNL

Ok, so to recap,

this problem does not occur in 2013

In NAV 2009 and earlier the problem only exist when sending data From NAV to ?

ppavuk

mdPartnerNL wrote:

Ok, so to recap,

this problem does not occur in 2013

In NAV 2009 and earlier the problem only exist when sending data From NAV to ?

to whatever external application, and reading from external apps.
In R2 conversion can be done by .NET stream reader,
In earliest versions - by XmlDom (base64)
There is also third way -

CREATE(ADOStreamUTF);
ADOStreamUTF.Open;
ADOStreamUTF.LoadFromFile(Filename);
ADOStreamUTF.LineSeparator := -1;
ADOStreamUTF.Charset := 'UTF-8';

CREATE(ADOStreamANSI);
ADOStreamANSI.Open;
ADOStreamANSI.Type := 2;
ADOStreamANSI.LineSeparator := -1;
ADOStreamANSI.Charset := 'x-ansi';


StreamLen := ADOStreamUTF.Size;
WHILE NOT ADOStreamUTF.EOS DO BEGIN
  ADOStreamANSI.WriteText(ADOStreamUTF.ReadText(1000));
END;
ADOStreamUTF.Close;
ADOStreamANSI.SaveToFile(TempFilename, 2);
ADOStreamANSI.Close;

Maybe there is more ways, but i think 3 is enough
All this make sense if your locale is not US or not UK (or not whatever English)

GoMaD

Just to clarify:

ADOStreamUTF and ADOStreamANSI are no dotNET variables but Automation variables linked to 'Microsoft ActiveX Data Objects 6.0 Library'.Stream'

And to get a working verion I also had to change the sequence of statements:

CLEAR(ADOStreamUTF);
CREATE(ADOStreamUTF);
ADOStreamUTF.Charset := 'UTF-8';
ADOStreamUTF.Open;
ADOStreamUTF.LoadFromFile(FilenameIn);
ADOStreamUTF.LineSeparator := -1;

CLEAR(ADOStreamANSI);
CREATE(ADOStreamANSI);
ADOStreamANSI.Charset := 'x-ansi';
ADOStreamANSI.Open;
ADOStreamANSI.Type := 2;
ADOStreamANSI.LineSeparator := -1;

lintStreamLen := ADOStreamUTF.Size;
WHILE NOT ADOStreamUTF.EOS DO BEGIN
  ADOStreamANSI.WriteText(ADOStreamUTF.ReadText(1000));
END;
ADOStreamUTF.Close;
ADOStreamANSI.SaveToFile(FilenameOut, 2);
ADOStreamANSI.Close;