Encoding of UTF-8 vs UTF8-BOM

jwilder

We have just gone live with NAV 2016 and we are creating a bunch of xml files using DotNet XMLDoc (XMLDoc := XMLDoc.XmlDocument;). Then I am using the XMLDocMgt codeunit to add nodes etc... In the end I am calling XMLDoc.Save to save the document which then is ftp'd to our website.

The problem is that these files encode in UTF-8-BOM not UTF-8. My declaration for all of these files is: <?xml version="1.0" encoding="UTF-8"?>. Turns out UTF-8-BOM doesn't work so well with our website. How can I force this to be UTF-8 and why is it choosing UTF-8-BOM?

One fix might be to change the encoding after the fact but not sure how to do that yet.

ara3n

You need to save the file the following way.


XmlWriterSettings := XmlWriterSettings.XmlWriterSettings();
Encoding := Encoding.UTF8Encoding(FALSE);
XmlWriterSettings.Encoding := Encoding;
XmlWriterSettings.Indent := TRUE;

XmlWriter := XmlWriter.Create(TEMPORARYPATH+ 'XMLFile.xml',XmlWriterSettings);
xmlDoc.Save(XmlWriter);
XmlWriter.Close();

jwilder

Thats what I was looking for, thanks!

Also found that I can open the file, stream to text variable and then write back to a new file. The trick is that when calling File.open before reading the file to set the encoding like this:
File.OPEN('MyFile.xml',TEXTENCODING::UTF8);

jwilder

A little more information. This isn't a NAV problem but an XMLDom problem. When you run XMLDoc.Save it saves as UTF-8-BOM. By the way XMLPorts save correctly as UTF-8 if that is what you specify in the Encoding property.

So the question still remains how to change a file encoded in UTF-8-BOM to UTF-8 in NAV 2016.

ara3n

You need to save the file the following way.


XmlWriterSettings := XmlWriterSettings.XmlWriterSettings();
Encoding := Encoding.UTF8Encoding(FALSE);
XmlWriterSettings.Encoding := Encoding;
XmlWriterSettings.Indent := TRUE;

XmlWriter := XmlWriter.Create(TEMPORARYPATH+ 'XMLFile.xml',XmlWriterSettings);
xmlDoc.Save(XmlWriter);
XmlWriter.Close();

jwilder

Thats what I was looking for, thanks!

Also found that I can open the file, stream to text variable and then write back to a new file. The trick is that when calling File.open before reading the file to set the encoding like this:
File.OPEN('MyFile.xml',TEXTENCODING::UTF8);

einsTeIn.NET

Facing the same issue currently. Did you get any further?

Miklos_Hollender

jwilder wrote: »

By the way XMLPorts save correctly as UTF-8 if that is what you specify in the Encoding property.

Well, that's not true. I ran into that kind of brick wall right now, NAV2015, created a CSV file with an XMLPort and the receiving party insists on UTF8NoBOM encoding, and it turns out in NAV UTF-8 is UTF8BOM. You don't see it in Notepad, but if you go to the directory where the file is saved, type "cmd" in the top line so that the command prompt opens in the right directory and then type "type filename.txt" you can see the first three characters are weird. That is the Byte Order Mark.

After hitting my head against this brick wall first in C/AL and then in PowerShell to no avail (PowerShell could work, but I want a one-liner I can call from NAV), I will probably have to look into messing about .NET calls. Does anyone have working code for stripping the first three bytes from a file?

Miklos_Hollender

I would like to add that doing it entire manually works, as in, no Byte Order Mark:

FileVar.CREATE('whatev.txt', TEXTENCODING::UTF8);
FileVar.WRITE('I häte this ßtüff');

(Plus the usual writemode,textmode, close stuff.)
This works, and the characters are correct UTF-8 right in Notepad, and going to the command prompt and doing a "type whatev.txt" shows there is no BOM. Excel does not show the characters right but I don't care about that.

Still, this is the worst way to make a CSV file so I will rather now look into if the CSV Buffer is doing it right.

EDIT: D*mn, no CSV Buffer in NAV 2015. Sigh.