System.Xml.XmlWriter Memory Footprint

toor

Hi there,

for writing large xml files I want to use System.Xml.XmlWriter.
So I wrote a small test codeunit to check if it works well from within NAV.

The resulting xml file looks like expected, but the NST grabs some hundred MBs auf memory when running the code and it doesn't release the memory after completing the code (or closing the client).

When running the "same" code in C# the program grabs less than 5 MB of memory.

What the hell is NAV doing here?
Any ideas on how to avoid the huge memory consumption?
I've seen the same behavior when using XmlReader a while ago.

I tested with NAV 2013 R2 CU11.

The NAV code:

xmlWriterSettings := xmlWriterSettings.XmlWriterSettings();
xmlWriterSettings.Indent := TRUE;

xmlWriter := xmlWriter.Create('c:\temp\nav.xml', xmlWriterSettings);
xmlWriter.WriteStartElement('Items', 'http://mynamspace');

FOR i := 1 TO 200000 DO BEGIN

  xmlWriter.WriteStartElement('InventoryRecord');

  xmlWriter.WriteElementString('ItemNo', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo2', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo3', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo4', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo5', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo6', 'ITM000001');

  xmlWriter.WriteEndElement();

END;

xmlWriter.WriteEndElement();
xmlWriter.Flush();

xmlWriter.Close();

The C# code:

using System;
using System.Xml;

namespace ConsoleApplication4
{
    class Program
    {
        static void Main(string[] args)
        {
            XmlWriterSettings settings = new XmlWriterSettings();
            settings.Indent = true;

            using (XmlWriter writer = XmlWriter.Create(@"c:\temp\inventory.xml", settings))
            {
                writer.WriteStartElement("Items", "http://mynamspace");
                for (int i = 0; i < 200000; i++)
                {
                    writer.WriteStartElement("InventoryRecord");
                    writer.WriteElementString("ItemNo", "ITM000001");
                    writer.WriteElementString("ItemNo2", "ITM000001");
                    writer.WriteElementString("ItemNo3", "ITM000001");
                    writer.WriteElementString("ItemNo4", "ITM000001");
                    writer.WriteElementString("ItemNo5", "ITM000001");
                    writer.WriteElementString("ItemNo6", "ITM000001");
                    writer.WriteEndElement();
                }
                writer.WriteEndElement();
                writer.Flush();
            }

            Console.WriteLine("done");
            Console.ReadLine();

        }
    }
}

thanks & best regards
Tobias

CorinaS

Hi Tobias,

First of all, thank you for posting your feedback!
We are sorry to hear you ran into this issue and feel this way.

I understand that you have also logged a case with us in this regards, and my colleague is currently working on it.
His feedback to me was that he was able to reproduce and is in the process of creating a CR with the DEV team to check if this is a problem.

We will contact you back via our support case as soon as we have updates, but please do let me know if there is anything more I can assist with at this point.

Thanks!

Vjeko

I don't have time to verify my claims here, but the problem is that your C# and C/AL code are not exactly equivalent. In C#, you put your XmlWriter into a using block, and the using block is automatically calling the Dispose method of the IDisposable interface. In your C/AL code, you never did that. That's why you end up having a lot of unmanaged resources (in other words - a memory leak) even after your session closes.

Furthermore, the XmlWriter class allocates the unmanaged resources, making it mandatory to clean them up, using the Dispose method.

A quick fix could be to make the XmlWriter a local variable in a function, and do all the calls locally. If Microsoft documentation is still valid, they will call the Dispose method for you if a variable is an IDisposable. This was implicit in .NET Interop in NAV 2009 R2, when all local DotNet variables were used in a using block in C#. In NAV 2013 it's a bit different, so maybe this call to Dispose happens somewhere else, explicitly, I can't check that.

Another quick fix would be to call XmlWriter.Dispose immediately after you call XmlWriter.Close - that will clean up the resources. However, calling Dispose directly can have unexpected side effects, because NAV might be calling it for you if it is a local variable - check the first solution, then the second one.

I hope this helps.

toor

What I've got from MS:

Reasons lay in situation that c\al calls in the loop .net method.
[...]
so when we have big loops, we have situation when memory is not released inside loop and increase with every step
[...]
Developer proposed workaround to put method WriteElementString to separated local procedure, then “out of scope” comes exactly when NAV goes out of procedure.

With this information I've created the two functions below.
Memory consumption is now pretty low, CPU consumption is higher (because the XmlWriter will be created and disposed very often, I think), but that's not a problem in my case.

In the first function I initialize a global stream variable and write the "xml-frame" with a global XmlWriter.
Inside the loop I call the second function.
The second function creates a local XmlWriter an writes some records to the global stream.
When the second function finishes, the "local resources" will be released.

It's weird, to write to the same stream from multiple XmlWriters, and you can end up with invalid xml pretty fast, but as long as it works I'm fine with this.

WriteXml()
Ns := 'http://mynamespace';
Stream := IoFile.Create('c:\temp\test.xml');

XmlWriterSettings := XmlWriterSettings.XmlWriterSettings();
XmlWriterSettings.CloseOutput := TRUE;
XmlWriterSettings.OmitXmlDeclaration := FALSE;
XmlWriterSettings.Encoding := Enc.UTF8;

XmlWriter := XmlWriter.Create(Stream, XmlWriterSettings);
XmlWriter.WriteStartElement('InventoryRecords', Ns);
XmlWriter.Flush();

XmlWriterSettings.CloseOutput := FALSE;
XmlWriterSettings.OmitXmlDeclaration := TRUE;
XmlWriterSettings.ConformanceLevel := ConformanceLevel.Fragment;

FOR i := 1 TO 5000 DO BEGIN
  WriteRecords(i=1, 500);
END;

XmlWriter.WriteEndElement();
XmlWriter.Flush();
XmlWriter.Close();

Stream.Close();
Stream.Dispose();

WriteRecords(useGlobalWriter : Boolean;numOfRecsToWrite : Integer)
IF NOT useGlobalWriter THEN BEGIN
  xmlWriterLoc := xmlWriterLoc.Create(Stream, XmlWriterSettings);
END ELSE BEGIN
  xmlWriterLoc := XmlWriter;
END;

FOR i := 1 TO numOfRecsToWrite DO BEGIN

  xmlWriterLoc.WriteStartElement('InventoryRecord');

  xmlWriterLoc.WriteElementString('ItemNo', 'ITM000001');
  xmlWriterLoc.WriteElementString('ItemNo2', 'ITM000001');
  xmlWriterLoc.WriteElementString('ItemNo3', 'ITM000001');
  xmlWriterLoc.WriteElementString('ItemNo4', 'ITM000001');
  xmlWriterLoc.WriteElementString('ItemNo5', 'ITM000001');
  xmlWriterLoc.WriteElementString('ItemNo6', 'ITM000001');

  xmlWriterLoc.WriteEndElement();

END;

xmlWriterLoc.Flush();

IF NOT useGlobalWriter THEN BEGIN
  xmlWriterLoc.Close();
  CLEAR(xmlWriterLoc);
END;

mdPartnerNL

so you keep on using 1 global var?

toor

I don't understand you question, sorry.

I'm using one global stream variable, one global XmlWriter and a lot of local XmlWriter.
All the XmlWriter write to the global stream.

Because the local XmlWriter goes out of scope everytime the function ends, there happen some kind of memory cleanup.

mdPartnerNL

Ok, now I understand, thx.