FieldRef suddently slower after about 1000 recordrefs - HELP

henrikohm

Hi

I'm using a fieldref to write info into a recordref like this:

For every do a lot of times (130.000) approx.
    record.init;
    for every field in an array list arrFieldNo
         field := record.FIELD(arrFieldNo[i]);
         do something.....codecodecode....
    record.insert;

The problem is that when the outer loop is from 1..1000 approx.
the

   field := record.FIELD(arrFieldNo[i]);

Is quite fast, but after that getting the field reference process-time
increases exponentially???!!!

From 1..1000 records is approx. 10 seconds, but from 1000..1300 it is approx. 20 seconds more.....!!!

Can anyone help me or has the same problem?

Best regards

Henrik Ohm

kine

I think that problem is your Array variable. Try to use some temporary table to save the field nos. Array variable must be saved into your RAM and if you don't have enough physical RAM, OS must swap etc...

pduck

i think your windows temporary file "pagefile.sys" will also become very large during this process and you can't free memory out of navision

fb

Here's a few more random thoughts:

1) Prior to v3.70, there was a problem with RecordRefs/FieldRefs declared as Globals. Is your 'field' Global or Local?

2) First 1000 records insert at 100recs/sec, next 300 at 10 recs/sec. Is it possible that this is 'normal' in the sense that the first 1000 recs simply fill Commit Cache, but from then on inserts are limited by a disk i/o bottleneck?

3) If the reuse of 'field' is somehow leaking memory, how about trying to CLEAR(field) between each use...

henrikohm

Makes sense. Actually I have it all in a complex table, but in order not to confuse things...

kine wrote:

I think that problem is your Array variable. Try to use some temporary table to save the field nos. Array variable must be saved into your RAM and if you don't have enough physical RAM, OS must swap etc...

henrikohm

Yes, that's a "minor" thing about Navision, but if I break the process and run it again, same thing happens: 1..1000 is fast and then past 1000 records it gets VEERY slow...

pduck wrote:

i think your windows temporary file "pagefile.sys" will also become very large during this process and you can't free memory out of navision

henrikohm

*1 : My FieldRef is a local variable.
*2 : I have tried using COMMIT (shame on me...) every 500 recs and it doesn't help at all. It only seems (for some weird reason) that the FieldRef is slower to be assigned to a field if the parent recordref is getting bigger.
*3 : sigh... have also tried this didn't work.

I have also tried

field := recordref2.field(...)
field.value := .....
recordref := recordref2.duplicate;
recordref.insert

to be sure that the two recordrefs are NOT data-related, because duplicate doesn't make recordref a pointer to recordref2 as if recordref := recordref2 would do.

fb wrote:

Here's a few more random thoughts:

1) Prior to v3.70, there was a problem with RecordRefs/FieldRefs declared as Globals. Is your 'field' Global or Local?

2) First 1000 records insert at 100recs/sec, next 300 at 10 recs/sec. Is it possible that this is 'normal' in the sense that the first 1000 recs simply fill Commit Cache, but from then on inserts are limited by a disk i/o bottleneck?

3) If the reuse of 'field' is somehow leaking memory, how about trying to CLEAR(field) between each use...

fb

So (to pursue a theory that new-ing/free-ing nRecs x nFields FieldRef objects is subject to n-squared run time,) how about the following:

• Declare an array[bigNumber] of FieldRefs;
• Open the RecordRef;
• For each fieldID in ArrayOfFieldIDs, set FieldRefs to RecordRef.FIELD(fieldID);
• Then, enter your 'for every lots and lots of inputs...' code, but use FieldRef as you walk through the fields of the record...

The idea here is to get the FieldRefs from the RecordRef once, and re-use them every time through the loop, instead of the current pattern that creates/destroys a single fieldref (nRecs x nField times)...

henrikohm

Heureka!!!!!!!

I made a new function "fieldref2recordref(...,...)" that
does the sub-assigning of fieldrefs as before. I didn't make use of the
array, just moving the fieldref -> recordref assigning to a new function.!

"This problem exists because of a memory leak in the RecRef/FieldRef definition" - Thomas Brodkorb (m b s o n l i n e . o r g )

Thank you for your suggestions FB! Local variables in local functions is apparently the KEY-issue in Navision for keeping memory etc. low.

Have a nice day

Best regards
Henrik Ohm

fb wrote:

So (to pursue a theory that new-ing/free-ing nRecs x nFields FieldRef objects is subject to n-squared run time,) how about the following:

• Declare an array[bigNumber] of FieldRefs;
• Open the RecordRef;
• For each fieldID in ArrayOfFieldIDs, set FieldRefs to RecordRef.FIELD(fieldID);
• Then, enter your 'for every lots and lots of inputs...' code, but use FieldRef as you walk through the fields of the record...

The idea here is to get the FieldRefs from the RecordRef once, and re-use them every time through the loop, instead of the current pattern that creates/destroys a single fieldref (nRecs x nField times)...

ajhvdb

I'm a bit puzzled what your solution is. Use local fieldrefs, recordrefs?
Could you explain this a bit more?

henrikohm

ok..

Whereever you can: Use local vars. expecially for RecordRef + FieldRef

in this case:

for every recordref
    recordref.init
    fieldref2recordref(....., VAR recordref)
    recordref.insert
loop end

ok?