Hidden Characters (Carriage Return and Line Feed)

joshplummer

I've recently seen a rash of situations where people are putting hidden characters into fields in NAV.
Specifically, they are getting Carriage Returns and Line Feeds into things like Invoice Numbers, Addresses, Names, etc...

We do a lot of extracting of data for analytical purposes, alot of which goes to flat files for exchange with corporate systems and warehouses.

When these files are created, they are screwed up because the bad data causes an premature break, adding extra lines and causing data exchange jobs to fail.

It's nearly impossible to actually type in a Line Feed or Carriage Return by hand, accidentally.

What we've found is that users are copying and pasting from websites (for addresses, etc...) and that is carrying along these hidden characters.

Has anyone else had this problem?

Savatage

How are you extracting data? with a dataport? perhaps you can detect & delete these characters as they come up?

Might not be the best way but you can check on entry of the field

"Your Field" - OnValidate()

CHAR10 := 10;
CHAR13 := 13;
POS := STRPOS("Your Field", FORMAT(CHAR10));
IF POS <> 0 THEN ERROR('Pasting Canceled Carrage Return Exists');
POS := STRPOS("Your Field", FORMAT(CHAR13));
IF POS <> 0 THEN ERROR('Pasting Canceled Line Feed Exists');

Unfortunately sould like alot of fields they are pasting into. That's why I was thinking of cleaning it up on export. :-k

Malajlo

Change Codeunit:1 in function MakeText. I'm allowing pasting CR and LF in some fields that's why it is used only on text that is shorter or equal 50 chars (longer text is for composing additional descriptions on Items and similar)

Local Vars
Name	DataType	Subtype	Length
cr	Char		
lf	Char		
tab	Char		

...
MakeText(VAR Text : Text[250]) : Integer
//AX001 - Start
Position := 1;
cr := 13 ;
lf := 10 ;
tab := 9 ;
Length := STRLEN(Text);
IF Length <= 50 THEN
  BEGIN
    Text := DELCHR(Text,'=',FORMAT(cr)) ;
    Text := DELCHR(Text,'=',FORMAT(lf)) ;
    Text := DELCHR(Text,'=',FORMAT(tab)) ;
  END ;
//AX001 - End
Length := STRLEN(Text);
ReadCharacter(' ',Text,Position,Length);
IF NOT ReadSymbol('?',Text,Position,Length) THEN
  EXIT(0);
...

Belias

Malajlo wrote:

Change Codeunit:1 in function MakeText. I'm allowing pasting CR and LF in some fields that's why it is used only on text that is shorter or equal 50 chars (longer text is for composing additional descriptions on Items and similar)

Local Vars
Name	DataType	Subtype	Length
cr	Char		
lf	Char		
tab	Char		

...
MakeText(VAR Text : Text[250]) : Integer
//AX001 - Start
Position := 1;
cr := 13 ;
lf := 10 ;
tab := 9 ;
Length := STRLEN(Text);
IF Length <= 50 THEN
  BEGIN
    Text := DELCHR(Text,'=',FORMAT(cr)) ;
    Text := DELCHR(Text,'=',FORMAT(lf)) ;
    Text := DELCHR(Text,'=',FORMAT(tab)) ;
  END ;
//AX001 - End
Length := STRLEN(Text);
ReadCharacter(' ',Text,Position,Length);
IF NOT ReadSymbol('?',Text,Position,Length) THEN
  EXIT(0);
...

interesting.... :-k
another solution is: try with this(i've never used it, maybe you want to test it)
http://www.mibuso.com/dlinfo.asp?FileID=1011

joshplummer

The suggested code works beautifully for text fields (modifying the MakeText function).

BUT...it doesn't do anything for CODE fields.

And, the first problem I encountered was on the Vendor Invoice No. field, which is a CODE field.

Is there a similar function that handles input to all Code fields?

matteo_montanari

Hi

You can modify your personalization like this:

...
MakeText(VAR Text : Text[250]) : Integer
//AX001 - Start
Position := 1;
Text2Remove[1] := 13;
Text2Remove[2] := 10;
Text2Remove[3] := 9;
IF STRLEN(Text) <= 50 THEN
  Text := DELCHR(Text,'=',Text2Remove) ;
//AX001 - End
Length := STRLEN(Text);
ReadCharacter(' ',Text,Position,Length);
IF NOT ReadSymbol('?',Text,Position,Length) THEN
  EXIT(0);
...

Bye

Matteo

gerdhuebner

joshplummer wrote:

...BUT...it doesn't do anything for CODE fields...

Unfortunately the MakeText trigger in Codeunit 1 is only called automatically for fields of type Text (not for Code fields).

gerdhuebner

joshplummer wrote:

...What we've found is that users are copying and pasting from websites

Well, the copy and paste problem for NAV field input is not restricted to copying from websites.
To see what I mean, just open Notepad and type "TEST" followed by a the enter key. Then select the whole text (with Ctrl+A) and paste it into a NAV text or code field... - you will see some extra characters (namely CR and LF) at the end of the string, when the input field in NAV is highlighted.
Since (as you already mentioned) it is nearly impossible to input CR/LF manually (not even with Alt+010 or Alt+013, etc.), I would concern this as a bug. There is a principle philosophy about copy and paste: It should not be possible to paste something that could not be entered manually or by means of a "legal" method.

joshplummer

gerdhuebner wrote:

joshplummer wrote:

...What we've found is that users are copying and pasting from websites

Well, the copy and paste problem for NAV field input is not restricted to copying from websites.
To see what I mean, just open Notepad and type "TEST" followed by a the enter key. Then select the whole text (with Ctrl+A) and paste it into a NAV text or code field... - you will see some extra characters (namely CR and LF) at the end of the string, when the input field in NAV is highlighted.
Since (as you already mentioned) it is nearly impossible to input CR/LF manually (not even with Alt+010 or Alt+013, etc.), I would concern this as a bug. There is a principle philosophy about copy and paste: It should not be possible to paste something that could not be entered manually or by means of a "legal" method.

Thanks. I understand the scope of the problem. Websites just happen to be where we are getting the majority of our issues.
In either case, the solution above fixes all the Text fields....but I still need a solution for the CODE fields.
Is there a way to intercept the function that applies the UPPERCASE to Code Fields? If so, I could add the appropriate code there....I assume not though.

Savatage

So i guess you're looking for an "all over" solutions like modifying cu1. Is adding code to the problem code fields OnValidate too much work?

Belias

Is there a way to intercept the function that applies the UPPERCASE to Code Fields? If so, I could add the appropriate code there....I assume not though.

No, there is not such a function in nav...it's the datatype that uppercases the text...the "trigger" is the same that throws the error if you try to type 'a' in an integer field.

gerdhuebner

joshplummer wrote:

...Is there a way to intercept the function that applies the UPPERCASE to Code Fields?

Well, I think there might be some passable way to solve the problem... - In NAV there is a functionality called "Change Log". Every time records are changed, inserted or deleted some triggers in Codeunit 1 are called.
Here is a sample code snippet to prevent the user from modifying code fields with "illegally" pasted characters - provided you are not using the standard "Change Log" functionality (otherwise the code had to be modified...):
1.) Put the following code in the beginning of the method GetGlobalTableTriggerMask in Codeunit 1:

EXIT(2);

This will cause NAV to call the method OnGlobalModify every time an arbitrary record is modified by the user.
2.) Put the following code in the beginning of the method OnGlobalModify in Codeunit 1:

illegalString[1] := 13;
illegalString[2] := 10;
// ...
Field.SETRANGE(TableNo,RecRef.NUMBER);
Field.SETRANGE(Type,Field.Type::Code);
Field.SETRANGE(Class,Field.Class::Normal);
Field.SETRANGE(Enabled,TRUE);
IF Field.FIND('-') THEN
  REPEAT
    fRef := RecRef.FIELD(Field."No.");
    IF DELCHR(FORMAT(fRef),'=',illegalString) <> FORMAT(fRef) THEN
      ERROR('Illegal characters in field "%1".',Field."Field Caption");
  UNTIL Field.NEXT = 0;

Here, illegalString is a text variable of suitable length, containg the "illegal" characters you want to test. Field is a record variable for the Field-Table (virtual table no. 2000000041), fRef is a FieldRef variable.
Be sure to restart your NAV client after modifying Codeunit 1 for changes to take effect.

One disadvantage of the above method (besides evtl. decrease in performance) is that the error is thrown not until the record is to be modified, causing all evtl. inputs in other fields to be lost. The advantage of course is that you can check all code fields in all tables with little coding effort. The method can be extended (if necessary) for insert and/or rename transactions. Instead of throwing an error it is also possible to remove the illegal characters automatically (without notice for the user). This can be achieved with the following modification:

illegalString[1] := 13;
illegalString[2] := 10;
// ...
Field.SETRANGE(TableNo,RecRef.NUMBER);
Field.SETRANGE(Type,Field.Type::Code);
Field.SETRANGE(Class,Field.Class::Normal);
Field.SETRANGE(Enabled,TRUE);
IF Field.FIND('-') THEN
  REPEAT
    fRef := RecRef.FIELD(Field."No.");
    fRef.VALUE := DELCHR(FORMAT(fRef),'=',illegalString);
  UNTIL Field.NEXT = 0;

Duikmeester

Very late reply, but you can copy the MakeText trigger, rename it MakeCode and give it ID 109.

100 MakeSpecial (DateFormula, RecordID, TableFilter)
200 MakeSpecialFilter
101 MakeBoolean
201 MakeBooleanFilter
102 MakeOption
202 MakeOptionFilter
103 MakeInteger
203 MakeIntegerFilter
104 MakeDecimal
204 MakeDecimalFilter
105 MakeDate
205 MakeDateFilter
106 MakeTime
206 MakeTimeFilter
107 MakeText
207 MakeTextFilter
109 MakeCode
209 MakeCodeFilter
110 MakeBinary
210 MakeBinaryFilter
111 MakeBLOB
211 MakeBLOBFilter
112 MakeBigInteger
212 MakeBigIntegerFilter
113 MakeDuration
213 MakeDurationFilter
114 MakeDateTime
214 MakeDateTimeFilter
115 MakeGUID
215 MakeGUIDFilter

Did not test them all but just know the IDs when they are called.

FDickschat

=D> =D> =D> =D> =D> =D> =D> =D> =D> =D>
You are the Best! :thumbsup:

In Fu MakeText I added the below code to remove CR/LFs ages ago. Now I added Fu MakeCode (ID 109) with the same code and there are no more CR/LFs in the DB \:D/

CharToDelete_l[1] := 10;
CharToDelete_l[2] := 13;
CharToDelete_l[3] := 9;

CodeText := DELCHR(CodeText, '=', CharToDelete_l);

Another good reason to call all the old customers again and make them a little happier.

JFGcanes

Duikmeester wrote:

Very late reply, but you can copy the MakeText trigger, rename it MakeCode and give it ID 109.

100 MakeSpecial (DateFormula, RecordID, TableFilter)
200 MakeSpecialFilter
101 MakeBoolean
201 MakeBooleanFilter
102 MakeOption
202 MakeOptionFilter
103 MakeInteger
203 MakeIntegerFilter
104 MakeDecimal
204 MakeDecimalFilter
105 MakeDate
205 MakeDateFilter
106 MakeTime
206 MakeTimeFilter
107 MakeText
207 MakeTextFilter
109 MakeCode
209 MakeCodeFilter
110 MakeBinary
210 MakeBinaryFilter
111 MakeBLOB
211 MakeBLOBFilter
112 MakeBigInteger
212 MakeBigIntegerFilter
113 MakeDuration
213 MakeDurationFilter
114 MakeDateTime
214 MakeDateTimeFilter
115 MakeGUID
215 MakeGUIDFilter

Did not test them all but just know the IDs when they are called.

Where do I specify the ID Number for the new function? Where are the ID Number's for Functions located in general. Sorry if this is basic. Thanks.

Luc_VanDyck

JFGcanes wrote:

Where do I specify the ID Number for the new function? Where are the ID Number's for Functions located in general. Sorry if this is basic. Thanks.

When you look at the names of the functions in the C/AL Globals form, you can press Shift+F4 to view their properties. Functions only have 2 properties: ID and Local.

Duikmeester

In the window C/AL Globals on the tab Functions choose Properties (Shift+F4)

Edit: and on the next page already a reply from Luc ...

tomddlc

And for those who (like me until five minutes ago) didn't know - If you install the latest hotfix for NAV 2009 R2 (Build 33413 or later), this is actually fixed in the client. So now pasting a text with CR/LF into NAV will automatically strip the illegal chars from the string.

Nice little improvement, that is going to save me a lot of grief

kriki

[Topic moved from 'NAV/Navision Classic Client' forum to 'NAV Tips & Tricks' forum]

agentz

if u use wevservices to get the objects via code, i have a reflection-based enumerator which can replace values of sny datatype in the object and return the new object with the scrubbed values.

i do this because sometimes systems which consume my data require NULL as a string instead of the actual null or empty value.

the code is yours if u like. just pm me for it.