Hidden Characters (Carriage Return and Line Feed)

joshplummerjoshplummer Posts: 39Member
edited 2012-07-27 in NAV Tips & Tricks
I've recently seen a rash of situations where people are putting hidden characters into fields in NAV.
Specifically, they are getting Carriage Returns and Line Feeds into things like Invoice Numbers, Addresses, Names, etc...

We do a lot of extracting of data for analytical purposes, alot of which goes to flat files for exchange with corporate systems and warehouses.

When these files are created, they are screwed up because the bad data causes an premature break, adding extra lines and causing data exchange jobs to fail.

It's nearly impossible to actually type in a Line Feed or Carriage Return by hand, accidentally.

What we've found is that users are copying and pasting from websites (for addresses, etc...) and that is carrying along these hidden characters.

Has anyone else had this problem?

Comments

  • SavatageSavatage Posts: 7,132Member
    How are you extracting data? with a dataport? perhaps you can detect & delete these characters as they come up?

    Might not be the best way but you can check on entry of the field

    "Your Field" - OnValidate()
    CHAR10 := 10;
    CHAR13 := 13;
    POS := STRPOS("Your Field", FORMAT(CHAR10));
    IF POS <> 0 THEN ERROR('Pasting Canceled Carrage Return Exists');
    POS := STRPOS("Your Field", FORMAT(CHAR13));
    IF POS <> 0 THEN ERROR('Pasting Canceled Line Feed Exists');
    

    Unfortunately sould like alot of fields they are pasting into. That's why I was thinking of cleaning it up on export. :-k
  • MalajloMalajlo Posts: 297Member
    Change Codeunit:1 in function MakeText. I'm allowing pasting CR and LF in some fields that's why it is used only on text that is shorter or equal 50 chars (longer text is for composing additional descriptions on Items and similar)
    Local Vars
    Name	DataType	Subtype	Length
    cr	Char		
    lf	Char		
    tab	Char		
    
    ...
    MakeText(VAR Text : Text[250]) : Integer
    //AX001 - Start
    Position := 1;
    cr := 13 ;
    lf := 10 ;
    tab := 9 ;
    Length := STRLEN(Text);
    IF Length <= 50 THEN
      BEGIN
        Text := DELCHR(Text,'=',FORMAT(cr)) ;
        Text := DELCHR(Text,'=',FORMAT(lf)) ;
        Text := DELCHR(Text,'=',FORMAT(tab)) ;
      END ;
    //AX001 - End
    Length := STRLEN(Text);
    ReadCharacter(' ',Text,Position,Length);
    IF NOT ReadSymbol('?',Text,Position,Length) THEN
      EXIT(0);
    ...
    
  • BeliasBelias Posts: 2,951Member
    Malajlo wrote:
    Change Codeunit:1 in function MakeText. I'm allowing pasting CR and LF in some fields that's why it is used only on text that is shorter or equal 50 chars (longer text is for composing additional descriptions on Items and similar)
    Local Vars
    Name	DataType	Subtype	Length
    cr	Char		
    lf	Char		
    tab	Char		
    
    ...
    MakeText(VAR Text : Text[250]) : Integer
    //AX001 - Start
    Position := 1;
    cr := 13 ;
    lf := 10 ;
    tab := 9 ;
    Length := STRLEN(Text);
    IF Length <= 50 THEN
      BEGIN
        Text := DELCHR(Text,'=',FORMAT(cr)) ;
        Text := DELCHR(Text,'=',FORMAT(lf)) ;
        Text := DELCHR(Text,'=',FORMAT(tab)) ;
      END ;
    //AX001 - End
    Length := STRLEN(Text);
    ReadCharacter(' ',Text,Position,Length);
    IF NOT ReadSymbol('?',Text,Position,Length) THEN
      EXIT(0);
    ...
    
    interesting.... :-k
    another solution is: try with this(i've never used it, maybe you want to test it)
    http://www.mibuso.com/dlinfo.asp?FileID=1011
    -Mirko-
    "Never memorize what you can easily find in a book".....Or Mibuso
    My Blog
  • joshplummerjoshplummer Posts: 39Member
    The suggested code works beautifully for text fields (modifying the MakeText function).

    BUT...it doesn't do anything for CODE fields.

    And, the first problem I encountered was on the Vendor Invoice No. field, which is a CODE field.

    Is there a similar function that handles input to all Code fields?
  • matteo_montanarimatteo_montanari Posts: 189Member
    Hi

    You can modify your personalization like this:
    ...
    MakeText(VAR Text : Text[250]) : Integer
    //AX001 - Start
    Position := 1;
    Text2Remove[1] := 13;
    Text2Remove[2] := 10;
    Text2Remove[3] := 9;
    IF STRLEN(Text) <= 50 THEN
      Text := DELCHR(Text,'=',Text2Remove) ;
    //AX001 - End
    Length := STRLEN(Text);
    ReadCharacter(' ',Text,Position,Length);
    IF NOT ReadSymbol('?',Text,Position,Length) THEN
      EXIT(0);
    ...
    

    Bye

    Matteo
    Reno Sistemi Navision Developer
  • gerdhuebnergerdhuebner Posts: 123Member
    ...BUT...it doesn't do anything for CODE fields...
    Unfortunately the MakeText trigger in Codeunit 1 is only called automatically for fields of type Text (not for Code fields).
  • gerdhuebnergerdhuebner Posts: 123Member
    ...What we've found is that users are copying and pasting from websites
    Well, the copy and paste problem for NAV field input is not restricted to copying from websites.
    To see what I mean, just open Notepad and type "TEST" followed by a the enter key. Then select the whole text (with Ctrl+A) and paste it into a NAV text or code field... - you will see some extra characters (namely CR and LF) at the end of the string, when the input field in NAV is highlighted.
    Since (as you already mentioned) it is nearly impossible to input CR/LF manually (not even with Alt+010 or Alt+013, etc.), I would concern this as a bug. There is a principle philosophy about copy and paste: It should not be possible to paste something that could not be entered manually or by means of a "legal" method.
  • joshplummerjoshplummer Posts: 39Member
    ...What we've found is that users are copying and pasting from websites
    Well, the copy and paste problem for NAV field input is not restricted to copying from websites.
    To see what I mean, just open Notepad and type "TEST" followed by a the enter key. Then select the whole text (with Ctrl+A) and paste it into a NAV text or code field... - you will see some extra characters (namely CR and LF) at the end of the string, when the input field in NAV is highlighted.
    Since (as you already mentioned) it is nearly impossible to input CR/LF manually (not even with Alt+010 or Alt+013, etc.), I would concern this as a bug. There is a principle philosophy about copy and paste: It should not be possible to paste something that could not be entered manually or by means of a "legal" method.

    Thanks. I understand the scope of the problem. Websites just happen to be where we are getting the majority of our issues.
    In either case, the solution above fixes all the Text fields....but I still need a solution for the CODE fields.
    Is there a way to intercept the function that applies the UPPERCASE to Code Fields? If so, I could add the appropriate code there....I assume not though.
  • SavatageSavatage Posts: 7,132Member
    So i guess you're looking for an "all over" solutions like modifying cu1. Is adding code to the problem code fields OnValidate too much work?
  • BeliasBelias Posts: 2,951Member
    Is there a way to intercept the function that applies the UPPERCASE to Code Fields? If so, I could add the appropriate code there....I assume not though.
    No, there is not such a function in nav...it's the datatype that uppercases the text...the "trigger" is the same that throws the error if you try to type 'a' in an integer field.
    -Mirko-
    "Never memorize what you can easily find in a book".....Or Mibuso
    My Blog
  • gerdhuebnergerdhuebner Posts: 123Member
    ...Is there a way to intercept the function that applies the UPPERCASE to Code Fields?
    Well, I think there might be some passable way to solve the problem... - In NAV there is a functionality called "Change Log". Every time records are changed, inserted or deleted some triggers in Codeunit 1 are called.
    Here is a sample code snippet to prevent the user from modifying code fields with "illegally" pasted characters - provided you are not using the standard "Change Log" functionality (otherwise the code had to be modified...):
    1.) Put the following code in the beginning of the method GetGlobalTableTriggerMask in Codeunit 1:
    EXIT(2);
    
    This will cause NAV to call the method OnGlobalModify every time an arbitrary record is modified by the user.
    2.) Put the following code in the beginning of the method OnGlobalModify in Codeunit 1:
    illegalString[1] := 13;
    illegalString[2] := 10;
    // ...
    Field.SETRANGE(TableNo,RecRef.NUMBER);
    Field.SETRANGE(Type,Field.Type::Code);
    Field.SETRANGE(Class,Field.Class::Normal);
    Field.SETRANGE(Enabled,TRUE);
    IF Field.FIND('-') THEN
      REPEAT
        fRef := RecRef.FIELD(Field."No.");
        IF DELCHR(FORMAT(fRef),'=',illegalString) <> FORMAT(fRef) THEN
          ERROR('Illegal characters in field "%1".',Field."Field Caption");
      UNTIL Field.NEXT = 0;
    
    Here, illegalString is a text variable of suitable length, containg the "illegal" characters you want to test. Field is a record variable for the Field-Table (virtual table no. 2000000041), fRef is a FieldRef variable.
    Be sure to restart your NAV client after modifying Codeunit 1 for changes to take effect.

    One disadvantage of the above method (besides evtl. decrease in performance) is that the error is thrown not until the record is to be modified, causing all evtl. inputs in other fields to be lost. The advantage of course is that you can check all code fields in all tables with little coding effort. The method can be extended (if necessary) for insert and/or rename transactions. Instead of throwing an error it is also possible to remove the illegal characters automatically (without notice for the user). This can be achieved with the following modification:
    illegalString[1] := 13;
    illegalString[2] := 10;
    // ...
    Field.SETRANGE(TableNo,RecRef.NUMBER);
    Field.SETRANGE(Type,Field.Type::Code);
    Field.SETRANGE(Class,Field.Class::Normal);
    Field.SETRANGE(Enabled,TRUE);
    IF Field.FIND('-') THEN
      REPEAT
        fRef := RecRef.FIELD(Field."No.");
        fRef.VALUE := DELCHR(FORMAT(fRef),'=',illegalString);
      UNTIL Field.NEXT = 0;
    
  • DuikmeesterDuikmeester Posts: 71Member
    Very late reply, but you can copy the MakeText trigger, rename it MakeCode and give it ID 109.

    100 MakeSpecial (DateFormula, RecordID, TableFilter)
    200 MakeSpecialFilter
    101 MakeBoolean
    201 MakeBooleanFilter
    102 MakeOption
    202 MakeOptionFilter
    103 MakeInteger
    203 MakeIntegerFilter
    104 MakeDecimal
    204 MakeDecimalFilter
    105 MakeDate
    205 MakeDateFilter
    106 MakeTime
    206 MakeTimeFilter
    107 MakeText
    207 MakeTextFilter
    109 MakeCode
    209 MakeCodeFilter
    110 MakeBinary
    210 MakeBinaryFilter
    111 MakeBLOB
    211 MakeBLOBFilter
    112 MakeBigInteger
    212 MakeBigIntegerFilter
    113 MakeDuration
    213 MakeDurationFilter
    114 MakeDateTime
    214 MakeDateTimeFilter
    115 MakeGUID
    215 MakeGUIDFilter

    Did not test them all but just know the IDs when they are called.
  • FDickschatFDickschat Posts: 380Member
    =D> =D> =D> =D> =D> =D> =D> =D> =D> =D>
    You are the Best! :thumbsup:

    In Fu MakeText I added the below code to remove CR/LFs ages ago. Now I added Fu MakeCode (ID 109) with the same code and there are no more CR/LFs in the DB :D \:D/
    CharToDelete_l[1] := 10;
    CharToDelete_l[2] := 13;
    CharToDelete_l[3] := 9;
    
    CodeText := DELCHR(CodeText, '=', CharToDelete_l);
    
    Another good reason to call all the old customers again and make them a little happier.
    Frank Dickschat
    FD Consulting
  • JFGcanesJFGcanes Posts: 11Member
    Very late reply, but you can copy the MakeText trigger, rename it MakeCode and give it ID 109.

    100 MakeSpecial (DateFormula, RecordID, TableFilter)
    200 MakeSpecialFilter
    101 MakeBoolean
    201 MakeBooleanFilter
    102 MakeOption
    202 MakeOptionFilter
    103 MakeInteger
    203 MakeIntegerFilter
    104 MakeDecimal
    204 MakeDecimalFilter
    105 MakeDate
    205 MakeDateFilter
    106 MakeTime
    206 MakeTimeFilter
    107 MakeText
    207 MakeTextFilter
    109 MakeCode
    209 MakeCodeFilter
    110 MakeBinary
    210 MakeBinaryFilter
    111 MakeBLOB
    211 MakeBLOBFilter
    112 MakeBigInteger
    212 MakeBigIntegerFilter
    113 MakeDuration
    213 MakeDurationFilter
    114 MakeDateTime
    214 MakeDateTimeFilter
    115 MakeGUID
    215 MakeGUIDFilter

    Did not test them all but just know the IDs when they are called.


    Where do I specify the ID Number for the new function? Where are the ID Number's for Functions located in general. Sorry if this is basic. Thanks.
  • Luc_Van_DyckLuc_Van_Dyck WilrijkPosts: 3,620Member, Moderator, Administrator
    JFGcanes wrote:
    Where do I specify the ID Number for the new function? Where are the ID Number's for Functions located in general. Sorry if this is basic. Thanks.
    When you look at the names of the functions in the C/AL Globals form, you can press Shift+F4 to view their properties. Functions only have 2 properties: ID and Local.
    No support using PM or e-mail - Please use this forum. || Search is your friend
  • DuikmeesterDuikmeester Posts: 71Member
    In the window C/AL Globals on the tab Functions choose Properties (Shift+F4)

    Edit: and on the next page already a reply from Luc ;) ...
  • tomddlctomddlc Posts: 8Member
    And for those who (like me until five minutes ago) didn't know - If you install the latest hotfix for NAV 2009 R2 (Build 33413 or later), this is actually fixed in the client. So now pasting a text with CR/LF into NAV will automatically strip the illegal chars from the string.

    Nice little improvement, that is going to save me a lot of grief :D
  • krikikriki Posts: 7,866Member, Moderator
    [Topic moved from 'NAV/Navision Classic Client' forum to 'NAV Tips & Tricks' forum]
    Regards,Alain Krikilion
    Use the SEARCH,Luke! || No PM,please use the forum. || May the <SOLVED>-attribute be in your title! || Read Forum Rules before making a posting

  • agentzagentz Posts: 14Member
    if u use wevservices to get the objects via code, i have a reflection-based enumerator which can replace values of sny datatype in the object and return the new object with the scrubbed values.

    i do this because sometimes systems which consume my data require NULL as a string instead of the actual null or empty value.

    the code is yours if u like. just pm me for it.
Sign In or Register to comment.