Can I Replace or Strip characters from Strings?

[bldcm]

I am trying to replace "invalid characters" so they can be saved in paragraph fields like description. Below is a snipit of a V1 configuration with character maps. If only one of the characters is present then the mapping works fine. If multiple characters are present it does not work and logging seems to indicate that each mapping starts with the original source data to do the mapping, meaning only the last found mapping takes affect.

Can multiple mappings be done on a single field?
Can multiple fields use the same mapping (i.e. have multiple source fields and targetfields) with repeating the configuration for each field?

"FieldMaps": [
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uFFFF",
"replacement": "U+FFFF"
},
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uF1A4",
"replacement": "U+F1A4"
},
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "*",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uF210",
"replacement": "U+F210"
},

[MrHinsh]

Correct. this is not the tool for that job. Use the StringManipulatorTool instead!

 {
  "MigrationTools": {
    "Version": "16.0",
    "CommonTools": {
      "StringManipulatorTool": {
        "Enabled": "True",
        "Manipulators": [
          {
            "$type": "RegexStringManipulator",
            "Description": "Remove invalid characters from string",
            "Enabled": "True",
            "Pattern": "[^( -~)\n\r\t]+",
            "Replacement": ""
          }
        ],
        "MaxStringLength": "1000000"
      }
    }
  }
}

You can add multiple manipulators or create one uber pattern... there is also a fix in the latest preview to enable you to override the default rather than merge it in... note that the Tfs Object Model does not support some characters...

note: The workaround is to change the appsettings.json and remove the default manipulator if you don't have the version with the fix.

---

Funding for us doing work on this tool is generally through 1) customer funding, or 2) donated free time.

Sponcer a Feature | Hire us directly

[bldcm]

I want to match 7 things and replace with the same thing, "". I have tried the default pattern, a pattern I tested with a regex tester for my specific characters and a pattern with the whole string, each individually, not all at same time. None of them worked. If I remove the string from the source, the WI migrates, so I need a way to get rid of these individual characters.

These are the characters: "��O펧甛O"

"Pattern": "[^( -~)\n\r\t]+",
"Pattern": "[\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+",
"Pattern": "(\uffff)?\uf1a4\u004f\ud3a7\u751b\uf210\u004f",

I've attached logs of all three failures, one for each pattern I have tested.

20241015153139-logOfIndividualCharactersPattern.zip
20241009112303-logOfWholeStringPatern.zip
20241015152932-logOfDefaultPattern.zip
configuration-TFS2ADOPreProd-PowerPDF-NOTO-upgraded-DIAGS.json

Side question, how do I increasing the logging verbosity to get more details in log if needed?

[MrHinsh]

The default pattern only allows through asking characters between - and ~, everything else will be stripped.

"Pattern": "[\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+",

This is not a valid pattern acording to https://regex101.com/

"Pattern": "(\uffff)?\uf1a4\u004f\ud3a7\u751b\uf210\u004f",

This is also not a valid pattern.

In the config, set

"Serilog": {
  "MinimumLevel": "Verbose"
},

---

Funding for us doing work on this tool is generally through 1) customer funding, or 2) donated free time.

Sponcer a Feature | Hire us directly

[bldcm]

The website you referenced, https://regex101.com/, for regex supports multiple "flavors" and 5 of 8 found the reg expression valid as shown in the image below

I also believe it is valid because I have used similar regexpr in the migration tool as shown in the first post of this discussion and repeated here:

{
  "$type": "RegexFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Description",
  "targetField": "System.Description",
  "pattern": "\uFFFF",
  "replacement": "U+FFFF"
},    
{
  "$type": "RegexFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Description",
  "targetField": "System.Description",
  "pattern": "\uF1A4",
  "replacement": "U+F1A4"
},
{
  "$type": "RegexFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Description",
  "targetField": "System.Description",
  "pattern": "\uF210",
  "replacement": "U+F210"
},
{
  "$type": "FieldtoFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Team",
  "targetField": "System.Team",
  "defaultValue": "Power PDF"
},

It was after this I was directed to use the RegexStringManipulator tool.

So unless the two methods use different regular expression flavors, I believe it is valid.

[MrHinsh]

What are you expecting [\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+ to match? Are those characters always together?

They use the same regex engine, but do not have the same match, this would be the same match:

 {
  "MigrationTools": {
    "Version": "16.0",
    "CommonTools": {
      "StringManipulatorTool": {
        "Enabled": "True",
        "Manipulators": [
          {
            "$type": "RegexStringManipulator",
            "Description": "Remove invalid characters from string",
            "Enabled": "True",
            "Pattern": "\uF1A4",
            "Replacement": "U+F1A4"
          }
        ],
        "MaxStringLength": "1000000"
      }
    }
  }
}

however [\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+ does not match \uF1A4".

[MrHinsh]

The advantage of the StringManipulatorTool is that it applies to any string field.

[bldcm]

I'm expecting the pattern to match any instance of any individual character in the set defined by the pattern [\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+, and replace each character found with "". In this example the characters happen to be together, but that is not necessarily the case.

I tested the pattern on the website you use for regex testing and it identifies each character in the string whether they are together or not. See screen capture below:

Not sure what what your statement, "however [\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+ does not match \uF1A4".", is trying to say. Do you think there is a casing problem or that the set of \u characters is not valid for the StringManiuplatorTool?

[bldcm]

Does anyone have an example of using StringManipulatorTool to remove unicode characters defined in a RegEx set?

[bldcm]

I wanted to give some feedback on this as we have come up a solution.

The StringManipulatorTool was suggested as a way to modify fields like Description. Unfortunately, at least in our environment, this tool only works on fields of type string and description is of type HTML. So the fields we were trying to modify did not work because they were being skipped. I don't remember seeing this in the description/documentation, even though the tool is named "string" manipulation too.

So we had to go back to the Field mapping tool and use what we learned about functioning regex expressions and sets of characters to replace.

We actually had a coding / configuration option which also worked, but we chose to go without making a code change so we wouldn't have to maintain it. We added a "ProcessHTMLFields" option to the tool so that HTML fields would not be exempt. The field mapping tools don't need this because you explicitly set which field you want searched.

If the string tool got the HTML process update it would make configuration easier because we wouldn't have to do it for every field.

Side note: During debugging, we found a performance issue in BOTH the string manipulator tool and field mapping tools. EVERY revision processed the field in question as if it had never been processed before regardless of whether or not it had been changed in the revision. This may be a problem with TFS/ADO setting a false revision flag, but I know the field I was testing got search each and every revision processed and oddly enough it ALWAYS had the starting value of the field and always replaced the characters. The tool didn't seem to know it had already been processed.