Bug when a multi-lines regex is used in the 'Search', 'Replace' or 'Mark' dialog

guy038

Hello, @mkupper, @alan-kilborn and All,

@mkupper, I repeated all your process and indeed, your method and explanations were very instructive !

I just do NOT understand one point, yet. You said, in a previous post :

Hopefully, both the normal and extended search allow for 30,000 character searches.

Well, repeating the points 1 to 5 of The 30000 characters Find what field limit section, with the FIND dialog pre-configured in Normal or Extended mode, it just matches the first 2,046 characters of the 70000___10 string, although the Find field do contain the first 30,000 chars of the 70000___10 string ?!

Best regards,

guy038

Alan Kilborn

@mkupper said:

** The 1024 character automatic select and load into Find what limit **
This is the item that started this forum thread.

As this IS what started the thread, I’d like to concentrate on that specific aspect.

While the magic number is 1024 this is apparently unrelated to the magic number 1024 found in Settings / Preferences / Searching (tab) setting for Minimum Size for Auto-Checking “In-selection”.

To evaluate the “apparently unrelated” part, I started changing the value for Minimum Size for Auto-Checking “In-selection”. And, unless I missed something, it does actually appear related. For example, creating 1022 and 1023 “ruler” lines and then changing the number in the box to 1023 yields what I’d expect (based on the 1024 behavior): That is, invoking Ctrl+f with the caret in the 1023 ruler checkmarks In selection, while invoking it with the caret in the 1022 ruler uncheckmarks it.

Thus I don’t think that 1024 is a “magic number”; it’s a default setting value, with no magic.

But I still feel like I’m missing something about the point @mkupper was trying to make about this.

Alan Kilborn

30000 vs. 32767

I’ve no idea where 30000 originates. A quick search of the Notepad++ source code won’t find it literally. So… apologies to @mkupper about the “sloppily” thing; I did think you were speaking in “ballpark” terms.

Alan Kilborn

Instead of Excel, why not use a bit of PythonScript to generate the “ruler” lines?:

        accum = ''
        for j in range(1020, 1030 + 1):
            desired_len = j
            des_len_as_str = str(desired_len)
            s = des_len_as_str
            tens_count = 0
            while True:
                if (len(s) + 1) % 10 == 0:
                    if (tens_count + 2) * 10 <= desired_len:
                        s += str((tens_count + 1) * 10)
                        tens_count += 1
                if len(s) >= desired_len: break
                s += '_'
            s = s[:-len(des_len_as_str)] + des_len_as_str
            accum += s + '\r\n'
        editor.copyText(accum)

The example above generates ruler lines of length 1020 through 1030, inclusive. The ruler data ends up in the clipboard after the script runs.

Note that mine might be different from the earlier ruler lines discussed – I chose that the intermediate numbers start in their indicated column, e.g. after you paste the output of the script into a new tab, if you put the caret just to the left of the 8 in 890, the status bar will indicate Col: 890.

To select 890 characters from that same example line, put the caret between the 8 and the 9 and then press Shift+Home.

Here’s some output from the script:

1020_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010___1020
1021_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010____1021
1022_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010_____1022
1023_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010______1023
1024_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010_______1024
1025_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010________1025
1026_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010_________1026
1027_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010__________1027
1028_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010___________1028
1029_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010____________1029
1030_____10________20________30________40________50________60________70________80________90________100_______110_______120_______130_______140_______150_______160_______170_______180_______190_______200_______210_______220_______230_______240_______250_______260_______270_______280_______290_______300_______310_______320_______330_______340_______350_______360_______370_______380_______390_______400_______410_______420_______430_______440_______450_______460_______470_______480_______490_______500_______510_______520_______530_______540_______550_______560_______570_______580_______590_______600_______610_______620_______630_______640_______650_______660_______670_______680_______690_______700_______710_______720_______730_______740_______750_______760_______770_______780_______790_______800_______810_______820_______830_______840_______850_______860_______870_______880_______890_______900_______910_______920_______930_______940_______950_______960_______970_______980_______990_______1000______1010______1020___1030

guy038

Hello, @mkupper, @alan-kilborn and All,

I did some tests and I succeeded to find a work-around in order to use multi-lines regexes of more than 1,024 characters, as long as, of course, the total amount of chars does not exceed 2,047 characters. All the tests were done with the last N++ release v8.8.1.

Here my method, to be followed rigorously.

In a new tab, paste, successively, the two MULTI-lines regexes, below :

(?x-i)                  # Search SENSIBLE to CASE
(?<=\x20)               # Preceded with SPACE
(?:                     # Start NON-CAPTURING group
0[0-2][0-9A-F][0-9A-F]  |
03[7-9A-F][0-9A-F]      |
04[0-9A-F][0-9A-F]      |
05[0-8][0-9A-F]         |
10[A-F][0-9A-F]         |
1C[8-B][0-9A-F]         |
1D[0-9AB][0-9A-F]       |
1[EF][0-9A-F][0-9A-F]   |
20[7-C][0-9A-F]         |
21[0-8][0-9A-F]         |
24[6-9A-F][0-9A-F]      |
25[A-F][0-9A-F]         |
27[0-B][0-9A-F]         |
2C[6-9A-F][0-9A-F]      |
2D[012][0-9A-F]         |
A6[4-9][0-9A-F]         |
A7[2-9A-F][0-9A-F]      |
AB[3-6][0-9A-F]         |
FB[01][0-9A-F]          |
FF[0-5E][0-9A-F]        |
102[EF][0-9A-F]         |
105[0-2][0-9A-F]        |
107[89AB][0-9A-F]       |
1CC[DEF][0-9A-F]        |
1D[4-7][0-9A-F][0-9A-F] |
1DF[0-9A-F][0-9A-F]     |
1E0[3-8][0-9A-F]        |
1F1[0-9A-F][0-9A-F]     |
1FB[0-9A-F][0-9A-F]
)                     # End NON-CAPTURING group
(?=\x20)              # Followed with SPACE

And :

(?x-si)
  (?:
    ^ ~~~ [\h\l]* \R (?s:.+?) ^ ~~~ \h* \R |
    ^ ``` [\h\l]* \R (?s:.+?) ^ ``` \h* \R |

    ^ \h* (?: - \h* )? (?i: FIND | SEARCH | REPLACE ) \x20 .+
  |
    (?<= \s )
    (?:
#        Case *......* / Case **......** / Case `......`

     \*   [^`* \t\r\n] (?: [^*\r\n]* [^`* \t\r\n] )? \*   |
     \*\* [^`* \t\r\n] (?: [^*\r\n]* [^`* \t\r\n] )? \*\* |
     `    [^` \t\r\n]  (?: [^\r\n]*  [^` \t\r\n] )? `     |

#        Case **`......`**

     \*\*                                          ` [^` \t\r\n] (?: [^`\r\n]* [^` \t\r\n] )? `                                        \*\* |
     \*\*                                          ` [^`\r\n]    (?: [^`\r\n]* [^`\r\n]    )? ` (?: [^*`\r\n] [^`\r\n]* )? [^` \t\r\n] \*\* |
     \*\* [^` \t\r\n] (?: [^`\r\n]*   [^*`\r\n] )? ` [^`\r\n]    (?: [^`\r\n]* [^`\r\n]    )? `                                        \*\* |
     \*\* [^` \t\r\n] (?: [^`\r\n]*   [^*`\r\n] )? ` [^`\r\n]    (?: [^`\r\n]* [^`\r\n]    )? ` (?: [^*`\r\n] [^`\r\n]* )? [^` \t\r\n] \*\* |

#        Case *`......`*

     \*                                            ` [^` \t\r\n] (?: [^`\r\n]* [^` \t\r\n] )? `                                          \* |
     \*                                            ` [^`\r\n]    (?: [^`\r\n]* [^`\r\n]    )? ` (?: [^*`\r\n] [^*`\r\n]* )? [^` \t\r\n]  \* |
     \*   [^` \t\r\n] (?: [^*`\r\n]* [^`*\r\n] )?  ` [^`\r\n]    (?: [^`\r\n]* [^`\r\n]    )? `                                          \* |
     \*   [^` \t\r\n] (?: [^*`\r\n]* [^`*\r\n] )?  ` [^`\r\n]    (?: [^`\r\n]* [^`\r\n]    )? ` (?: [^*`\r\n] [^*`\r\n]* )? [^` \t\r\n]  \*
    ){1}+
    (?= [\s,;.-] | 's | \z )
  )
  (*SKIP) (*F)  # CORRECT cases are IGNORED
|
  [*`]+         .+?  [*`]+  |
  [*`]+  \x20*  .+?  \x20+

Note that the first regex contains 1,022 characters and the second contains 1,802 chars, so one regex are less than 1,024 characters and the other more than 1,024.

Add, in this new tab, the two lines below ( the first line should be matched by the first regex and the second line should be matched by the second regex ).

 2C63 | LATIN CAPITAL LETTER P WITH STROKE | Ᵽ

*AB D *

To end, save this new tab as Test_RE.txt

First manipulation :

Switch to the Test_RE.txt file
Select the contents of the first regex ( 1,022 chars )
Open the Find dialog ( Ctrl + F )

=> As you can see, the end of this multi-lines regex is visible and you should note the part # Followed with SPACE

Uncheck all the box options if any ( note that the In selection option was already not checked )
Note that the Wrap around option will stay uncheched during all the tests
Select the Regular expression search mode
Click on the Find Next button

=> As expected, it matches the 2C63 string

Close the Find dialog

Second manipulation :

Switch to the Test_RE.txt file
This time, select the contents of the second regex ( 1,802 chars )
Open the Find dialog ( Ctrl + F )

=> You can observe two things :

The In selection option is checked ( Logical, as the amount of chars is greater than 1,024 )
Surprisingly, the regex, shown in the search field, is STILL the first regex and not the second expected multi-lines regex !
Hit the Find Next button : as expected, it returns, again, the 2C63 string ( Note that the In selection button is NOT concerned when you hit the Find Next button )
Close the Find dialog

At this point, you can, either :

Re-open the Find dialog
Close and re-load the Test_RE.txt file
Close and re-start Notepad++

Whatever you decided, if you re-open the Find dialog, the first regex, ending with the string # Followed with SPACE, is STILL present in the search field, although we previously select the second regex ???!!! Why ?

Third manipulation :

Switch to the Test_RE.txt file
Select, again, the contents of the second regex ( 1,802 chars )
Before opening the Find dialog, just hit the Ctrl + F3 shortcut ( This is the WORK-AROUND ! )
Open the Find dialog
This time, you can verify that the search field contains the correct regex : the second one
Hit the Find Next button => This time, it correctly matches the *AB D * string

Note that sometimes, I needed to cancel the selection, right before opening the Find dialog, in order to get this match !

It’s important to add that, modifying in Preferences... > Searching > When Fin Dialog is Invoked the value 1,024 to the value 0, does NOT change the global behavior

As a conclusion, I would say that all that logic seems rather unclear. Can anyone reproduce these steps and see the problem ?

Best Regards,

guy038

P.S. :

Note that without this work-around, I should have used the v8.4.9 release, which is the last version before the auto-checking of the In selection option

Alan Kilborn

@guy038 said:

Second manipulation :
…
…select the contents of the second regex ( 1,802 chars )
…
The In selection option is checked ( Logical, as the amount of chars is greater than 1,024 )
…
Surprisingly, the regex, shown in the search field, is STILL the first regex and not the second expected multi-lines regex !
…
Whatever you decided, if you re-open the Find dialog, the first regex, ending with the string # Followed with SPACE, is STILL present in the search field, although we previously select the second regex ???!!! Why ?

The way it works (I think) is that if In selection is going to become checkmarked due to the N++ code doing it, i.e., if the number of bytes in the selected text is greater-than-or-equal-to¹ the setting “Minimum Size for Auto-Checking ‘In selection’”, then the selected text is NOT supposed to be copied to Find what, regardless of the setting for Fill Find Field with Selected Text.

Why not?

Well, if the code has determined that you are going to be searching within the selected text, copying the selected text to Find what doesn’t make sense. You already know how many matches that would generate (exactly one). So, it is waiting for you to put something different in Find what.

¹ : In attempting to verify this before posting, I found out that it has to be greater-than, not greater-than-or-equal-to. Thus, for the default case of 1024, if 1024 bytes are selected when Ctrl+f is invoked, the checkbox will become checkmarked AND the selected text will be copied to Find what. This seems wrong to me; it should be as I first stated, greater-than-or-equal-to.

Third manipulation :
…
Select, again, the contents of the second regex ( 1,802 chars )
Before opening the Find dialog, just hit the Ctrl + F3 shortcut ( This is the WORK-AROUND ! )

Ctrl+F3 is Select and Find Next which is wholly different from a “select” followed by a Find Next. It’s not affected by, nor constrained by, the In selection setting.

It’s important to add that, modifying in Preferences… > Searching > When Find Dialog is Invoked the value 1,024 to the value 0, does NOT change the global behavior

I’m unclear on what you mean by this.

guy038

Hi, @mkupper, @alan-kilborn and All,

Sorry, I was out these last three hours !

When I said :

It’s important to add that, modifying in Preferences... > Searching > When Fin Dialog is Invoked the value 1,024 to the value 0, does NOT change the global behavior

I meant that my three manipulations produce the same results if you previously chose the 1,024 value or the 0 value : NO change in behavior.

BR

guy038

Thinking about it, I don’t know it this is judicious. If I choose the zero value, as auto-checking is disabled, if I select an important amount of text and immediately invoke the Find dialog, it should fill up the Find what field up to 2,046 characters !

Alan Kilborn

@guy038 said:

If I choose the zero value, as auto-checking is disabled, if I select an important amount of text and immediately invoke the Find dialog, it should fill up the Find what field up to 2,046 characters !

I’d say that that sounds reasonable.

mkupper

@guy038, @Alan-Kilborn, and others

Congratulations on discovering that Ctrl+F3 trick to bypass the 1024 character selection to the Find-what field limit.

That prompted me to do a test.

I created a 5000 character long ruler line with ________10________20________30 … ______4980______4990______5000.
I duplicated that line a few times.
I put the caret on one of the lines and did Ctrl+F3. Notepad++ immediately jumped to the next line and there’s a bunch of text selected.
I did Ctrl+C to load the selected text into the copy/paste buffer, and pasted to a blank line in the area below my list of 5000 character rulers.
I saw that the new line is 2047 characters long and runs from ________10________20________30 … ______2030______2040______2

My first thought was, “wait, I thought the search limit was 2046 characters. Apparently the quick search thing using Ctrl+F3 will search for up to 2047 character patterns.”

I verified that a reverse quick search using Ctrl+Shift+F3 also allows or up to 2047 character patterns.

I moved the caret to a blank line (it can be any blank area) and did Ctrl+F to activate the normal find dialog.
I see that Find what is populated and that it has 2027 characters in the Find what field.
I click [Find Next] which selects some stuff, Esc to close the dialog, Ctrl+C to load the text I found into the copy/paste buffer, and paste that to a new line.

The search using [Find Next] matched the first 2046 characters that were in the Find-what field.

Anyway, that’s excellent that we can have regular expressions that are up to 2046 characters long and to get them into the Find-what field without using copy/paste. The procedure will be to:

Select the long regexp
Crtl+F3 or Shift+Crtl+F3
Move the caret to a blank area so that the caret is not within nor touching a word.
Crtl+F and the Find-what field will be populated.

This will be handy as something I frequently do is to load the replacement string into the copy/paste buffer, select the search pattern, Ctrl+H, tab down to the Replace with field, and Ctrl+V to fill in the replacement pattern.

That works for search patterns for up to 1024 characters and now with @guy038’s Ctrl+F3 trick I can get around the 1024 character limit and don’t need to use the copy/paste buffer to do that.

A few weeks ago I took a look at the Notepad++ source code to better understand the 1024 and 2046 character limits.

The 1024 character limit comes from the default value for the Minimum size for auto-checking in-selection setting. It’s a Notepad++ bug as that setting is unrelated to the the limit for auto-loading the selection into the Find what field. Unfortunately, the fix is not easy as the internal constant that has the 1024 is used in several different way.
The 2046 character limit seems to be either a different Notepad++ bug or a Scintilla limit or bug. The buffers that hold search and replace patters are 2028 16-bit characters long. The pattern is NUL terminated meaning we should be able to have up to 2047 character long patterns. The code has an extra subtraction somewhere that causes 2047 to be 2046. The extra subtract seems to be buried in the Notepad++ logic that’s dealing with Scintilla.

While looking at the 2046 character issue I saw that the pattern buffer uses 16-bit wide characters. I verified that you can search for up to 2046 16-bit characters such as ⛱⛱⛱...⛱⛱⛱ (U+26F1). If you search for an extended Unicode character such as 🦎🦎🦎...🦎🦎🦎 (U+1F98E or the surrogate pair \x{D83E}\x{DD8E} ) then you will be limited to 1023 characters.

I did discover some weirdness with Ctrl+F3.

Using Ctrl+F3 when the caret is within a short string such as ⛱⛱⛱⛱⛱⛱⛱⛱ seems to do nothing. The text is not loaded into the Find what field. I suspect that ⛱ is not a word character.
❽ (U+277D) is a word character but Ctrl+F3 gets confused by ❽❽❽...❽❽❽ as it seems to be selecting and searching for the entire line. I was able to do 5000 character search matches using long strings of ❽❽❽...❽❽❽. As I knew the buffers are 2024 words long I suspected there was a buffer overflow.

Alan Kilborn

@mkupper said:

The 1024 character limit comes from the default value for the Minimum size for auto-checking in-selection setting. It’s a Notepad++ bug as that setting is unrelated to the the limit for auto-loading the selection into the Find what field

It isn’t really a bug, it’s more of a historical vestige. Before the setting existed, 1024 was used for even more purposes. Think of the setting as always-existing, but at an unchangeable value: 1024.