Search++ (\W)'(\w) regex replace failure

[Coises]

@guy038 said:

I didn’t say that anormalous characters appear. I just said that no replacement occurs at all !

You might be confusing @m-andre-z-eckenrode’s topic (\W)'(\w) Regex replace failure, about Notepad++, and this topic, about Search++, with the same search.

---

He was surprised that Notepad++ found the text but would not replace it. The cause is how Notepad++ treats CRLF pairs combined with how it decides whether Replace should replace or find next — as I wrote there, whether it is a bug is, I suppose, a matter of opinion.

Search++ has a different potentially user-unfriendly behavior: with different find or replace expressions (any regular expression that can match starting with an LF, combined with a replacement string that doesn’t copy the first matched character as the first character of the replacement) a user could invisibly change line endings from CRLF to just CR (though if the user understands the regular expression and replacement entered, it would be expected).

Notepad++ has the same behavior as Search++ when using Replace All (including that it can replace the LF in a CRLF pair with something else); it’s just step-by-step replace that can find but fail to replace (for the same reason that \K doesn’t work in step-by-step native searches).

---

In this topic, the problem is that Search++ is, apparently randomly, replacing with garbage instead of the correct replacement string.

I have not yet found a possible cause. It does not happen on my machine.

[M Andre Z Eckenrode]

@Coises said:

I didn’t think to mention that quirk. If you start with a rectangular selection, Columns++ searches each row independently

Aha. That explains my own confusion. That was actually the first time I’d used the Search feature of Columns++. For the record, any time I’m doing regex search & replace in NPP that either turns out to be more complicated than I expected, or involves some trial and error, or which I’m going to want to utilize again elsewhere (such as this topic), I’m in the habit of typing it out right in the text document I’m working on, and then I select/copy the replacement regex, then select the find regex and open the necessary dialog, expecting the Find what to have been automatically populated from my most recent selection (which, of course, didn’t happen in that case). It didn’t occur to me that that wouldn’t work in Columns++. So, when I opened Columns++’s Search dialog with only the content of one line selected and tried to perform my regex search, it threw this at me:

This command requires a rectangular selection. Extend selection to the last line of the document?

And, Columns++ Search neophyte that I am, I didn’t heed the suggestion. I closed the dialog and created a rectangular selection.

Now that I’ve been made aware of the error of my ways, I’ve redone the experiment with all the my example text selected, and all instances of ' were successfully replaced with ’, without any anomalies.

[Coises]

@M-Andre-Z-Eckenrode said:

This command requires a rectangular selection. Extend selection to the last line of the document?

And, Columns++ Search neophyte that I am, I didn’t heed the suggestion.

Had you heeded it, you would still have gotten a rectangular selection, as it says, extending downward from your selection — not necessarily enclosing the whole document, and in any case not the same as selecting the whole document (or selecting nothing at all) first.

I should re-word that message, I’m just not sure what to say instead. It’s not actually true (since version 0.8) that a rectangular selection is required; as I mentioned before, the rules are complicated. They won’t easily fit into a message box. It makes sense when you’re using Columns++ for working with columns, but the search kind of took on a life of its own, apart from column work… hence, among other reasons, Search++.

I’ve redone the experiment with all the my example text selected, and all instances of ' were successfully replaced with ’, without any anomalies.

Thank you for that information. I do appreciate the help you’ve given me with this. While it doesn’t prove anything (almost nothing is ever proven when dealing with an intermittent bug), it suggests pretty strongly that something I changed in moving the Columns++ search process to Search++ has introduced a hidden instability. I’m still looking for it.

[M Andre Z Eckenrode]

@Coises

On the outside chance that specifics of the variants of my regex replacement anomalies are useful, here is some detailed information about them and the circumstances of their appearance… I started with an unsaved ANSI text tab/buffer with:

Search++ Regex

(\W)'(\w)
\1’\2

this
'Cos I am
that

this
'Round and 'round
that

------------------------------------------------------------------

… and the example text (this[CRLF]'Cos to 'round[CRLF]that followed by [blank line]/[dashed line]/[blank line]) repeated 9 times below (total of 10 times). The anomalies occurred on the following lines during this first experiment:

Line 27:
-’-os I am

Lines 51–52:
’Round and
’hound

Lines 62–63:
a’
ound and ’round

Second experiment:

Line 11:
-’-ound and ’round

Line 27:
-’-os I am

Line 57:
-’-os I am

Lines 87–88:
[blank line]
’tos I am

Third experiment:

Line 10:
this-’-ound and ’round

Line 16:
-’-os I am

Line 40:
-’-ound and ’round

Line 60:
’Round and-’-ound

Line 90:
’Round and-’-ound

[Coises]

@M-Andre-Z-Eckenrode said:

here is some detailed information

Thank you. At this point I am still stumped as to how to find this needle in a haystack. Any information could turn out to be helpful. I appreciate your perseverance.

[Coises]

@M-Andre-Z-Eckenrode:

Have you ever noticed arbitrary junk character replacements with any other regular expression / replacement pairs, or only with this particular one?

[M Andre Z Eckenrode]

@Coises said:

Have you ever noticed arbitrary junk character replacements with any other regular expression / replacement pairs, or only with this particular one?

I haven’t actually tried any others with Search++, so far. The vast majority of my regex operations are accomplished via PythonScript, and I only very infrequently need to perform a standalone regex. But I’ll keep Search++ in mind if/when another one becomes necessary.

[mpheath]

@M-Andre-Z-Eckenrode I may not fully understand the regex task with using groups when ' can be at the start of a line with a word character following as to \W can match the preceding LineFeed character. The LineFeed character seems unwise to capture in my opinion. \W can change behavior depending on encoding so might be challenging to use so perhaps if can be avoided, then that might be better.

The find pattern of \B'\b and replace with ’ might be safer to use as to using anchors to qualify the match instead of consuming with groups which may need more complex conditional handling.

[guy038]

Hello, @m-andre-z-eckenrode, @coises, @mpheath and All,

@mpheath, you said :

The find pattern of \B'\b and replace with ’ might be safer to use as to using anchors to qualify the match instead of consuming with groups which may need more complex conditional handling.

I totally agree with this statement !

That is why I proposed, in my initial post, this alternate formulation :

FIND (?<=\W)'(?=\w)

REPLACE ’

which do not contain any group !

---

Indeed, the \B'\b and (?<=\W)'(?=\w) Find regexes are equivalent and a Replace All operation would change this INPUT text, below :

't'e's't' 't'o' 's'e'e' 't'h'e' 'm'a't'c'h'e's'

By this one :

’t'e's't' ’t'o' ’s'e'e' ’t'h'e' ’m'a't'c'h'e's'

In other words, any single quote char, preceded by a non-word char and followed with a word char, is changed into the ’ character ( \x{2019} RIGHT SINGLE QUOTATION MARK )

Best Regards,

guy038

[M Andre Z Eckenrode]

@mpheath & @guy038

Thanks much for the suggestions, though as stated in my other similar topic, which isn’t specific to Search++, I already found and used a working alternate method to achieve what I wanted, and only really started that topic to try to understand why the (\W)'(\w) method did NOT work, while THIS topic is only really to report the aberrant replacements seemingly caused by Search++.

[M Andre Z Eckenrode]

@Coises

For what it’s worth, I asked my brother, whom I know also uses Notepad++, to indulge me by repeating my Search++ regex experiments on his own computer, and he kindly did so after encountering some initial problems getting Search++ to work at all (see below for details of that). Anyway, he used a fresh portable copy with the latest versions of both, as I have, and reports getting one anomalous replacement while stepping through a text field made up of 11 copies of my recently reported example text. I think that duplicating the problem almost invariably requires many matches and replacement attempts, since it’s so hit or miss for whatever reason.

About my brother’s initial failures to get Search++ to load in his portable copy: His first four or so tries were attempted in different virtual machines on his computer, including the very first one being Windows 11’s built-in “Sandbox”. All of them failed with messages such as “The specified module could not be found” and “Search++.dll is not compatible with the current version of Notepad++”. But when he tried the same set of files on his real machine, it worked. He routinely runs various other programs in virtual machines just fine.

[Coises]

@M-Andre-Z-Eckenrode said:
For what it’s worth, I asked my brother, whom I know also uses Notepad++, to indulge me by repeating my Search++ regex experiments on his own computer, and he kindly did so after encountering some initial problems getting Search++ to work at all (see below for details of that). Anyway, he used a fresh portable copy with the latest versions of both, as I have, and reports getting one anomalous replacement while stepping through a text field made up of 11 copies of my recently reported example text. I think that duplicating the problem almost invariably requires many matches and replacement attempts, since it’s so hit or miss for whatever reason.

Thank you, and thanks to your brother!

Is he also using 32-bit Notepad++?

About my brother’s initial failures to get Search++ to load in his portable copy: His first four or so tries were attempted in different virtual machines on his computer, including the very first one being Windows 11’s built-in “Sandbox”. All of them failed with messages such as “The specified module could not be found” and “Search++.dll is not compatible with the current version of Notepad++”. But when he tried the same set of files on his real machine, it worked. He routinely runs various other programs in virtual machines just fine.

I just tried the same thing in Windows 10 Sandbox on my system and it also failed, with the same message. I will have to investigate that. Off the top of my head, I don’t know why that would happen. My guess (that’s all it is for now) is there’s some sort of C++ runtime dependency that nearly everyone has installed on their main machines, but that doesn’t carry over automatically to a virtual machine or a sandbox.

[M Andre Z Eckenrode]

@Coises said:

Is he also using 32-bit Notepad++?

His routinely-used copy is v8.9.3 x64 portable.

[Coises]

@M-Andre-Z-Eckenrode said:

About my brother’s initial failures to get Search++ to load in his portable copy: His first four or so tries were attempted in different virtual machines on his computer, including the very first one being Windows 11’s built-in “Sandbox”. All of them failed with messages such as “The specified module could not be found” and “Search++.dll is not compatible with the current version of Notepad++”. But when he tried the same set of files on his real machine, it worked.

At least I was able to figure out the cause of this problem. The ICU (International Components for Unicode) libraries that I use require a specific Microsoft redistributable library. I never noticed, and most people probably have it installed already as part of installing something else, just as I do. It doesn’t automatically carry over to virtual machines or sandboxes, though.

There are a couple potential solutions (and if nothing else works out, I can at least explain in the readme how to install the redistributable). Thank you for reporting this.

[M Andre Z Eckenrode]

@Coises said:

At least I was able to figure out the cause of this problem.

Glad to hear that worked out.

@Coises said:

Have you ever noticed arbitrary junk character replacements with any other regular expression / replacement pairs, or only with this particular one?

I just tried a somewhat different regex experiment. I copied a list of words beginning with the letter a from a web page and pasted them into N++, then utilized Find (\W)a(\w) and Replace \1z\2 and stepped through. Out of 221 words, 21 replacements resulted in letters other than z being substituted for a. In a few cases, there was a letter other than z, followed by z itself, and followed again by another non-z letter. In a few other cases, extra line breaks were inserted mid-word. Note that in this experiment, no dashes (-) were inserted, as they often were in my previous experiments involving ' and ’. Presumably, that’s some sort of clue.

[M Andre Z Eckenrode]

@Coises

And just now tried Find (\w)a(\w) and Replace \1z\2 with the same list of words, which definitely resulted in at least one anomalous replacement (average became avevzie), possibly more, but the mistakes are more difficult to spot at a glance in this case.

[Coises]

@M-Andre-Z-Eckenrode said:

@Coises

And just now tried Find (\w)a(\w) and Replace \1z\2 with the same list of words, which definitely resulted in at least one anomalous replacement (average became avevzie), possibly more, but the mistakes are more difficult to spot at a glance in this case.

I hate continuing to bother you about this with different questions, but I am so stumped. Have you noticed:

1. Do bad substitutions happen if the document is set as UTF-8, or is it specific to ANSI? (For a test, just use Encoding | Convert to UTF-8 on a document containing your test text before trying to Replace.)

2. Have you ever seen a bad replacement where the replacement string did not use a capturing group reference (like \1 or \2 or $1 or $2)?

[M Andre Z Eckenrode]

@Coises said:

Do bad substitutions happen if the document is set as UTF-8, or is it specific to ANSI?

I’ve experienced it with both at this point.

Have you ever seen a bad replacement where the replacement string did not use a capturing group reference

Not that I’ve noticed so far.

New test:

I used the same list of words, but inserted the number 2 between the second and third letters of all words that consisted of at least three letters, then stepped through using Find ([a-z])\d([a-z]) and Replace \1\2. There are anomalies. Examples include:

ac2ademic became adjdemic

ad2ministration became a
ginistration

[Coises]

@M-Andre-Z-Eckenrode

I think I may have found the cause.

Would you try installing this version:

https://github.com/Coises/SearchPlusPlus/releases/tag/v0.6.3.2

and see if the problem goes away?

If the problem is what I think it is, this will make it go away. It isn’t a good solution (it will cause serious inefficiency for large files), but if you no longer see the problem with this version, I’ll know I’ve almost certainly identified the cause, and I can work on a proper resolution.

Thank you so much for all your effort.

[M Andre Z Eckenrode]

@Coises said:

Would you try installing this version

With v0.6.3.2 in place, I re-ran several of the tests I’ve been doing, and found no anomalies this time, so that seems to have taken care of it. Thank YOU! Out of curiosity, is there any kind of basic explanation you can relay in layman’s terms for why it was doing what it was doing?