Search++ (\W)'(\w) regex replace failure
-
here is some detailed information
Thank you. At this point I am still stumped as to how to find this needle in a haystack. Any information could turn out to be helpful. I appreciate your perseverance.
-
Have you ever noticed arbitrary junk character replacements with any other regular expression / replacement pairs, or only with this particular one?
-
Have you ever noticed arbitrary junk character replacements with any other regular expression / replacement pairs, or only with this particular one?
I haven’t actually tried any others with Search++, so far. The vast majority of my regex operations are accomplished via PythonScript, and I only very infrequently need to perform a standalone regex. But I’ll keep Search++ in mind if/when another one becomes necessary.
-
@M-Andre-Z-Eckenrode I may not fully understand the regex task with using groups when
'can be at the start of a line with a word character following as to\Wcan match the preceding LineFeed character. The LineFeed character seems unwise to capture in my opinion.\Wcan change behavior depending on encoding so might be challenging to use so perhaps if can be avoided, then that might be better.The find pattern of
\B'\band replace with’might be safer to use as to using anchors to qualify the match instead of consuming with groups which may need more complex conditional handling. -
Hello, @m-andre-z-eckenrode, @coises, @mpheath and All,
@mpheath, you said :
The find pattern of
\B'\band replace with’might be safer to use as to using anchors to qualify the match instead of consuming with groups which may need more complex conditional handling.I totally agree with this statement !
That is why I proposed, in my initial post, this alternate formulation :
FIND
(?<=\W)'(?=\w)REPLACE
’which do not contain any group !
Indeed, the
\B'\band(?<=\W)'(?=\w)Find regexes are equivalent and aReplace Alloperation would change this INPUT text, below :'t'e's't' 't'o' 's'e'e' 't'h'e' 'm'a't'c'h'e's'By this one :
’t'e's't' ’t'o' ’s'e'e' ’t'h'e' ’m'a't'c'h'e's'In other words, any single quote char, preceded by a non-word char and followed with a word char, is changed into the
’character (\x{2019}RIGHT SINGLE QUOTATION MARK )Best Regards,
guy038
-
Thanks much for the suggestions, though as stated in my other similar topic, which isn’t specific to Search++, I already found and used a working alternate method to achieve what I wanted, and only really started that topic to try to understand why the
(\W)'(\w)method did NOT work, while THIS topic is only really to report the aberrant replacements seemingly caused by Search++. -
For what it’s worth, I asked my brother, whom I know also uses Notepad++, to indulge me by repeating my Search++ regex experiments on his own computer, and he kindly did so after encountering some initial problems getting Search++ to work at all (see below for details of that). Anyway, he used a fresh portable copy with the latest versions of both, as I have, and reports getting one anomalous replacement while stepping through a text field made up of 11 copies of my recently reported example text. I think that duplicating the problem almost invariably requires many matches and replacement attempts, since it’s so hit or miss for whatever reason.
About my brother’s initial failures to get Search++ to load in his portable copy: His first four or so tries were attempted in different virtual machines on his computer, including the very first one being Windows 11’s built-in “Sandbox”. All of them failed with messages such as “The specified module could not be found” and “Search++.dll is not compatible with the current version of Notepad++”. But when he tried the same set of files on his real machine, it worked. He routinely runs various other programs in virtual machines just fine.
-
@M-Andre-Z-Eckenrode said:
For what it’s worth, I asked my brother, whom I know also uses Notepad++, to indulge me by repeating my Search++ regex experiments on his own computer, and he kindly did so after encountering some initial problems getting Search++ to work at all (see below for details of that). Anyway, he used a fresh portable copy with the latest versions of both, as I have, and reports getting one anomalous replacement while stepping through a text field made up of 11 copies of my recently reported example text. I think that duplicating the problem almost invariably requires many matches and replacement attempts, since it’s so hit or miss for whatever reason.Thank you, and thanks to your brother!
Is he also using 32-bit Notepad++?
About my brother’s initial failures to get Search++ to load in his portable copy: His first four or so tries were attempted in different virtual machines on his computer, including the very first one being Windows 11’s built-in “Sandbox”. All of them failed with messages such as “The specified module could not be found” and “Search++.dll is not compatible with the current version of Notepad++”. But when he tried the same set of files on his real machine, it worked. He routinely runs various other programs in virtual machines just fine.
I just tried the same thing in Windows 10 Sandbox on my system and it also failed, with the same message. I will have to investigate that. Off the top of my head, I don’t know why that would happen. My guess (that’s all it is for now) is there’s some sort of C++ runtime dependency that nearly everyone has installed on their main machines, but that doesn’t carry over automatically to a virtual machine or a sandbox.
-
-
About my brother’s initial failures to get Search++ to load in his portable copy: His first four or so tries were attempted in different virtual machines on his computer, including the very first one being Windows 11’s built-in “Sandbox”. All of them failed with messages such as “The specified module could not be found” and “Search++.dll is not compatible with the current version of Notepad++”. But when he tried the same set of files on his real machine, it worked.
At least I was able to figure out the cause of this problem. The ICU (International Components for Unicode) libraries that I use require a specific Microsoft redistributable library. I never noticed, and most people probably have it installed already as part of installing something else, just as I do. It doesn’t automatically carry over to virtual machines or sandboxes, though.
There are a couple potential solutions (and if nothing else works out, I can at least explain in the readme how to install the redistributable). Thank you for reporting this.
-
At least I was able to figure out the cause of this problem.
Glad to hear that worked out.
Have you ever noticed arbitrary junk character replacements with any other regular expression / replacement pairs, or only with this particular one?
I just tried a somewhat different regex experiment. I copied a list of words beginning with the letter
afrom a web page and pasted them into N++, then utilized Find(\W)a(\w)and Replace\1z\2and stepped through. Out of 221 words, 21 replacements resulted in letters other thanzbeing substituted fora. In a few cases, there was a letter other thanz, followed byzitself, and followed again by another non-zletter. In a few other cases, extra line breaks were inserted mid-word. Note that in this experiment, no dashes (-) were inserted, as they often were in my previous experiments involving'and’. Presumably, that’s some sort of clue. -
And just now tried Find
(\w)a(\w)and Replace\1z\2with the same list of words, which definitely resulted in at least one anomalous replacement (averagebecameavevzie), possibly more, but the mistakes are more difficult to spot at a glance in this case. -
And just now tried Find
(\w)a(\w)and Replace\1z\2with the same list of words, which definitely resulted in at least one anomalous replacement (averagebecameavevzie), possibly more, but the mistakes are more difficult to spot at a glance in this case.I hate continuing to bother you about this with different questions, but I am so stumped. Have you noticed:
-
Do bad substitutions happen if the document is set as UTF-8, or is it specific to ANSI? (For a test, just use Encoding | Convert to UTF-8 on a document containing your test text before trying to Replace.)
-
Have you ever seen a bad replacement where the replacement string did not use a capturing group reference (like
\1or\2or$1or$2)?
-
-
Do bad substitutions happen if the document is set as UTF-8, or is it specific to ANSI?
I’ve experienced it with both at this point.
Have you ever seen a bad replacement where the replacement string did not use a capturing group reference
Not that I’ve noticed so far.
New test:
I used the same list of words, but inserted the number
2between the second and third letters of all words that consisted of at least three letters, then stepped through using Find([a-z])\d([a-z])and Replace\1\2. There are anomalies. Examples include:ac2ademicbecameadjdemicad2ministrationbecamea
ginistration -
I think I may have found the cause.
Would you try installing this version:
https://github.com/Coises/SearchPlusPlus/releases/tag/v0.6.3.2
and see if the problem goes away?
If the problem is what I think it is, this will make it go away. It isn’t a good solution (it will cause serious inefficiency for large files), but if you no longer see the problem with this version, I’ll know I’ve almost certainly identified the cause, and I can work on a proper resolution.
Thank you so much for all your effort.
-
Would you try installing this version
With v0.6.3.2 in place, I re-ran several of the tests I’ve been doing, and found no anomalies this time, so that seems to have taken care of it. Thank YOU! Out of curiosity, is there any kind of basic explanation you can relay in layman’s terms for why it was doing what it was doing?
-
@M-Andre-Z-Eckenrode said:
With v0.6.3.2 in place, I re-ran several of the tests I’ve been doing, and found no anomalies this time, so that seems to have taken care of it. Thank YOU!Thank you. This error has been there since I first designed this regular expression search technique, in Columns++. It just apparently only shows up in rare circumstances. I’ve still never seen it.
Out of curiosity, is there any kind of basic explanation you can relay in layman’s terms for why it was doing what it was doing?
Notepad++ uses an open source component called Scintilla to display editable text. Scintilla maintains the text in an internal “split buffer,” meaning that there can be a gap at an arbitrary position within the text. It uses that gap to be able to make changes that add or remove characters in the middle of the text without having to copy all the following text to a new location every time there’s one small change.
For a few reasons, when I implemented search, I wanted to use the same regular expression engine that Notepad++ uses, Boost.regex, but I wanted to include my own copy in the plugin and manage it directly, rather than going through the hybrid Scintilla/Boost interface that Notepad++ exposes, and I wanted to access the Scintilla split buffer directly. To do that, I had to create something called iterators.
In C++, an iterator is essentially an indirect way of referencing data that has simple operations like “move to the next character” and “tell me what character you’re pointed at.” I designed iterators for UTF-8, for single-byte character set ANSI and for double-byte character set ANSI that could traverse the Scintilla buffer, mind the gap, and return the results to the Boost.regex engine as if it were seeing a contiguous string of UTF-32 Unicode characters, regardless of the actual encoding.
Now, there’s a caveat when you ask Scintilla for the location of its internal buffer positions. You get two segments (because of the gap). But Scintilla warns that you can’t trust those positions once you do anything else in Scintilla, or once any user interaction is possible (even if the user doesn’t change the text). So I made sure to “invalidate” my internal pointers whenever control left my plugin: such as between finding text and changing it.
In order to ask Boost.regex for the replacement string when doing a change, I have to keep a particular Boost.regex structure that references the original match, so it knows how to interpret capture group substitutions. I was careful to invalidate the buffer pointers before I called Boost.regex.format. Except…
The match structure in Boost.regex doesn’t store the Scintilla character positions of the match. It stores iterators. And the iterators I designed, to be as efficient as possible, hold copies of the pointers into the Scintilla split buffer. So invalidating the pointers I use to generate iterators for Boost.regex does nothing to the iterators Boost.regex already stored in the match structure.
For reasons I can’t guess, both you and your brother seem to have something about your systems that causes Scintilla to reposition the gap in its internal buffer between find and replace. It never seems to happen here, but I finally realized that my attempt to protect against that happening is ineffective.
What I did to test was add a Scintilla call each time I invalidate the buffer pointers that causes Scintilla to move everything to the beginning and put the gap at the end. That means all the iterators will have the same pointers. It’s not a good long-term solution, though, because in a large file moving the gap to the end every single time anything changes could make response sluggish.
So it might take me a little while to decide on the best solution. I’m glad you documented the error so well. Now I know what to fix, I just have to work out the best way to do it.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login