Find Repeated/Duplicate Text

[mikapanja]

Is there a way to find (and possibly highlight) repeated word groups in ePubs?

[theducks]

try
(\w+)\s\1

[mikapanja]

Originally Posted by theducks

try
(\w+)\s\1

No go. It only finds adjacent words of which the first one ends in and the second one starts with one or more identical letters, e.g. go on or he headed back...

[theducks]

Originally Posted by mikapanja

No go. It only finds adjacent words of which the first one ends in and the second one starts with one or more identical letters, e.g. go on or he headed back...

a space after should stop that. I added a starting space required as well (this won't find if the first and second word are the match

\s(\w+)\s\1\s

[mikapanja]

Originally Posted by theducks

a space after should stop that. I added a starting space required as well (this won't find if the first and second word are the match

\s(\w+)\s\1\s

This one finds identical words adjacent to one another, e.g. my my. I'd like it to find non-adjacent repeated word groups, i.e scattered throughout the text.

Sorry if I was unclear.

[BetterRed]

@mikapanja - Do you want to find and highlight all occurrences of let's say: "cathode ray tube" ?

BR

[theducks]

Actually, Spell check can do that.
remove the tick for only misspelled words
Double click on the word (left side),
repeat it (the d-click) jumps to the next occurrence

[mikapanja]

Originally Posted by BetterRed

@mikapanja - Do you want to find and highlight all occurrences of let's say: "cathode ray tube" ?

BR

Exactly, BR, word groups, not words. Hence the spell check, as suggested by theducks, won't work.

[BetterRed]

Sigil's PageEdit might be a better starting point, it's a lightweight wysiwyg content editor. I'll have a look-see

BR

[BetterRed]

Originally Posted by BetterRed

Sigil's PageEdit might be a better starting point, it's a lightweight wysiwyg content editor. I'll have a look-see

Marginal, probably not worth bothering with.

But, let's say you could highlight all the "cathode ray tubes", what would you do next?

BR

[kovidgoyal]

There's no such functionality. You would need to write a plugin for it.

[mikapanja]

Originally Posted by BetterRed

But, let's say you could highlight all the "cathode ray tubes", what would you do next?

Even finding the group would be more than enough.

It would also be useful to be able to define the group size (say, 3-5 words, or 3 words and larger).

Originally Posted by kovidgoyal

There's no such functionality. You would need to write a plugin for it.

I was hoping one of our resident experts could come up with a relevant regex or function. I know it's doable in Word via macro, albeit on a sentence level.

[BetterRed]

Originally Posted by mikapanja

I know it's doable in Word via macro, albeit on a sentence level.

No need for macros in Word, standard Ctrl+H Find & Replace on entire document or selection e.g. paragraph, page, section etc, viz:



And when you've done, set Highlight to None and repeat F&R.

BR

[mikapanja]

Originally Posted by BetterRed

No need for macros in Word, standard Ctrl+H Find & Replace on entire document or selection e.g. paragraph, page, section etc, viz:

Attachment 208356

And when you've done, set Highlight to None and repeat F&R.

That works in case you know what you are looking for. But what if you don't, which was my original thought? E.g. if you want to know if any 5-word group is repeated in the text.

[Doitsu]

Originally Posted by mikapanja

That works in case you know what you are looking for. But what if you don't, which was my original thought? E.g. if you want to know if any 5-word group is repeated in the text.

For those kind of searches you'll need to use a concordance tool. For example, Laurence Anthony's AntConc (freeware).

• Unzip the epub file.
• If the folder contains .xhtml files change their file extensions to .html.
• Open AntConc, select Open file(s) as 'Quick Corpus', select .html as the file type and then select the extracted .html files.
• Click the N-Gram tab, select the desired number of words and click Start.

I've attached a sample screenshot of the output.