Soft Hyphen U+00AD: remove the hint, preserve the word

SOFT HYPHEN U+00AD is a conditional hyphenation hint. It is often invisible while a word fits on one line and may appear as a hyphen when the line wraps. Copying from a PDF, ebook or typeset page can carry it into search, a product code or imported data. The usual data-cleaning action is removal, but only after checking what the text is for.
A word that looks continuous
international may render as international. Exact equality and string length still include the extra code point. In a search key, slug or identifier that forbids discretionary hyphenation, removing U+00AD restores the expected value. Replacing it with an ordinary space produces inter national and changes the word.
Where the hint is useful
Editorial prose can use a soft hyphen to suggest an acceptable break in a long word. It is not guaranteed to display in every program; layout engine, language, width and CSS all matter. If the goal is to preserve a designed article, it may belong in the source. If the goal is to extract searchable words, remove the hint from a derived copy and retain the original.
Visible hyphens and wbr are different
U+002D HYPHEN-MINUS is always part of the visible data, and U+2010 HYPHEN is a typographic character. HTML <wbr> supplies a break opportunity in markup rather than inserting a character into the text value. These mechanisms are not interchangeable. Preserve hyphens that belong to spelling, names or reference numbers.
A safe workflow
- Keep the original document or export.
- Reveal U+00AD and confirm it sits inside a word.
- Decide whether the output is formatted publication text or machine data.
- For data, remove U+00AD without inserting a blank.
- Check the reconstructed word against the source and repeat the failed search.
In a batch, record the removal count and review rows containing several hints. A high count can mean the input is a typeset source and should be exported differently.
Verify more than appearance
Compare code points and make sure genuine hyphens in compounds, names and reference numbers remain. If the cleaned text will be laid out again, test wrapping at narrow and wide widths. The cleaner identifies the character and position; it cannot know the author’s intended hyphenation.
JavaScript example
const word = "inter\u00ADnational";
word.replaceAll("\u00AD", ""); // international
word.replaceAll("\u00AD", " "); // inter nationalLimit the expression to a field where the rule is documented. A broader “non-printing characters” range may also remove line breaks or language controls that the data needs.
Test in an actual layout
Render two copies of a paragraph, with and without U+00AD, at several widths. Then copy the word into search and compare code points. This demonstrates both editorial value and data risk. If an automated publishing pipeline inserts soft hyphens, create a separate search or data export rather than repeatedly cleaning the published master.