AI detection systems get credited with abilities they don’t always have. The most common one is the idea that a detector can read hidden Unicode characters and use them to catch AI text. This post looks at AI detection and hidden Unicode specifically, and at what the available evidence says about whether the two are connected.
The short version: most AI text detectors judge writing patterns, not characters. Hidden Unicode matters a great deal in other kinds of detection, which is exactly why the two topics keep getting mixed up. Once you see where the line falls, you stop chasing fixes that were never going to change an outcome.
Can AI detection systems read hidden Unicode characters?
Most AI text detectors don’t use hidden Unicode as a signal. They estimate how predictable and uniform the wording is, then return a probability. Hidden characters may be stripped before analysis, pass through untouched, or disturb the scoring by accident, depending on the pipeline. None of those outcomes is a detection of AI authorship.
I can’t tell you what any specific commercial detector does with a zero width space, because vendors don’t publish their preprocessing. What I can do is lay out what is documented about how detectors work, and where hidden characters really are read and acted on.
What software can actually see
A reader sees glyphs on a screen. Software sees a sequence of code points and formatting instructions. That difference explains why an invisible character can be detected even when a paragraph looks completely normal.
Try this in Python:
a = "report"
b = "re\u200bport"
print(a == b, len(a), len(b))It prints False 6 7. Both strings look identical in a text editor. The second one carries a zero width space (U+200B), so a comparison fails and the length is off by one. Any program that handles text at the character level will notice the difference immediately.
Zero width characters, non breaking spaces, directional controls and variation selectors can all matter in particular contexts. None should automatically be labeled an AI watermark. Their meaning depends on where they occur and why they were introduced. A non breaking space often arrives from a word processor. A zero width non joiner is part of ordinary Persian and Urdu spelling. A variation selector turns a plain symbol into a color emoji. The LLMPulse documentation for its AI Watermark Remover treats these as contextual characters for exactly that reason, reporting them but leaving them in place unless you opt in.
How AI text detectors actually work
Most detectors measure two things. Perplexity is how predictable the word choices are to a language model. Burstiness is how much sentence length and structure vary. Human writing tends to wander. Machine text tends to run smoother and more uniform. A classifier trained on labeled human and AI samples then weighs those signals with many others.
CASRAI’s guide, How AI detection actually works, describes the process plainly. A passage is run through a language model, often a smaller open source one rather than the model that wrote the text. The scores are compared with a threshold or a trained classifier, and the output is a percentage or label that is only a statistical estimate. Grammarly’s overview, How do AI detectors work?, adds that some tools also compare text against databases of known AI output, and that both false positives and false negatives are part of the deal.
Notice what’s missing from that description. Nothing in it asks whether a zero width space is present. The signal is the wording.
False positives follow directly from the method. Any human text that happens to be predictable or uniform trips the same signals. CASRAI cites a study in which detectors flagged more than three in five essays by non native English writers as AI generated, while barely flagging essays by native speakers. OpenAI discontinued its own text classifier, according to a Coderhouse explainer, because it wasn’t accurate enough. If the people who built the model couldn’t make a reliable detector, a stray invisible character was never going to be the magic key.
Where hidden characters genuinely get detected
Here’s the source of the confusion. In security, hidden Unicode is read very carefully, because attackers use it. Researchers have shown that invisible tag characters (U+E0000 to U+E007F) and zero width characters can carry covert instructions that a person can’t see but a language model will parse. Security Arsenal’s write up on ASCII smuggling explains the technique and recommends that any system feeding raw content into an LLM strip or normalize non rendering Unicode first.
Developers building guardrails make the same call. A DEV Community post on homoglyph evasion shows how a zero width character inside a word breaks a keyword filter, and says the fix is to normalize before matching. The usual recipe strips zero width, byte order mark and bidirectional characters, then applies NFKC normalization to fold fullwidth and compatibility forms. The lookalike mappings typically come from the Unicode TR39 confusables data.
So a hidden character really can be detected. By a spam filter, a prompt injection scanner or a secret scanner, all of which care that the text has been tampered with. That has nothing to do with deciding whether an essay came from a model.
| Type of detection | What it reads | Role of hidden Unicode |
|---|---|---|
| Statistical AI text detectors | Predictability and variation in wording, classifier features | Not a documented signal. May be stripped or pass through as noise. |
| Claude watermark detection | Word choices checked against a secret key | None. Anthropic says nothing is added and there are no hidden characters. |
| Security scanners and prompt injection filters | Raw code points, usually after normalization | Central. Invisible characters are a known evasion and smuggling channel. |
| File provenance checks (C2PA) | Signed metadata attached to a file | None. It’s metadata, not characters in the text. |
Is a watermark detector a hidden character detector?
No. A watermark detector is a different tool from an AI text detector, and neither one scans for invisible characters. Anthropic’s explainer, How Claude’s text watermarking works, says Claude’s watermark adds nothing to the text and uses no hidden characters. It sits in word choices that only someone with Anthropic’s key can check. The same page separates this from detection software like Pangram, which has no key and instead looks at the phrasing habits that often give AI writing away.
Watermark detection is also limited in who can run it. According to Anthropic’s support page on how Claude marks AI-generated content, the detection API is in private preview for eligible organizations. I covered the full picture in How Anthropic Marks AI Content: A Guide to Claude Watermarks.
Does removing hidden characters change a detection result?
Not reliably. Suppose a detector does react to an odd character in one pipeline. Removing it can shift a score by a few points, but the score came from your wording, and cleanup doesn’t touch your wording. BleepingComputer’s report on watermark removers makes a related point. Stripping hidden characters works and can be counted, but the scores some removers show are measured against ordinary AI detectors, not against the real watermark, and that says little about whether the watermark is gone.
There’s a second trap. Because detectors are probabilistic, running the same text twice through two tools can give two different answers. A change after cleanup might be noise. If you test, change one thing at a time, use the same text, and treat small movements as meaningless. And never treat a detector result as a verdict about a person. Grammarly’s own guidance says false results are possible in both directions.
A practical checklist
| Stage | My check | Why it matters |
|---|---|---|
| Original | Keep an untouched copy before making changes. | Lets you compare before and after, and undo a bad cleanup. |
| Technical | Inspect characters, whitespace, encoding and formatting. | Finds what actually breaks search, scripts and shortcodes. |
| Editorial | Check accuracy, clarity, tone and usefulness. | This is where wording changes, and wording is what detectors score. |
| Privacy | Consider whether the material is appropriate for the processing environment. | Pasting client drafts into unknown tools sends them to someone’s server. |
| Publication | Preview the final page and test links and formatting. | Catches mangled quotes and broken links before readers do. |
For the technical step, this short script lists invisible format characters, unusual spaces and variation selectors, with their Unicode names:
import unicodedata
text = open("draft.txt", encoding="utf-8").read()
for i, ch in enumerate(text):
odd = unicodedata.category(ch) in ("Cf", "Zs") and ch != " "
if odd or 0xFE00 <= ord(ch) <= 0xFE0F:
print(i, hex(ord(ch)), unicodedata.name(ch, "UNKNOWN"))Read the output before deleting anything. A few non breaking spaces are boring. A long run of tag characters in a draft you didn’t write is not, and that’s a security question rather than an AI detection one.
Key takeaways
- Most AI detection relies on statistical writing patterns, not hidden characters.
- Hidden Unicode is detected seriously in security work, which is why people assume AI detectors read it too.
- Hidden Unicode may be stripped before a detector ever sees it, or pass through as noise.
- Removing hidden characters does not reliably change a detection result, because your wording stays the same.
Frequently asked questions
Do AI detectors actually read hidden Unicode characters?
Generally no. Documented detectors score predictability and variation in language. What a particular product does with an invisible character before scoring isn’t published, so treat any claim about it as unverified.
If I remove hidden characters, will that fool a detector?
Not reliably. Detection is based on writing style, which cleanup doesn’t change. Any score movement after cleanup is as likely to be noise as signal.
Why do hidden characters get discussed alongside AI detection so often?
Because both are invisible to a casual reader, which makes them easy to conflate. They also overlap in security, where invisible characters are scanned for on purpose. That’s a separate job from judging authorship.
Can hidden characters be used to attack an AI system?
Yes. Invisible tag characters and zero width characters have been used to hide instructions inside text that people can’t see but models may read. That’s why security guidance says to strip or normalize them before text reaches a model.
Should I strip every invisible character I find?
No. Some have honest jobs, including joiners in emoji and non joiners in Persian and Urdu. Clean English body text freely, and be careful with multilingual content.
Final thoughts
Hidden Unicode and AI detection are two separate topics that get lumped together. One is about what characters sit inside a string. The other is about how a passage reads. Understanding the difference prevents chasing a fix that was never going to change the outcome, and it points you to the work that actually matters: clean text, accurate content and honest disclosure where it’s required.