1. The Winnowing Algorithm & Karp-Rabin Rolling Hashes
Turnitin's plagiarism detection engine is fundamentally based on the Winnowing Algorithm (Schleimer, Wilkerson & Aiken, 2003). Here is the exact internal pipeline:
- Text Preprocessing: Punctuation, capitalization, and formatting are stripped into a continuous stream of words.
- K-gram Generation: The engine divides the stream into overlapping sliding windows of length
k=5consecutive words. For a 100-word paragraph, 96 k-grams are produced. - Karp-Rabin Rolling Hash: Each 5-word k-gram is mathematically hashed into a 64-bit integer in O(1) time. For example, "we analyze the heat equation" becomes
0x7A3F29B1. - Fingerprint Selection: In windows of size
w=4hashes, Turnitin records the minimum hash value. This yields a sparse fingerprint guarantee: any matching sequence of length ≥ k + w - 1 = 8 consecutive words is mathematically guaranteed to be detected.
2. AI Detection Signals: Perplexity vs. Burstiness
Turnitin AI, GPTZero, and CopyLeaks employ statistical classifiers analyzing two key linguistic signals:
Signal A: Perplexity Scoring
AI models pick the highest-probability next token (words like "delve into", "pivotal role", "tapestry"). Text with low perplexity is flagged as machine-generated. Open AI Humanizer substitutes rare scholarly synonyms that elevate perplexity to human levels.
Signal B: Burstiness Analysis
AI sentences follow a tight normal distribution with uniform sentence lengths. Humans write with extreme variation—mixing 6-word punchy sentences with 45-word multi-clause sentences. Open AI Humanizer introduces deliberate burstiness variance.
3. Zero-Width Unicode & Claude SynthID Purging
To comply with the EU AI Act (Article 50), modern frontier models embed invisible Unicode characters (such as U+200B zero-width spaces and U+FEFF byte order marks) and C2PA provenance markers. Commercial paraphrasers often copy these bytes forward. Open AI Humanizer scrubs all invisible Unicode markers at the raw character level.