November 2026 · methodology 2.0 · post-run policy revision
How the November humanizer benchmark is scored
One frozen set of 50 English inputs, 15 humanizers, 750 delivered outputs, five detector results per output, and eight source-to-output quality metrics. Three headline scores keep bypass, quality and the combined result visible.
Three headline scores
Humanization: for each output, add the weights of detectors it passes. Then average over all 50 outputs. A mixed, AI or uncertain verdict fails. We do not blend vendors’ raw confidence and text-share percentages.
Quality: sum the eight normalized quality metrics multiplied by their weights. Overall: 0.70 × humanization + 0.30 × quality for each output, then average. The additive formula is deliberately humanization-first. For example, 80 humanization and 90 quality gives 83 overall. Quality-only performance can still earn up to 30 overall points; this is not a hard quality gate.
Detector models, pass rules and justification
The percentages below are transparent editorial policy, not statistics proving one detector is this much stronger. Research uses different corpora, versions and thresholds. We have not calibrated these five current models to a common false-positive rate.
| Detector | Version | Weight | Pass mapping |
|---|---|---|---|
| Pangram | 4.1 | 40% | Native Human classification only; Mixed and AI fail. |
| GPTZero | 5.0b / 2026-11-04-base | 30% | predicted_class=human; mixed and ai fail. |
| Originality.ai 5.0 | model-5.0 | 20% | Native prediction_label=human; mixed and ai fail. |
| Winston AI | 5.2 | 5% | Document human score≥80; explicit benchmark mapping, not a native categorical API verdict. |
| ZeroGPT | Vendor-managed; no public version identifier | 5% | Unqualified Human Written or Most Likely Human written feedback passes. Mixed, uncertain and human labels qualified by possible AI fail. |
Pangram
Human score is the fraction of text classified human, not document confidence. Largest policy weight because independent research includes edited-AI detection evidence. This exact 40% is editorial, not a measured current accuracy. Supporting source.
GPTZero
Human score is a class probability, not the percentage of human-written words. A substantial second signal, informed by independent robustness and false-positive research; current-version shared calibration remains undone. Supporting source.
Originality.ai 5.0
Displayed human score is 100 minus AI-assisted percentage; the native label determines pass. Retains the 20% brand budget. Turbo was removed and Max was not run; no duplicate vendor model weight. Supporting source.
Winston AI
Winston returns a document Human Score. Sentence scores are not averaged to decide a pass. Reduced 15% to 5% after this run at the benchmark owner’s request because most outputs passed. This is a disclosed post-run choice, not independent validation of detector weakness. Supporting source.
ZeroGPT
Human score is 100 minus fakePercentage, a text-share estimate. It is not GPTZero. Small complementary signal; independent studies raise threshold and false-positive concerns. Account access does not establish calibration. Supporting source.
Jabarian and Imas’s Artificial Writing and Automated Detection (NBER working paper 34223, 2025) compares Pangram, GPTZero, OriginalityAI and RoBERTa across models, genres, lengths and humanizer treatments. Its false-positive/false-negative policy framework supports giving independently tested robustness substantial importance. It does not evaluate our complete five-detector November panel or establish a 40/30/20/5/5 split. Current versions require fresh calibration. Winston and ZeroGPT are outside that study’s commercial panel.
The ACL 2025 edited-text study and RAID (ACL 2024) inform the design; they do not validate these exact weights or establish the accuracy of our current model versions. GPTZero’s vendor response explains why its class probabilities and classifications should be used instead of the deprecated sentence-average field. It is not an independent replication.
Eight quality metrics
Content fidelity receives 50% in total: meaning 20%, factual consistency 10%, hallucination-free 5%, completeness 10% and stance 5%. Fluency receives 35%: readability 25% and grammar 10%. Structure receives 15%. These choices prioritize preserved meaning, retained details and usable prose; they are not statistically established consumer preferences.
| Metric | Weight | What it measures |
|---|---|---|
| readability | 25% | Naturalness, clarity and fluent phrasing. |
| meaning | 20% | Preserves the source message, argument and key points. |
| structure | 15% | Logical progression, paragraphing and coherence. |
| factual consistency | 10% | Preserves claims, names, numbers and relationships. |
| completeness | 10% | Retains substantive information and qualifications. |
| grammar syntax | 10% | Introduced grammar and syntax errors. |
| Hallucinations | 5% | Avoids unsupported additions. |
| stance preservation | 5% | Preserves position, certainty and intent. |
How to read the quality scores
Each delivered output is assessed automatically against its frozen source on the eight dimensions above. Dimension scores are expressed on a 0–100 scale and combined using the published weights. These grades reflect the evaluation rubric; they are not probabilities that a text is correct or a substitute for human editorial judgment.
The public release includes metric definitions, weights and per-output scores so readers can recompute the aggregate results. Proprietary evaluation instructions and implementation details are not part of the public release. The published scores support verification of the arithmetic, not independent reproduction of the underlying quality assessments.
All outputs received completed quality assessments. 602 of 750 outputs retain evidence-review flags; these are not 602 confirmed bad outputs. The automated scores remain in the ranking, and no independent human adjudication is claimed. Quality and overall results are provisional pending review.
Inputs and collection
The 50 frozen inputs were generated once through OpenRouter from five model families, 10 texts per model. Models: anthropic/claude-sonnet-5.5, deepseek/deepseek-v4.1-flash, google/gemini-4-argon, openai/gpt-6.1-sol, x-ai/grok-4.7. Topics, genres, exact prompts, returned model identifiers, word counts and SHA-256 hashes are in the frozen input CSV. The same exact source text went to every humanizer; source word counts are not uniform 500-word samples.
Eleven services were collected through APIs. Grammarly Pro, QuillBot, Solvely and Clever Humanizer were collected in their web interfaces. Grammarly Pro exports apply only its generated Humanizer rewrites, not unrelated proofreading edits. Phrasly used Ultra Max. Ryne was not included. The recorded settings on each service page describe the actual requests; we do not claim a uniformly strongest mode across incompatible products. No API/manual collection equivalence was independently validated.
Why Ryne is not included
The benchmark operator removed Ryne following weak prior performance and reports that its latest model remained detectable in preliminary testing for this edition. That preliminary observation is operator-reported; no reproducible preliminary scan record is published here. It is not scored as a failed November submission.
The published history does not show zero passes in both earlier runs: September 2026 recorded 59 of 250 detector checks passing (23.6%); October recorded 2 of 300 (0.67%). October passes were on ZeroGPT only. These historical thresholds and detector panels differ from November. September results · October results.
Retries, missing results and reproducibility
Collection used durable checkpoints and saved request/response receipts. Providers advanced independently within rate limits. Valid outputs and detector results were reused; only failed or missing results were retried. ZeroGPT returned five transient business errors, then succeeded; Originality returned one charged response without an AI result, which was retained and retried. No valid detector score was replaced. All 750 outputs have five valid detector results.
Length ratios remain diagnostic rather than an exclusion rule. All 50 outputs per service contribute to each mean. Raw humanized text and account-level receipts are not published on this site; the public metric export contains frozen text hashes, native classifications, quality dimensions and per-output composites. Account identifiers, API keys and balances are excluded.
What the benchmark can establish
This is an English-only 50-input comparison, not an independently calibrated measure of detector accuracy or a guarantee of academic acceptability. No held-out verified-human corpus was used to estimate false positives; no November source-only detector baseline was run. Detectors can be correlated. Results apply to these inputs, settings and recorded versions. Phrasly and StealthGPT use different modes from some earlier editions, and the weighting and quality suite changed; monthly scores are not directly comparable.
The full panel’s bypass/quality correlation is approximately −0.09. Individual tradeoffs exist; a universal inverse relationship and “only StealthGPT bucks the trend” are not supported. This site is built by the StealthGPT team, whose product is included.
Version history and post-run decisions
The earlier proposal used Pangram 35%, GPTZero 25%, Originality 20%, Winston 15% and ZeroGPT 5%, with Originality initially split between Turbo and 5.0. Turbo was removed. After seeing results, the owner reduced Winston to 5% and requested whole-number weights: Pangram 40%, GPTZero 30%, Originality 20%, Winston 5%, ZeroGPT 5%. Overall changed from bypass×quality/100 to the final 70/30 weighted sum. These changes were made after results were visible and are not described as preregistered, statistically calibrated or validated by the cited literature. Earlier saved scoring versions remain separate.
Future editions should estimate weights on disjoint human-calibration and held-out human/AI validation sets, using a common false-positive policy, sensitivity minus false positives, uncertainty bounds and normalized strengths. Freeze those weights before humanizer results are viewed. That validation is a future protocol—not something performed here.
Previous methodology (October 2026)
Verify and download
The website recalculates pass decisions, all eight metric weights, bypass and overall scores from the 750-row metric snapshot at build time and rejects mismatches. Public source and metric-file hashes appear in the manifest.
Verify the results without publishing humanized text
Following the earlier GitHub editions, the public report includes source texts, anonymized scores and summaries, while humanizer output text stays private. November also includes 750 source/output hash commitments and 15 per-humanizer result files. The saved outputs and all five detector tables were privately cross-checked before this release.
The public verifier checks the released file hashes, 50 frozen inputs, all 750 humanizer/sample pairs and every published score aggregate. Hash commitments establish consistency with retained evidence; they are not vendor-signed proof of origin or independent validation of the private quality judgments.
- GitHub report and per-humanizer results · snapshot
e96593a5 - Download the public verification bundle
- 750 output commitments CSV · File-integrity manifest · Sanitized provenance
- Read the standalone verifier · Python standard library only, no API keys or paid calls needed
After downloading the bundle:
unzip verification-bundle.zip
cd humanizereval-november-2026
python3 scripts/verify_november_report.py reports/2026-11-17-humanizer-comparisonHumanized output bodies, credentials, raw vendor responses and production quality instructions are excluded from this bundle.
Per-output scores CSV · Frozen inputs and prompts CSV · Scoring manifest · Verification bundle · GitHub report · Earlier editions