Same Quantity, Different Answer: Numerical Representation Invariance in Language ModelsSame Quantity, Different Answer: Numerical Representation Invariance in Language Models

Computer Science > Computation and Language

[Submitted on 27 Jul 2026]

Title:Same Quantity, Different Answer: Numerical Representation Invariance in Language Models

View PDF HTML (experimental)Abstract:Numerically equivalent word problems should yield the same canonical answer whether a quantity is written as a decimal, fraction, percentage, number word, scientific notation, or an exactly converted unit. We generate 3,600 exact-rational problems and 8,600 prompts spanning five identity-preserving transformation families, and evaluate five open-weight systems. After a fixed syntax audit that normalizes common answer forms without an LLM judge, canonical accuracy is 0.969-0.996, but orbit correctness falls to 0.848-0.981 and orbit invariance to 0.851-0.981; invariant-but-wrong orbits account for at most 0.003. Most of the broad strict-parser collapse arises because multiplication-form scientific notation lies outside the implemented number grammar, illustrating how evaluator interfaces can masquerade as reasoning failures. A distinct semantic pathology remains: Mistral Small 4 scores 0.699 on unit-converted inputs and produces 265 errors differing from the label by exact powers of ten. In a separate 9,000-call experiment that allocates equal calls to the compared arms, representation consensus does not outperform paraphrase consensus on a low-error subset and produces substantially more false alarms. The accompanying ancillary archive contains the frozen benchmark, evaluation and audit records, consensus raw responses, manifests, analysis code, and a one-command paper build.

Submission history

From: Ephraim Atta-Duncan [view email][v1] Mon, 27 Jul 2026 13:33:38 UTC (3,800 KB)

Ancillary-file links:

Ancillary files (details):

- MANIFEST.json

- Makefile

- README.md

- artifacts/consensus/gpt-oss-20b-mxfp4/consensus_metrics.json

- artifacts/consensus/gpt-oss-20b-mxfp4/manifest.json

- artifacts/consensus/gpt-oss-20b-mxfp4/responses.jsonl

- artifacts/consensus/mistral-small4-q4/consensus_metrics.json

- artifacts/consensus/mistral-small4-q4/manifest.json

- artifacts/consensus/mistral-small4-q4/responses.jsonl

- artifacts/consensus/qwen35-4b-q4/consensus_metrics.json

- artifacts/consensus/qwen35-4b-q4/manifest.json

- artifacts/consensus/qwen35-4b-q4/responses.jsonl

- artifacts/consensus/qwen35-9b-q4/consensus_metrics.json

- artifacts/consensus/qwen35-9b-q4/manifest.json

- artifacts/consensus/qwen35-9b-q4/responses.jsonl

- artifacts/consensus/qwen35-9b-q8/consensus_metrics.json

- artifacts/consensus/qwen35-9b-q8/manifest.json

- artifacts/consensus/qwen35-9b-q8/responses.jsonl

- artifacts/full/gpt-oss-20b-mxfp4/analysis.json

- artifacts/full/gpt-oss-20b-mxfp4/audit.json

- artifacts/full/gpt-oss-20b-mxfp4/eval.jsonl

- artifacts/full/gpt-oss-20b-mxfp4/manifest.json

- artifacts/full/gpt-oss-20b-mxfp4/metrics.json

- artifacts/full/mistral-small4-q4/analysis.json

- artifacts/full/mistral-small4-q4/audit.json

- artifacts/full/mistral-small4-q4/eval.jsonl

- artifacts/full/mistral-small4-q4/manifest.json

- artifacts/full/mistral-small4-q4/metrics.json

- artifacts/full/qwen35-4b-q4/analysis.json

- artifacts/full/qwen35-4b-q4/audit.json

- artifacts/full/qwen35-4b-q4/eval.jsonl

- artifacts/full/qwen35-4b-q4/manifest.json

- artifacts/full/qwen35-4b-q4/metrics.json

- artifacts/full/qwen35-9b-q4/analysis.json

- artifacts/full/qwen35-9b-q4/audit.json

- artifacts/full/qwen35-9b-q4/eval.jsonl

- artifacts/full/qwen35-9b-q4/manifest.json

- artifacts/full/qwen35-9b-q4/metrics.json

- artifacts/full/qwen35-9b-q8/analysis.json

- artifacts/full/qwen35-9b-q8/audit.json

- artifacts/full/qwen35-9b-q8/eval.jsonl

- artifacts/full/qwen35-9b-q8/manifest.json

- artifacts/full/qwen35-9b-q8/metrics.json

- configs/consensus_benchmark_gpt_oss_20b.json

- configs/consensus_benchmark_mistral_small_4.json

- configs/consensus_benchmark_qwen35_4b.json

- configs/consensus_benchmark_qwen35_9b.json

- configs/consensus_benchmark_qwen35_9b_q8.json

- configs/families_benchmark_gpt_oss_20b.json

- configs/families_benchmark_mistral_small_4.json

- configs/families_benchmark_qwen35_4b.json

- configs/families_benchmark_qwen35_9b.json

- configs/families_benchmark_qwen35_9b_q8.json

- data/generated/base_benchmark.jsonl

- data/generated/views_benchmark.jsonl

- data/manifests/model_hashes.json

- paper/figs/consensus_dissent.pdf

- paper/figs/fingerprint.pdf

- paper/figs/source_data.json

- paper/main.tex

- paper/references.bib

- paper/tmlr.bst

- paper/tmlr.sty

- pyproject.toml

- scripts/build_supplement.py

- src/num_equiv/__init__.py

- src/num_equiv/analyze.py

- src/num_equiv/audit.py

- src/num_equiv/consensus.py

- src/num_equiv/dataset.py

- src/num_equiv/evaluate.py

- src/num_equiv/generate.py

- src/num_equiv/paper_figures.py

- src/num_equiv/parse.py

- src/num_equiv/render.py

- src/num_equiv/reproduce.py

- src/num_equiv/run.py

- src/num_equiv/units.py

- tests/test_audit.py

- tests/test_consensus.py

- tests/test_evaluate.py

- tests/test_generate.py

- tests/test_parse.py

- tests/test_render.py

- tests/test_roundtrip.py

- tests/test_units.py

- uv.lock

References & Citations

Loading...

Bibliographic and Citation Tools

Bibliographic Explorer (What is the Explorer?)

Connected Papers (What is Connected Papers?)

Litmaps (What is Litmaps?)

scite Smart Citations (What are Smart Citations?)

Code, Data and Media Associated with this Article

alphaXiv (What is alphaXiv?)

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub (What is DagsHub?)

Gotit.pub (What is GotitPub?)

Hugging Face (What is Huggingface?)

ScienceCast (What is ScienceCast?)

Demos

Recommenders and Search Tools

Influence Flower (What are Influence Flowers?)

CORE Recommender (What is CORE?)

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.