devwithjev
— reading now— views
Submit a build

Jev Hallucination and Confidence Test

邦法 tested Jev with yes/no questions and confidence scores to assess whether it hallucinated. Jev consistently answered “no” to a question about a manager’s message and gave lower confidence for questions with no answer in the material.

View on X
邦法@bangbuilds𝕏
拿到了 Jev 的 key，第一件事：测它会不会瞎编，也就是常说的 AI 幻觉。 它不写字，只回答「是 / 否」，外加一个把握。 我给它看一句话： 晚上 11 点，老板发微信：「明天上午的会你不用来了。」 问它：他是不是要被开除了？ 问 20 遍：20 次都说「不是」，把握 27%～32%，前后一致。 换 6 种问法：还是都说「不是」，但把握从 27% 跳到 45%。 问资料里根本没有的事：没瞎编。有答案的答 99%，没答案的只给 23%、57%。
Sep 21, 2026X postsView on X
The manager-message question was repeated 20 times, and Jev said “no” each time with 27%–32% confidence. Across six rephrasings, it still answered “no,” while confidence ranged from 27% to 45%. The author also reports 99% confidence when the material contained an answer, versus 23% and 57% for questions it could not answer.

Also filed under Tools & apps