Fresh drop from r/PromptEngineering: u/Bitman321 shipped ChatJev, a small web app that lets you talk to Jev through a plain chat box. The twist is that the reply is not the answer. The scores are. Most chatbots hand you a confident paragraph and leave you to decide whether to believe it. This one hands you a paragraph and a receipt.
What’s new
You type like a normal chat, or drop in a file or a photo. Instead of just generating text, ChatJev converts what you sent into a handful of structured questions that Jev can score: yes or no, pick a choice, or a point on a scale. Jev returns scores, and an LLM turns those scores into one short reply.
Picture a few everyday cases. You paste in a rental agreement and ask whether it looks fair. ChatJev might break that into questions like “Is the deposit within a normal range?” (yes or no), “How strict is the late fee?” (a point on a scale), and “Which side does the maintenance clause favor?” (pick a choice). Or you drop in a photo of a used bike and ask if it’s worth buying. The app turns that into scored checks on visible rust, tire wear, and frame condition. You are not asking one model to improvise an opinion. You are asking a scorer to answer small, specific questions, then letting a writer phrase the result.
The twist
Under the reply sits a card that lists every score. So you can read the sentence, then check it against what Jev actually said. The chatty part is just a summary layer. The judgment lives in the numbers, and you can audit it. That’s rare in a chatbot, where you usually have to trust the vibe.
Why this matters in practice: when the sentence says “this looks mostly fine,” you can open the card and see whether that came from a row of strong scores or from one high score hiding two weak ones. A smooth sentence can round off the bad news. A list of numbers has a harder time doing that. Splitting judgment from wording means the part that sounds nice is no longer the part you have to trust.
Two more details worth noticing:
- A document goes in as text. A photo gets a short description of what’s visible, and the picture itself also goes to Jev, so it isn’t judging from a caption alone. That matters because captions drop detail. A scratch, a blurry label, or an odd color can vanish in a one-line description but still show up in the image.
- If your file doesn’t cover a question, an LLM plus a web scraper looks it up and hands the findings to Jev to judge. So a gap in your document does not mean a blank score. The missing piece gets researched, then scored like everything else.
Mini-workflow to try it 🛠️
- 🌐 Open chatjev.vercel.app
- 💬 Type a question, or drop in a file or photo
- 🔎 Read the short reply
- 📊 Open the card under it and compare each score to the sentence
- Change your input and watch which scores move
Keep your first test small. One page of text or one clear photo is plenty. Big inputs make it harder to tell which change caused which score to shift, and the whole point is to see cause and effect.
Pro tips
- Test it with a document that only partly covers your question. That’s where the web lookup kicks in, and you can see how the outside info shifts the scores.
- Treat the card as the source of truth. If the sentence sounds smoother than the numbers justify, that’s the gap worth noticing.
- Run the same question twice with one small edit, like changing a single clause in your document or swapping one photo for a slightly different angle. Then compare the cards side by side. If a score jumps for no clear reason, you have learned something about how stable the judgment really is.
- Ask questions that have a clear yes, no, or scale answer. “Is this contract risky?” gives the scorer something to grip. “Tell me about this contract” gives it less to work with.
- The code is MIT licensed on GitHub (JamesANZ/ChatJev), so you can read exactly how the question-and-score pipeline is wired. Steal the pattern of separating scoring from wording for your own prompts. In your own setup, that could mean one prompt that outputs only a JSON list of scores, and a second prompt that turns that JSON into a friendly summary.
Fair warning from the author: it’s for educational purposes only, use at your own risk. Don’t lean on it for legal, medical, or money decisions. Use it to learn how scored answers behave, then apply that thinking to your own tools.
Call to Action
Give it a spin at chatjev.vercel.app, then tell me which score surprised you most. ☠ And if you build your own scored-then-summarized chat, send it my way.
Play around with Jev as a chatbot
by u/Bitman321 in PromptEngineering