Most assistants are judged on the questions their builders wrote. Murshid is an Arabic assistant over a set of Saudi government-service FAQs, and the part worth reading is the test it got afterwards. I froze its rules and ran them on 93 questions they had never seen. They took the right path 62 times, against 26 of 26 on the questions they were tuned on, and the misses show exactly where rules written from a handful of examples stop working.

Context

Murshid (مرشد, Arabic for guide) answers questions about five services: residence permits, traffic fines, sponsorship transfer, municipal permits and the labour office. People ask about them in formal Arabic, in Saudi and Gulf dialects, and in a mix of both with English words such as OTP or Absher. With a model behind it, it answers in the asker's register; either way it quotes its sources word for word.

The documents are invented: 20 of them in 84 passages, where every fee, rule, date and reference number is made up. It started as a time-boxed brief in May 2026, and in November I came back to it to measure it properly.

The constraint

For official procedures, a wrong answer and a wrong refusal both cost. Someone told the wrong deadline pays a fine. Someone refused an answer the documents contain goes to an office for nothing. And a language model's sense of when to refuse moves with the wording of the question.

So the behaviour is decided before any model is called. Every question ends one of four ways: an answer, a partial answer that says what to take elsewhere, a refusal that says where to go instead, or a question back. Rules pick which.

The decisions that mattered

  • Rules decide, a model only writes. Keyword rules send the question to one of the five services, refuse it when it strays into finance, medicine or religious rites, or ask which service is meant. A short date with no calendar marker gets a question back, because the same digits mean different days in the Hijri and Gregorian calendars. A support gate refuses "I heard that…" and "is there a special exemption…" questions whose specific terms appear in no retrieved passage, before any model sees them.
  • Retrieval built for Arabic, not translated into English. Each question is searched three ways: as typed, lightly normalised (letter variants, elongation and diacritics removed, nothing more) and rewritten into formal Arabic. BM25 and BGE-M3 embeddings each rank the passages, and the rankings are fused. Normalisation stays light on purpose: collapsing more letter forms can help rough keyword matching, but it damages legal wording and proper names.
  • Loanwords are not dialect. "iqama" or "OTP" in a question says nothing about how formally someone writes, so a short list of such words is ignored when the register is detected.
  • It runs without a key. Besides Claude and OpenAI, an extractive mode answers by quoting the retrieved passages. The whole pipeline, its tests and the test below run on it, for free.
  • The second opinion went off. A critic, the same model checking its own answer, was meant to catch replies the sources do not support. In May, across both providers on the 26 tuned questions, it overruled 11 answers, and all 11 should have stood. It is off in the recommended setting.
A question in formal Arabic, dialect or a mix is routed by rules to one of five services, or refused, or answered with a question back; retrieval ranks passages with BM25 and BGE-M3; a rule-based support gate refuses bait whose terms no passage contains; only the last step, the answer, is written by a model or by the extractive mode.Questionformal Arabic, dialect, or a mixRoute, by rulesone of five services, refuse, or askRetrieveBM25 and BGE-M3, three viewsSupport gate, by rulesrefuse bait no passage supportsAnswera model, or quotes with no model
How a question is answered. Rules decide the behaviour in the first steps. A model writes only the dashed one, and the extractive mode can write it with no model at all.

What it looks like

The recorded runs from May are a page of their own: each of the 26 tuned questions, what each provider did with it, and the answers as they were saved. Nothing on the page calls a model. Those runs used a prompt whose two examples held the answers to two of the questions; the page says so, and no number in this article rests on their quality scores.

The top of the recorded-runs page: a banner saying the answers were recorded on 23 May 2026 with an earlier system prompt and that the corpus is synthetic, then a table of how often each provider took the expected behaviour: Claude 26 of 26 with the critic off and 17 with it on, OpenAI 26 of 26 off and 24 on.
The top of the page. Both models took the expected path on all 26 tuned questions with the critic off. With it on, they lost 9 and 2.
A Gulf-dialect question about transferring a sponsorship after a contract ends on 20 Sha'ban 1447 AH, its gold answer, and three recorded answers: Claude with the critic off answers in dialect and quotes the source, Claude with the critic on refuses, and OpenAI with the critic off answers.
A question in Gulf dialect about moving a sponsorship once a contract ends on a Hijri date. With the critic off, both models answer from the same passages, Claude in the asker's dialect. With it on, both answers were overruled into refusals.
A question asking whether residents over sixty working at small firms get a special exemption from iqama renewal fees. All recorded runs refuse it and point to the responsible office, and each shows the note support_gate_fired: special_exemption.
A question baiting an exemption the documents never mention. The support gate refused it before any model was called, in every run.

The test it was not tuned for

Rules written from 26 examples will pass those 26. That number measures fit, not whether the rules work. So in November an AI agent wrote a second set of 68 questions and 25 adversarial cases from the documents, without seeing the rules or the code. I froze the rules, tagged the commit, and ran both sets with no model in the loop.

Expected outcomeTuned setUnseen set
An answer16 of 1650 of 59
A question back3 of 39 of 10
A partial answer and escalation2 of 21 of 10
A refusal5 of 52 of 14
All26 of 2662 of 93

62 of 93 is 67%, and with this few questions the honest range is 57% to 75%. Answers held up. The four questions written to tempt an over-cautious system into refusing were all answered. Refusals and escalations collapsed, and the assistant asked a question back 22 times when 13 of those called for something else. By register, it handled 20 of 27 unseen questions in formal Arabic, 19 of 30 in dialect and 6 of 11 in a mix. With no model in the loop the step that rewrites dialect into formal Arabic is skipped; the paid run below does rewrite, and all 119 cases took the same path there as here.

What broke, and why

The misses are not noise: the report names the rule behind each one, and two patterns account for 25 of the 31. When the router recognises no service, it asks which one is meant, so questions about a driving licence, a family visit visa or a recruitment agency got a question back instead of a refusal, and so did six questions it should have answered, at least in part. That is 12 misses.

When no refusal or escalation rule fires, the question is answered. That is the right default for most questions, and it is how a question about paying fees from zakat, one about a lawyer's fees, an attempt to override its instructions and two medical emergencies were simply answered. That is 13 more.

Some of the faults are specific enough to fix in a line, and three of them are one habit: a rule fires wherever its letters appear.

  • "Germany" (ألمانيا) begins with "pain" (ألم), so a question about a first residence permit was refused as a medical one.
  • "Passport" (جواز السفر) contains "travel" (السفر), so two passport questions were sent to escalation as travel matters.
  • "Bank" as an adjective (البنكي) contains "the bank" (البنك), so a question about paying a traffic fine from a bank account was refused as a finance one.
  • The check for a month named without a year matched one literal phrase, "the fifth month", so "the tenth month" went straight through.
  • The support gate's patterns were written from the two cases that taught them. On the unseen set it fired on none of the 93, including the three questions it existed to stop.

None of this is fixed yet, on purpose. A fix written from the unseen failures would make the next score meaningless. Each fix moves its case into the tuned set, and a new number needs a new unseen set.

Answer quality, for sixty cents

Taking the right path is one question; whether the answer was any good is another, and it needs a model. I ran both sets once more through a small model, Claude Haiku 5.5, with another small model, Gemini 2.5 Flash, as the judge, under a hard cap of two dollars. It cost $0.60.

On the unseen set the answers scored 2.58 of 3 for staying with their sources and 1.48 of 3 for correctness against the 31 gold answers, and the adversarial cases passed their rubrics 10 times in 25. The run's real finding was a fault: the pipeline gives each answer 1,024 tokens and never checks whether it ran out. 20 of the 66 answers on the unseen set and 11 of 18 on the tuned set were cut off mid-answer. Complete answers scored about 1.76 for correctness on both sets, cut-off ones about 0.8. The limit stays where it is until the next unseen set, for the same reason the rules do. With a small judge and 31 gold answers, read these scores as a direction, not a grade.

What it cannot claim

  • Users or real data. It is a single-turn tool over invented documents.
  • Labels checked by a person. An AI agent wrote the unseen set. A second AI read 20 of its labels, drawn before any result existed, and agreed with 19. No person has checked them.
  • Broad dialect coverage. The tuned dialect questions are mostly Najdi. Hijazi appears only in the unseen set, where it scored 7 of 11.

How it was built

An AI coding assistant wrote much of the code, under a contract I set: the four outcomes, which decisions belong to rules, and how each is measured. The November work was the opposite of building. It was making the evidence hold: a set of questions the rules never saw, a frozen and tagged commit, a test that needs no key, and a report that lists every miss with its cause. 435 tests pass with 88.8% coverage against an 80% floor, linting and strict type checks are clean, and every file states its author and licence, which is checked as well.

What it demonstrates

A score on the questions you tuned on measures fit. The number that matters is the one taken on questions you never saw, reported as it came out, with the reason for each miss. Here it is 62 of 93, and every miss has a cause you can point to: rules too few for questions nobody had written down, rules that fire wherever their letters appear, and a gate written from two examples. Finding that before anyone relies on the assistant is what the test is for.

All case studies

More case studies