17 September 2026 · २०८३ असोज १ · 6 minutes
AI hallucination in legal drafting, the run that scared us
I spent most of that year worrying about the wrong thing.
I worried about the made-up citation. That is what everyone worries about. It has already embarrassed lawyers abroad, in cases that reached the newspapers.
We built against it with some care. One model works out which laws may apply. It may send back search words and law names, and nothing else. Code, not a model, then settles which of those matched a real file. A later pass removes any section number that file does not hold. I have written about that in how the citation checks work.
Then we ran a divorce petition. That is when I saw the real failure.
What happened, in order
The first stage did its job well. It read the brief. It worked out that this was a divorce petition. It then listed, in Nepali, every fact such a petition needs. The list was correct.
If I had stopped there and shown that list to a lawyer, they would have nodded. Then they would have called the client.
The run did not stop there. The drafting stage wrote 11,000 characters.
It was a good petition. I need to be clear about that. The word "hallucination" makes people picture nonsense. This was not nonsense. The structure was right. The Nepali was right. The law was right: real sections, from real files.
And the facts were invented. Every fact the first stage had just listed as missing, the drafting stage filled in with something believable. The type of paper had been chosen by hand. So the search found the right sections for a divorce petition. It attached them to a client who did not exist.
The draft then went through every check we had. All of them passed it.
Of course they did. They look for made-up law. There was no made-up law in that draft.
Why this is the worse hallucination
A made-up section number announces itself. Someone looks it up. It is not there. The draft is wrong, and the draft gets fixed. It is embarrassing and it is survivable. It is also the kind of thing a machine is good at catching.
Correct law about invented facts announces nothing. It reads like careful work. It reads like a junior who understood the matter. That is the most dangerous thing a paper in a law office can pretend to be.
The reviewer's eye slides over it. Everything that would normally catch attention is in order. There is no clumsy clause. No odd citation. No heading out of place.
The reviewer cannot see the problem by reading. The problem is that the draft is not about the client.
I will not blame a model version here. The pipeline is ours. We asked a model to write a petition. We handed it a matter with holes in it. And we had put the rule about not inventing facts in an instruction, not in code. An instruction is a request. The run did what our design allowed.
What we changed
Three things. None of them clever.
First, the input floor moved out of the instructions and into code. A run with no parties, no facts and no brief text now stops before any draft exists.
That floor is low on purpose. Missing facts alone do not stop a run. A first draft that the lawyer completes is the whole product. A tool that refuses every thin brief is a tool nobody opens twice. But a brief with nothing in it is now a stop, written in code. It is no longer one more line in a prompt, competing with every other line.
Second, facts the brief does not carry are written as blanks and listed for the lawyer. That rule was there before. What changed is how much weight I will put on it in public. It is a design against invention. It is not proof that nothing is invented. That run is why I say that sentence in every meeting.
Third, every draft carries a do-not-file line in its own body. It shows on the web view, in print, and in the raw file: "AI-generated draft — not lawyer-verified. Review every citation, fact, and clause before relying on it. Do not file without a licensed advocate's review."
The warning belongs in the document
I will hold one opinion here and then stop.
A warning in a pop-up is aimed at the person clicking. It gets clicked away. That is what pop-ups are for. After the third time, nobody reads it.
It also does not travel. The draft gets exported. It is mailed to a colleague. It is opened on a phone in a taxi. It is printed by a clerk who never saw the pop-up. By then the warning is gone. What is left looks like finished work.
A line in the body of the draft is uglier. Everyone who sees our drafts asks, sooner or later, whether it has to be there. It has to be there. It is the only part of the warning that goes where the draft goes.
What I would say to a lawyer who has already decided against this
That the checks are real. That they close off one family of failures. That they close off nothing about whether the facts are the client's facts.
That part is the lawyer's job, and it stays the lawyer's job. A lawyer reviews, a lawyer signs. We are not building towards a version where that stops being true. If we ever write copy that suggests otherwise, it will be copy, not engineering.
The run's own identifier sits in a comment beside the input floor, with an account of what happened. When someone new joins and asks why the floor is code, that comment is what they are shown. It explains it better than I can.
Hardflip Technologies
Founder, Hardflip Technologies
17 September 2026 · २०८३ असोज १
Product facts in this piece are as of 14 September 2026.
This tool does not give legal advice. Every draft is usable only after review by a licensed advocate.