TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A test run by Thorsten Meyer AI gave GLM-5.3 FlashX Legal, which its maker describes as optimized for legal workflows, a score of 8 out of 100 across 15 legal questions. The test found citation problems across published runs, but its results are a spot check of models answering without internet or tools, not a broad measure of how they perform with legal documents.
GLM-5.3 FlashX Legal, a model that its maker Z.ai describes as optimized for legal workflows, scored 8 out of 100 in a test of 15 legal questions published by Thorsten Meyer AI on November 4. The test’s organizer says the model fully solved none of the tasks and provided no answer for two, raising questions about how much a legal label alone says about a model’s ability to handle deadline calculations, court decisions and citations.
The test, called Lexassay, gave 42 AI configurations the same 15 questions covering German, U.S. and EU law. Models answered without internet access or external tools. Ten questions and their answers are public; five are hidden, with only their scores published. The organizer says the hidden questions are intended to limit the chance that future models can learn answers from the web.
Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The test report says even its top model met every criterion on only eight of the ten public tasks. Mistral Large 4 scored 10, while GLM-5.3 FlashX Legal scored 8. The report says GLM’s base model scored 3. These are results from the specific test conditions, not a measure of every legal task or product configuration.
Lexassay grades answers against prepared answers using three to six weighted criteria per question, and checks citations against official databases, according to the report. It classifies citations as real, misattributed, invented or not findable. Across all published runs, the organizer says it confirmed 2,644 real citations and found 804 citations that were invented or did not support the model’s claim. Each invented or misattributed citation costs a quarter of the task’s points under the stated rules; an answer without a legal-advice disclaimer or pointer to a lawyer loses 5 percent.
Can a law-built model handle legal questions better? It scored 8/100
Lexassay tested legal answers without internet access or tools. Its results are a limited snapshot, but they show why deadlines and citations need careful checking.
What the scores show
The test compared 42 AI configurations under the same no-tools conditions.
GLM-5.3 FlashX Legal
Scored 8/100, fully solved none of the tasks, and gave no answer on two.
Claude Opus 5.5
Led with 79/100, yet met every criterion on only eight of the ten public tasks.
Different test, different ranking
A separate benchmark with supplied documents ranked Mistral Large 4 ahead of Claude Opus 5.5 and GPT-6 Astra.
How Lexassay checks answers
The test covers deadlines, legal changes, court decisions, and questions with false premises.
Models respond to 15 questions without internet access or external tools.
Three to six weighted criteria per question guide the grading.
Citations are checked against official databases for accuracy and support.
Why a one-day error matters
In the test’s German civil appeal example, a public holiday changed the deadline by jurisdiction.
| Case location | Appeal deadline | Calendar detail |
|---|---|---|
| Munich | May 28 | Corpus Christi is a public holiday in Bavaria |
| Berlin | May 27 | Corpus Christi is not a public holiday in Berlin |
What the results leave open
A single answer per task cannot show how every model performs across legal work or with different support.
- Answers without internet or tools
- One answer per model and task
- Published tasks, prepared answers, and citation checks
- Search or legal database access
- Supplied case files and documents
- Repeat runs, prompts, and model settings
Why Legal Deadlines Need Checking
The results point to a practical risk for people who use chatbots to interpret law: a fluent answer can still miss a deadline or cite a decision that does not support it. In a German civil appeal, the report notes, a filing submitted late can be rejected as inadmissible. That makes a one-day error consequential, even when the rest of an answer sounds plausible.
One test example involved two German civil cases with judgments served on the same date, one in Munich and one in Berlin. Lexassay’s correct answer accounted for Corpus Christi being a public holiday in Bavaria but not Berlin: the appeal deadline fell on May 28 in Munich and May 27 in Berlin. The report says one model put Berlin’s deadline a day late, while another placed the deadline for written appeal grounds a month too late. These examples illustrate the test’s focus; they do not establish how often such errors occur in real legal use.
For readers, the score is a reason to treat general chatbot output as a starting point, especially when a deadline, legal filing, contract or government authority is involved. The test organizer advises checking cited laws and rulings in their original sources and consulting a lawyer. The scores alone cannot show whether a lawyer using a model with verified sources, tools or supplied documents would get a different result.
Products Worth Considering
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
As an affiliate, we earn on qualifying purchases.
How Lexassay Tests Legal Answers
Lexassay is a test run by Thorsten Meyer AI, the same publisher reporting these results. It covers deadline calculations, recent legal changes, leading court decisions and questions that contain a false premise. The questions are answered without internet access or tools, which the organizer says is meant to measure what a model can provide on its own, similar to a chatbot without search.
The report describes the exercise as a spot check, rather than a strict benchmark: each model and task receives one answer, so small score differences may be noise. Its grading method compares responses with a prepared answer and verifies citations against official databases. The publisher says the questions, prepared answers and citation checks are public, allowing readers to examine the grading.
Results can differ under other conditions. The report cites Vals AI’s legal agent benchmark, where models work with supplied documents: Mistral Large 4 ranked eighth, ahead of Claude Opus 5.5 in 21st place and GPT-6 Astra in 24th. That comparison measures a different setup from Lexassay’s no-tools questions, so the rankings are not directly interchangeable.
““optimized for professional legal workflows””
— Z.ai, as quoted in the Lexassay report
Limits of the Published Scores
The test does not establish how the models perform across the full range of legal work or when connected to search, legal databases or supplied case files. The organizer says each task received one answer per configuration, so repeat runs could differ and small score gaps may not be meaningful. The available results also do not show whether changes to prompts or model settings would alter the scores.
There is a disclosed conflict of interest: Thorsten Meyer AI runs Lexassay, and several Claude agents graded the answers. A Claude model ranked first in this test. The publisher says it discloses this on Lexassay’s method page and publishes answers, prepared answers and citation checks. Those disclosures let readers inspect the process, but the organizer’s role and the use of Claude graders remain relevant when interpreting the rankings.
The report does not establish that the results predict the quality of legal services or that any model is safe to rely on for a particular case. It describes a test of model answers, not legal advice. Whether the same systems perform better with document retrieval, lawyer review or other safeguards remains outside the reported comparison.
Checking Sources Before Acting
The next useful step for readers assessing a model is to examine the published Lexassay tasks, prepared answers and citation checks, and compare those with the test rules. The organizer says five tasks remain hidden and their scores, rather than full answers, are published; future model results may add to the comparison, though the report does not give a publication schedule.
For someone facing an active legal matter, the report’s practical advice is to verify every cited statute and ruling against the source, and not to take a filing deadline from a chatbot. A lawyer can assess the applicable jurisdiction, dates and procedural rules. Lexassay’s results provide a limited snapshot of model performance under specified conditions; they do not replace that case-specific review.
Key Questions
Which model scored 8 out of 100?
GLM-5.3 FlashX Legal scored 8/100 in Lexassay’s 15-question test. The report says it fully solved none of the tasks and gave no answer on two.
Which model had the highest Lexassay score?
Claude Opus 5.5 scored 79/100, the highest score reported. It met every criterion on eight of the ten public tasks, according to the organizer.
Does this test show how AI performs with legal documents?
No. Lexassay’s models answered without internet access or tools. A separate benchmark cited in the report gives models supplied documents and produces a different ranking.
Can I rely on a chatbot for a legal deadline?
The report advises readers to check the cited law and ruling at the source and not take a deadline from a chatbot. Consult a lawyer when a filing deadline or other legal matter is at stake.
Source: Thorsten Meyer AI
