HaSeen هـ · سين

I built HaSeen to read an Arabic turn and say, for every word, whether it belongs to the dialect the voice agent was told to speak, and what it is when it does not. It runs offline on a CPU, with no LLM. The name is the idea: Egyptian says هيتم, MSA says سيتم; one letter decides the dialect.

What Tuner asked, and what I decided

Given a text and a target dialect, label every word as conforming or deviating and say what a deviating word is instead. No cloud LLM at inference: the system must run air-gapped. Graded on architecture, productisation, ML judgment, code quality, reproducibility and communication.

I sent five questions within hours of receiving the task; Mohamed answered all five in 16 minutes. The "your call" answers became decisions I state in the README.

I led the research; Claude was my assistant

I worked with Claude Code as a research assistant in the terminal: it read papers and dataset cards for me, wrote and ran code, drove the cloud GPU sandboxes, and drafted labels and pages. Every direction, data source, judgment call and spend was mine. This is the split, with the evidence for it.

What the field already knew

Before choosing anything I read the papers, listed every dataset with its licence, and ran three small spikes (about $0.03) to check what really exists. These findings shaped everything after.

Notes: landscape · papers · data and models · repos and tools

30+ sources studied, each with a verdict

I studied every source one by one: what it really contains, its licence, how to get it, and whether its text is human, model-transcribed or synthetic. Then I set one rule for all three builds.

One note per source, with counts and verdicts: research/sources/01–15 · dataset overview · V3 data audit

From "which kind of model" to every layer

I chose the architecture in four steps, each settled by evidence before training. Click a step.

The model barely mattered, the labels did

Six small runs on one L4 (about $0.95) compared backbones, warm starts and heads, then two ways of labelling the same data, on 30 realistic service sentences.

Changing only how shared words were labelled lifted realistic F1 from 0.627 to 0.800, while all backbones and heads sat within about one point. That lesson held for three days: every big gain afterwards also came from the labels. Pilot report

A test built first, then frozen

In every build I built the evaluation before the training data, from different sources, and froze it with sha256 hashes. The final test is the hardest of the three.

Bench card · on Hugging Face: HaSeen-3-Bench · public part: data/eval_public

Labels from evidence, not from an LLM

The research says an LLM teacher can be worse than the student on dialects, so training labels come from an evidence-ordered labeler that says "unknown" when it is not sure.

Data card · labeler · on Hugging Face: HaSeen-3-Mix

Twenty runs, failures kept

I wrote the expected result before every round and kept every run, including the one that collapsed. Fixes came from dev data, never from the test.

1. 1

   Build 1: a collapse, then 0.963

   The first pointer head collapsed: its three option vectors were nearly identical, so it learned one "naturalness" score. I gave each option a learned residual and it trained. Three seeds: Egyptian F1 0.966 ± 0.005, 74 behaviour tests, ECE 0.0005. Shipped the seed with the best dev score, not the best gold, to keep the gold clean.

2. 2

   Build 2: nine varieties, then a gate I added

   Four runs in parallel on one L4, then three more after labeler fixes. The run with the best dev F1 failed the task example (أبغى), so I added a known-answer gate and shipped the only run that passed 54/54, recording the deviation from my own rule.

3. 3

   Build 3, round 1: stopped at 0.83, and I asked why

   The training curve was flat, so more epochs would not help. I scored the labeler itself on dev: 0.815. The model had reached its teacher's ceiling.

   teacher, round 1model, round 1teacher, round 2model, round 2
4. 4

   Build 3, round 2: review first, then train

   I did not start the GPU until the data was right: labeler review, a clean rebuild (an earlier build was corrupted by two writers and discarded), the leakage filter, a label spot check, and the whole pipeline run end to end on a tiny model. Round 2: dev 0.876, known answers 71/73 (round 1 had failed at 69/73).

5. 5

   The choice

   Rule written before the results: pass the known-answer gate, then the higher dev F1. Linear 0.876 vs pointer 0.871 → linear. Exported to int8, thresholds refit on the int8 outputs, test scored once.

Pre-registration and decision records · selection · known answers

Precise everywhere, strongest on MSA and Egyptian

All six targets stay under 1% false alarms on native speech, and what HaSeen flags is right 91–97% of the time. MSA and Egyptian pass the F1 ≥ 0.90 gate; Saudi and Gulf are within 0.02 of it; Levantine and Iraqi miss about a quarter of swapped words. Dev 0.874, test 0.869: no sign of over-fitting.

RESULTS.md · test JSON · Build 2 on this test · same test through the package

One model, four ways to run it, all offline

int8 takes the same decision as fp32 on 99.41% of 59,733 dev words. report.json

uv tool install "git+https://github.com/NightPrinceY/haseen-v3"
haseen download                                        # once, 156 MB; later runs are offline
haseen "أنا أبغى أروح الاجتماع بكرة بس سوف أتأخر شوية" -t egy
haseen serve --port 8000                               # POST /detect {"text", "target"} · GET /health
docker run --rm --network none -p 8000:8000 nightprincey/haseen3:3.0.0

The calls I made, and the evidence behind each

Where it is weak, and what I would do

• Egyptian words under target Gulf. In a probe of 31 frequent Egyptian service words, Gulf flags 8 (حضرتك، دلوقتي، هيوصل) and misses عايز، إحنا، أوي، مفيش، برضه. Some of the 31 are also Hijazi, but most misses are real. Cause: Gulf is "not natural" only when Saudi, Gulf-east and Yemeni all say no, and Yemeni is often unknown. Fix: treat that unknown as no when Saudi and Gulf-east say no and Yemeni text barely uses the word, then retrain (about 1.5 hours, $1.3).
• Words it has never seen. Swaps from held-out concepts are caught 32–63% of the time; held-out dictionary entries 57%. Next: more lexicon coverage for Levantine and Iraqi.
• The first label. The label set holds the right variety 83–92% of the time, its first item 38–79%. Next: rank labels by how specific a variety is to the word.
• LLM-style replies. Agent-set F1 0.49–0.71; the agent labels are mine with Claude's help, not checked by native speakers of every dialect. Next: native reviewers per dialect, starting with Gulf.
• Licences. MADAR, SADA, Birzeit and QCRI sets are research or non-commercial, so the datasets are shared for research and review only, each source under its own licence, and the models are released for research use. Production: the same pipeline rebuilt on commercial-friendly data and Tuner's own call transcripts.
• Text only. Accent and pronunciation reach HaSeen through STT output; running it on ASR transcripts is the natural next integration.

Everything, linked

HaSeen inside a real Tuner workspace

Tuner's evals are LLM judges, and its rules can test call metadata (call.metadata.<key>). So HaSeen plugs in with no change on Tuner's side: it checks every agent turn, writes the result into the call's metadata, and one rule turns a FAIL into a red flag. I tried it on my own Tuner workspace on 20 Oct.

What happened in Tuner

• A Custom API agent, one rule made in the UI (Extra Metadata call.metadata.haseen_verdict = FAIL → red flag dialect_drift), and five Arabic demo calls sent with haseen tuner demo_calls.json --send (HTTP 201 each).
• dialect_drift landed on exactly the three calls HaSeen failed (Egyptian agent saying سوف، ولن; Saudi agent saying حضرتك، هبعتلك، دلوقتي; Gulf agent saying سنقوم، وسيتم) and on neither call it passed.
• Tuner's own LLM analysis ran on the same calls (outcome, sentiment, summary, evals). None of it noticed the dialect drift.
• hallucination_detected is on all five because the demo agent's prompt has no menu or prices to ground against; it comes from the demo setup, not from HaSeen.

src/haseen/tuner.py · examples/tuner · demo/tuner · demo conversations written for the test (made-up companies); every result on them is the model's.