Which embedding model actually speaks Hebrew?

Every RAG demo I've built with Spring AI in the past two years follows the same recipe. Chunk the documents, embed them, stash the vectors in pgvector, let the LLM do the talking. It works beautifully in English. And then, every single time, someone in the room asks the question I've been dreading:

"Does it work in Hebrew?"

Of course it does. Hebrew goes in, vectors come out, the demo finds something. That was never the question. The real question, the one hiding behind the polite one, is how well. Is the model actually understanding Hebrew, or is it pattern-matching on the handful of words it recognizes and getting lucky? My honest answer used to be "well enough, I think," which is not an answer an architect should be giving. This is the story of how I finally measured it, what I found along the way, and why the answer surprised me.

Where it started: the leaderboard

Like everyone else, I started on the MTEB leaderboard on Hugging Face. If you haven't seen it, it's the closest thing the embedding world has to a Top 40 chart. Hundreds of models, dozens of tasks, one big sortable table. You click the "Multilingual" tab, sort by score, and take the top open-weight model. Done, right?

That's how I got my first two candidates. Qwen3-Embedding-8B has been sitting at or near the top of the multilingual board since Alibaba released it, and NVIDIA's embedding models have been trading punches with it ever since, with Nemotron-3-Embed-8B as their latest contender. Both are 8 billion parameters, both claim broad multilingual coverage, both have model cards full of impressive numbers.

The third candidate needed no leaderboard. nomic-embed-text has been my go-to embedding model for almost two years. It runs under Ollama, starts in seconds, doesn't heat up the room, and just about every RAG demo I've shipped in that time was built on it, Hebrew ones included. It did its job, and it did it well. Its current multilingual version is a 475M-parameter mixture-of-experts model, which next to the two 8B giants looks like a rounding error.

And that, really, was the whole point of this exercise. I wasn't looking for a replacement for nomic. I wanted to know whether stepping up to the 8B tier was worth the trouble, or whether I'd been fine all along and just didn't have the numbers to prove it.

Two leaderboard picks and one incumbent. Let the battle begin!

Where the leaderboard stops

Instead of trusting the model's "Multilingual" label, I asked what it actually measures for Hebrew.

MTEB ships as a Python package, so the answer is a few lines of code away:

import mteb
all_tasks = mteb.get_tasks()
hebrew = mteb.get_tasks(languages=["heb"])
print(len(all_tasks), len(hebrew))

At the time of writing, MTEB contains 1,467 tasks. 22 of them include Hebrew in any form. Of the 454 text retrieval tasks, which is the thing RAG actually does, exactly three touch Hebrew. And when I looked at what those three are, my heart sank a little:

• Belebele, reading-comprehension passages machine-translated from English into 120+ languages.

• MKQA, Google's Natural Questions translated into 26 languages.

• WebFAQ, FAQ pages scraped from the web.

Every one of them is either translated from English or scraped. Not one contains a Hebrew speaker typing a Hebrew question about a Hebrew document. The multilingual benchmark that the leaderboard sorts by is even thinner: 131 tasks, 6 with Hebrew, and 4 of those are bitext mining, which is "match this sentence to its translation." That's a lovely task. It is not retrieval.

So the number I had been trusting was an average over a hundred languages, in which Hebrew contributes a few translated datasets to a few tasks. A model could be mediocre at Hebrew and still sit at the top of that table. Or it could be brilliant at Hebrew and nobody would know.

"Multilingual" on a model card tells you the model saw your language. It doesn't tell you it understands it. For that, you need your own data.

Finding our own data

I needed real Hebrew questions, from real people, matched to real Hebrew documents. That's a surprisingly specific ask, and for a while I assumed I'd have to build it myself.

Then I remembered the MAFAT Hebrew Semantic Retrieval National Challenge from a couple of years back, and it led me to NNLP-IL, Israel's National Natural Language Processing program. Together with Webiks they had published exactly what I needed, built from Kol-Zchut, the "all rights" website that explains citizens' legal rights in plain Hebrew.

The dataset has two parts:

• A paragraph corpus. Every page on Kol-Zchut, split by HTML headings and merged into chunks of up to 512 tokens. 24,487 chunks from 7,007 pages.

• A question set. 3,890 questions that real visitors typed into the site, each paired with the paragraph that answers it. After deduplication, 2,951 unique questions.

Here's a typical pair:

Question: דחו לי את התביעה לאבטלה, איך מגישים ערעור? (My unemployment claim was rejected, how do I file an appeal?)

Answer paragraph: a 1,566-character chunk from the "unemployment benefits" page covering repeat claims, appeals to the labor court, relief for women aged 57 to 67, and a table of benefit rates.

Look at the mismatch. The question is short and colloquial. The answer is long and covers four topics, of which one is what the person wanted. That's what retrieval looks like when the users are real, and it's exactly what the leaderboard's translated datasets don't capture.

Getting the data took a bit of detective work. The GitHub repos contain only READMEs, with the actual files sitting behind Google Drive links inside them. The corpus JSON stores every Hebrew letter as a \u05d3-style escape, so opening it in a terminal shows a wall of backslashes and a 29 MB text corpus balloons to 150 MB on disk. And 966 paragraphs are boilerplate that appears on many pages, so if you naively match questions to paragraphs by text, some questions end up with 65 "correct" answers. Matching on the (page, text) pair fixed that.

Inspect the data before you write the benchmark. Every dataset has a quirk that will silently skew your numbers, and you'll only find it by looking.

The contenders, and their secret handshakes

Model	Parameters	Dimensions	Query prefix	Document prefix
Nemotron-3-Embed-8B	8B	4096	query:	passage:
Qwen3-Embedding-8B	8B	4096	Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:	(none)
nomic-embed-text-v2-moe	475M (MoE)	768	search_query:	search_document:

The prefix column is the part people skip, and it's the part that bites. Modern embedding models are trained asymmetrically: a question is embedded with one prompt and a document with another. Leave the prefix out and your scores drop, sometimes by a lot, with no error message to tell you why. Each model ships its prompts in config_sentence_transformers.json, and sentence-transformers applies them for you if you pass prompt_name="query". I checked each model's config by hand before trusting it, and I'm glad I did, because the three models use three different naming conventions.

Running 8B models on a MacBook

My daily driver is an M1 Max with 64 GB of unified memory. No CUDA, no FlashAttention, no cloud GPU. Could it embed 21,000 Hebrew paragraphs with an 8B model?

Yes. Slowly. Here's the load code that worked for all three:

model = SentenceTransformer(
    "nvidia/Nemotron-3-Embed-8B-BF16",
    device="mps",
    model_kwargs={"dtype": torch.float16, "attn_implementation": "sdpa"},
    processor_kwargs={"padding_side": "left"},
)
model.max_seq_length = 2048

• device="mps" puts the model on the Apple GPU. All three ran there without falling back to CPU.

• float16 halves the memory. An 8B model fits in 16 GB with room to spare.

• attn_implementation="sdpa" replaces FlashAttention, which doesn't exist on macOS. The model cards default to flash_attention_2, so you have to override it or the load fails.

• padding_side="left" matters for models that pool the last token. Get it wrong and you're embedding padding.

The evening I'd like to save you

None of this was particularly hard. All of it was annoying.

Disk space. Two 8B models in bf16 are 31 GB of safetensors. My Mac had 36 GB free. I ended up downloading Nemotron to a network share that reads at 11 MB/s, which turned every model load into a 23-minute coffee break. Check df -h before you check snapshot_download.

Stalled downloads. My first attempt sat at 0 bytes for five minutes. The new xet transfer backend in huggingface_hub didn't like something about my network. HF_HUB_DISABLE_XET=1 fixed it instantly.

Silent dtype fallback. Nomic's custom model code only honors the legacy torch_dtype argument, not the newer dtype. My first run loaded it in float32 without a single warning. Now I always print next(model.parameters()).dtype after loading. Trust, but verify.

Throughput. An 8B model on the M1 Max embeds roughly one Hebrew paragraph per second. Qwen3 took 6.4 hours for the corpus. Nemotron took 7.5. Nomic took 16 minutes. I cached the corpus embeddings to .npy files with a checkpoint every 1,024 paragraphs, so a crash at hour five wouldn't send me back to zero, and reruns only re-embed the questions. Most of this benchmark ran overnight while I slept, which felt like the appropriate amount of effort for a question I'd been avoiding for two years.

Embedding the corpus is the expensive part. Cache it to disk the first time and never do it again.

How I scored it

For each question, I ranked all 21,142 paragraphs by cosine similarity and computed:

• NDCG@20, the metric the MAFAT challenge uses. It rewards putting the right paragraph near the top of the first page.

• recall@1, @5, @10: how often the correct paragraphs show up in the top k.

I also computed a document-level NDCG@20 where paragraphs collapse to their page, because that's what the reference evaluator from Webiks does and I wasn't certain which granularity the published baseline used.

Results

Model	NDCG@20	recall@1	recall@5	recall@10	doc NDCG@20	corpus time
Nemotron-3-Embed-8B	0.465	0.247	0.510	0.625	0.645	449 min
Qwen3-Embedding-8B	0.371	0.178	0.405	0.507	0.543	382 min
nomic-embed-text-v2-moe	0.333	0.157	0.367	0.468	0.509	16 min

For reference, the published baseline for this dataset is multilingual-e5-base at 0.397 NDCG@20.

What I actually learned

The leaderboard would have picked wrong. Had I taken the multilingual table at face value, I'd have shipped Qwen3. On Hebrew, Nemotron beats it by 25% relative on NDCG@20 and finds the right paragraph in its top 10 for 62% of questions versus 51%. Same size, same dimensions, similar cost. It's also the only model that clears the e5-base baseline at paragraph level. Two models that look interchangeable on the leaderboard are not interchangeable in Hebrew, and there's no way to know that without Hebrew data.

Nomic held its ground. The model I've been relying on for two years landed within 11% of Qwen3 while being 17 times smaller and 24 times faster. If someone had told me an 8B model would barely edge out a 475M model on retrieval, I'd have politely disagreed. Now I've measured it. Parameter count is not a quality metric, and my two years of "well enough, I think" turn out to have been better founded than I feared.

So, was the next level worth it? It depends which next level. Qwen3: no. Twenty-four times the compute for an 11% gain is not a step up, it's a tax. Nemotron: yes, when retrieval quality is the product. It finds 62% of answers in the top 10 where nomic finds 47%, and for a customer-facing assistant that difference is the difference between "helpful" and "annoying." For internal tools and demos, nomic at 16 minutes per corpus and 768-dimensional vectors, plugged straight into Spring AI's OllamaEmbeddingModel, remains the right tradeoff. I now know exactly where the line is, and that alone was worth the weekend.

The absolute numbers are humbling. Even the best model puts the right paragraph in first place only a quarter of the time. Some of that is the dataset: chunks are long and cover several topics, a quarter of the questions have multiple correct answers, and the questions are written the way people actually type. But some of it is that none of these models have ever seen Israeli legal-rights text. The Webiks team fine-tuned multilingual-e5-large on exactly this training set, and I'd bet it beats all three general-purpose models. Fine-tuning is probably the biggest lever left, and it's next on my list.

If you can afford the compute, use Nemotron. If you can't, use nomic and fine-tune it on your own questions. Either way, measure on your own language before you trust a leaderboard.

Caveats

This is one dataset, one domain, one language. Everything ran in float16 on Apple Silicon, which can nudge scores slightly compared to bf16 on an NVIDIA GPU. I used each model's default English task instruction; a Hebrew instruction might help the instruct-style models. And I only tested three models. E5, BGE-M3, and the commercial embedding APIs from Google and OpenAI are obvious candidates I haven't run yet.

Try it yourself

The benchmark script, requirements and results are on GitHub: odedia/hebrew-embedding-benchmark. It's a single Python file. Clone it, swap in your own model, and tell me what you find. I would genuinely love to see numbers for the models I haven't tried, and I'd love even more to be proven wrong about one of these.

Thanks to NNLP-IL and Webiks for publishing the Kol-Zchut dataset. Without it, this would have been a much longer weekend.

Note: AI was used to improve and refine the wording in this article, however the work and the opinions are my own.