u/Dhiadev-tn

I'm an 18-year-old independent student from Tunisia. I built and I'm leading an open, from-scratch machine-translation pipeline and parallel corpus for Tunisian Darija. Sharing it for feedback.

Why: Tunisian Darija, written in Arabizi (Latin letters + numerals like 3/7/9/5 for Arabic phonemes), has almost no open NLP resources. Existing Arabic tools route it through MSA and mishandle the orthography. To the best of my knowledge there was no open parallel

corpus or from-scratch baseline for it.

What I built (all open):

- Arabizi-aware SentencePiece BPE tokenizer (3/7/9/5 as protected symbols), shared 16k vocab.

- ~15.6M-param encoder–decoder Transformer, from scratch (no pretrained LM): transfer-learned from cleaned Moroccan Darija, then fine-tuned on hand-crafted Tunisian pairs.

- Full cleaning / training / eval pipeline.

Honest results & limitations: v1 BLEU is 3.89 on a small locked test set low, and I'll be upfront about it. The corpus is ~553 hand-crafted pairs, so data is the bottleneck, not architecture. I treat 3.89 as a first honest baseline to beat as the corpus grows.

Where I'm taking it: I'm expanding this into a larger, ethically-collected Darija corpus that I curate and validate consent-documented field collection, every pair provenance-tagged. I'm looking for contributors to help grow it, with every contribution reviewed

to keep quality and consent standards.

Looking for: technical feedback/critique, and anyone interested in contributing data or collaborating on low-resource / dialectal Arabic MT.

Links:

github repo: https://github.com/Dhiadev-tn/darija-translator

Hugging faces dataset: https://huggingface.co/datasets/Dhiadev-tn/tunisian-darija-english

hugging faces model: https://huggingface.co/Dhiadev-tn/darija-translator

I'm 18, from Tunisia, and I built this because nobody else had.

Tunisian Darija is what 12 million Tunisians actually speak. Not Modern Standard Arabic. Not Moroccan. A separate dialect that borrows from Arabic, French, Italian, and Amazigh, written online in Arabizi Latin letters with numbers for Arabic sounds (3→ع, 7→ح, 9→ق, 5→خ).

When I searched for a parallel corpus to build a translation model, I found nothing. TUNIZI covers sentiment analysis. TunBERT does dialect classification. But zero parallel datasets existed for Tunisian Darija-to-English translation. Not one.

So I built the first one from scratch with no funding, no university affiliation, no mentor, and no institutional support. Just me, a laptop, and the language I grew up speaking.

The first 500 pairs came from my own memory as a native speaker, covering 50 categories of real Tunisian daily life cafe culture, Ramadan traditions, wedding customs, bac exam stress, barbershop talk, louage rides, haggling at the medina, football arguments, bureaucracy nightmares, olive harvest season, Friday afternoon naps, and more. Zero automated generation. Every pair hand-written and validated.

Then I left my desk and started collecting from real people:

• My father's childhood memories growing up in Ain Draham, a mountain village in northwestern Tunisia the scent of the forest, nearly getting bitten by a snake, his cousin falling off his uncle's horse
• My grandmother's stories about her father's farm cows, sheep, thieves stealing the neighbors' animals at night, and her father calmly finishing his morning prayer before stepping outside to check
• An elderly man from Siliana I met at a cafe who speaks a dialect I barely recognized — words I had to ask about, rhythms I'd never heard

Every pair is provenance-tagged with its source: self, family-father, family-grandmother, community-siliana. Every collection session is logged with date, place, speaker context, and consent status.

I excluded an entire session of data because I hadn't established consent before the conversation began. The language was rich. I threw it all away anyway. A dataset built on trust means sometimes throwing away good data.

What this dataset has that scraped corpora don't:

• Regional dialect diversity: urban , mountain Ain Draham, rural Siliana
• Generational variation: grandmother's speech vs mine
• Provenance: every pair traces to a known speaker, region, and context
• Documented ethics: consent logged, exclusions documented, no anonymous mass scraping

I trained the first Tunisian Darija-to-English translation model on this dataset a 15.6M parameter Transformer built from scratch on an RTX 3050 (4GB VRAM). v1 BLEU: 3.89 on a held-out test set. Low, but the first benchmark ever measured for this language. A published ACL researcher who found my work on Reddit said it's 'basically guaranteed to be novel.'

I'm heading toward 1,000+ pairs through continued community collection and will be presenting this research at Tunisia's AI National Summit (AINS 4.0) later this month the first high schooler to ever present at the event.

The dataset is CC BY-NC-SA 4.0 and public on HuggingFace. 110+ downloads so far.

If you work on low-resource NLP, Arabic dialect processing, or sociolinguistic data it's yours.

HuggingFace: huggingface.co/datasets/Dhiadev-tn/tunisian-darija-english
Full pipeline + model: github.com/Dhiadev-tn/darija-translator

I built and deployed an open-source translation system for Tunisian Darija, a dialect spoken by 12M people with near-zero NLP representation. The model and dataset are both on HuggingFace

The model is a 15.6M parameter encoder-decoder Transformer with a custom BPE tokenizer handling Arabizi script. Pre-trained on 36K Moroccan Darija pairs, fine-tuned on 500 hand-crafted Tunisian pairs

The dataset has been downloaded 110+ times organically without any promotion which tells me there's genuine demand for low-resource Arabic dialect data in the research community

v1 BLEU: 3.89 on a held-out test set. Early days. This summer I'm expanding the dataset through community field collection in Tunisia and retraining for v2

Model: huggingface.co/Dhiadev-tn/darija-translator

Dataset: huggingface.co/datasets/Dhiadev-tn/tunisian-darija-english

GitHub: github.com/Dhiadev-tn/darija-translator

Anyone working on low-resource Arabic NLP or dialect MT would love to connect