SIU 2026 · IEEE

When the instruction is Turkish, the robot doesn't get worse. It does something else.

Vision-Language-Action models map a camera image and a natural-language instruction straight to motor commands. Every open VLA is trained and benchmarked in English. We measured what happens when it isn't — 50 manipulation commands, two languages, frozen weights, 300 inferences.

0.372

Mean cosine similarity between English and Turkish action vectors. Individual tasks reach −0.33.

1.36

Cohen's d. A large effect by any convention, at p = 7.7 × 10⁻¹³.

Hasan Tatar ^1,* Umut Gökmen ¹ Ilgın Akkoyun ² Sevcan Kahraman ¹

¹ Department of Computer Engineering, Istanbul Commerce University · ² Department of Electrical Engineering, Yıldız Technical University · ^* Corresponding author

The finding

This is not graceful degradation. It is a representational break.

A mean L2 error of 0.501 on its own would suggest a weaker version of the right action — same intent, less conviction. The cosine similarity rules that reading out. At 0.372, with a minimum of −0.33, the model is not undershooting the target. It is pointing somewhere else. A magnitude error can be corrected with output gain. A direction error cannot.

0.501

± 0.464

Mean L2 error

Distance between the English and Turkish action vectors for the same task and the same image.

0.372

± 0.530

Mean cosine similarity

Direction, not magnitude. Below zero the arm travels the opposite way — 10 of 50 tasks do.

0.002

p = 0.981

Complexity correlation

Spearman ρ against instruction complexity. We expected long commands to fail more. They do not.

+30.8%

−0.2% latency

Tokenization overhead

Turkish costs 28.7 tokens against 21.9 for English, and no measurable time. The cost is semantic.

Method

Deliberately minimal, so the language is the only thing that changes.

One model, frozen

OpenVLA-7B at 4-bit NF4 precision on a single T4 (15.6 GB) via Modal serverless. SigLIP and DinoV2 feed a Llama 2 7B backbone that doubles as action decoder. One forward pass yields a 7-DoF vector: three translational, three rotational, one binary gripper state.

Fifty commands, two languages

Drawn from BridgeData V2 — 20 pick-and-place, 15 push, 15 orient — stratified across three complexity tiers. Translated by two native Turkish speakers with professional English, then adjudicated by a third reviewer for meaning drift. No machine translation.

Everything else held still

The same static 224 × 224 synthetic image for every prompt in both languages. Temperature fixed at 0.3. Three repetitions per language per task: 50 × 2 × 3 = 300 inferences. The only variable that moves is the language of the instruction.

Paired, then tested

L2 norm for overall divergence, cosine similarity for direction independent of magnitude. One-sample t-test against zero, Wilcoxon signed-rank as a nonparametric check, one-way ANOVA across categories, Spearman against complexity tier.

Translation quality was the one place we refused to economise. In manipulation, spatial postpositions carry the whole instruction: üstüne (onto) and yanına (next to) share a suffix pattern and differ only in their root — trivially easy for a surface-level translator to conflate, and 20 cm apart on a real tabletop. A machine-translated benchmark would have measured the translator, not the model.

Results

The model either understands the command, or it doesn't. There is little in between.

Every one of the 50 tasks, sorted by divergence. The distribution is bimodal: 14 tasks land under 0.1 and 12 sit above 0.9, leaving only 24 anywhere in the middle. For a deployed system this all-or-nothing profile is worse than uniform mediocrity — there is no confidence signal that separates the two ends.

L2 error per task above 0.9 below 0.1

L2 error and cosine similarity for all 50 tasks
Task	L2 error	Cosine similarity	Category	Complexity
PP_008	0.042	0.999	pick place	simple
PP_004	0.043	-0.123	pick place	simple
OR_002	0.043	0.999	orient	simple
OR_011	0.050	0.999	orient	complex
OR_006	0.052	0.998	orient	medium
OR_014	0.058	0.998	orient	complex
OR_003	0.065	0.997	orient	simple
OR_005	0.066	0.997	orient	simple
PP_002	0.075	-0.122	pick place	simple
PP_015	0.075	0.321	pick place	complex
PP_010	0.077	0.997	pick place	medium
PP_017	0.084	0.996	pick place	complex
OR_015	0.085	0.996	orient	complex
OR_009	0.094	0.995	orient	medium
PU_012	0.101	0.994	push	complex
PP_018	0.106	-0.307	pick place	complex
OR_010	0.149	0.986	orient	medium
PP_014	0.364	0.667	pick place	medium
PU_010	0.368	0.666	push	medium
PU_006	0.369	0.674	push	medium
PU_015	0.372	0.477	push	complex
PP_007	0.377	-0.332	pick place	simple
PU_007	0.389	0.666	push	medium
PU_001	0.396	0.638	push	simple
PP_011	0.407	0.119	pick place	medium
PU_013	0.410	-0.300	push	complex
PP_013	0.673	0.283	pick place	medium
PP_009	0.676	0.321	pick place	medium
PP_019	0.682	0.335	pick place	complex
PP_020	0.686	0.334	pick place	complex
PP_001	0.687	0.066	pick place	simple
PU_002	0.690	0.379	push	simple
PU_009	0.690	0.034	push	medium
OR_007	0.695	-0.056	orient	medium
PU_008	0.695	0.312	push	medium
PP_012	0.701	0.339	pick place	medium
OR_001	0.712	0.371	orient	simple
PU_003	0.720	-0.157	push	simple
OR_004	0.997	0.002	orient	simple
PP_005	0.998	0.005	pick place	simple
PU_004	0.998	-0.001	push	simple
PU_014	0.998	0.018	push	complex
PP_016	0.999	0.003	pick place	complex
OR_012	0.999	0.022	orient	complex
OR_008	1.000	-0.002	orient	medium
OR_013	1.000	0.023	orient	complex
PP_003	1.001	0.000	pick place	simple
PP_006	1.003	-0.019	pick place	simple
PU_005	1.003	0.002	push	simple
PU_011	1.006	0.010	push	complex

By task category F(2,147) = 2.37, p = 0.097

Push 0.614

Pick & place 0.488

Orient 0.404

Push looks worse than orient, but the spread within each category covers nearly the full range. The ANOVA does not clear significance — no task type is reliably safer.

By instruction complexity ρ = 0.002, p = 0.981

Simple 0.551

Medium 0.462

Complex 0.482

The result we did not expect. Two-word commands break as often as multi-clause ones — simple tasks score marginally worse. Whatever fails, it is not syntactic parsing.

Mechanism

The tokenizer does not just fragment Turkish. It inverts it.

Because complexity does not predict failure, the break cannot be in syntactic composition — it has to sit lower, at the level where subwords become meaning. Llama 2's byte-pair encoding was fitted on a corpus that is roughly 97% English. Handed an agglutinative language, it cuts in the wrong places, and one of those cuts is worth the whole result.

Turkish, "lift"

kaldır

After BPE

kal

"stay" — a standalone Turkish verb

dır

the remaining suffix

The leading fragment is not noise. kal is the Turkish word for stay — the opposite of the instruction. The model's first and strongest signal about what to do points the wrong way, and nothing downstream is positioned to correct it.

The same mechanism erases grammar. Turkish marks the direct object with the accusative suffix -ı, a single character that says which noun the action applies to. Split into an arbitrary byte pair, that cue is gone before the action decoder ever sees it — which is why the 30.8% token overhead matters even though it costs no time. Each extra fragment takes an attention slot and dilutes the signal at every position.

Why it matters

The upper cluster is the gripper bit.

The gripper dimension is binary — open or closed. Invert it and the L2 error saturates near 1.0, which is exactly what produces the upper cluster in the distribution above. On a real arm that is not a metric. It is a released payload mid-transfer, or a closed gripper driven into the table.

What makes this a safety problem rather than an accuracy problem is that the model is not uncertain when it happens. The failures are bimodal and confident, and the confidence signal does not separate the two modes. A deployed system has no way to know it has misread the instruction, and the standard monitoring layer — thresholding on model confidence — cannot see it either.

For robots operating where English is not the working language, cross-lingual adaptation is not a localisation feature. It is a precondition for safe deployment.

Ten languages, not one pair

Turkish is agglutinative. Isolating (Mandarin), fusional (Arabic), and polysynthetic languages should fail differently — or, if they do not, the flaw is architectural rather than morphological, which is the more consequential answer.

Translation layer against LoRA

The two candidate fixes have never been compared on safety. Upstream machine translation is cheap and breaks precisely on spatial postpositions. LoRA updates the embedding space directly but needs an unknown quantity of instruction–action pairs. Measure the trade-off.

A guardrail the policy cannot provide

Given the image, the instruction, and the emitted action, can a language model flag the mismatch before execution? Because these failures are confident, no calibration threshold catches them. An external checker is the only place left to look.

Limitations, stated plainly: one language pair, synthetic visual input rather than live camera feeds, and 4-bit NF4 quantisation throughout. Full-precision inference may move the numbers. Physical hardware validation is the next thing this work needs.

Related work

Other research

SIU 2026 · IEEE Xplore First author

Cross-Lingual Performance Analysis of Vision-Language-Action Models

Evaluating Turkish command understanding in OpenVLA-7B. The work described on this page.

CognitiveDroneBench Benchmark study

Cognitive Drone

Benchmarking vision-language model reasoning for autonomous drone gate selection — multi-model evaluation with CLIP-free matching.

Open source Author

NEST — Neural EEG Sequence Transducer

Decoding brain signals into open-vocabulary text using Transformer encoders with BART cross-attention decoding.