Abstract

Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression. Although recent advances in vision-language models have led to substantial improvements in REC tasks, current REC benchmarks often hold simple scenarios and the assumption that each expression maps to a unique object. These limitations hinder the deployment of REC models in open-world environments. To fill this gap, we introduce OpenRef, a new benchmark for REC in complex visual and linguistic scenarios.

OpenRef features three key advancements: 1) Diverse visual scenarios: spanning diverse visual domains, including ground views, drone views, dark scenes and adverse weather conditions; 2) Variable target counts: breaking the single-target limitation with multi-target and none-target samples; 3) Rich vocabulary types: incorporating proper nouns, polysemous words and ordinal terms to fit a wider range of expression needs. Furthermore, as traditional metrics are insufficient for open-world setting, we leverage F1 to measure grounding accuracy and propose N3R (Negative Relative Rejection Reliability) to assess relative rejection reliability against negative expressions.

Finally, we introduce Multi-task Consistency Checker (MCC), a training-free but plug-and-play strategy that enhances model performance with one click by enforcing consistency self-verification. Extensive experiments demonstrate that this work significantly advances the performance of existing REC models in complex scenarios, paving the way for open-world REC.

Benchmark Design

Overview and representative qualitative examples of the OpenRef benchmark.

Figure 1: Comprehensive visualization of OpenRef Benchmark.

Benchmark Statistics

OpenRef benchmark establishes a large-scale, robust evaluation environment tailored for open-world Referring Expression Comprehension. The detailed statistical composition is summarized below:

32,735
Total Expressions

17,586
Total Images

15,149
Positive Expressions

17,586
Negative Expressions

Figure 2: Detailed statistical distribution of referring target sizes, target numbers, and phrase vocabularies.

Across these samples, OpenRef comprises a total of 37,698 object instances. Notably, each expression refers to an average of 4.24 objects, presenting a significant challenge for precise localization and negative rejection reliability (N3R).

🏆 OpenRef Leaderboard

Rank	Model	Size	Landmark	Celebrity	Logo	Polysemy	Ordinal	Single	Multi	Avg.
🥇 1	Qwen3-VL	8B	88.6	90.8	57.0	71.4	27.7	74.7	35.5	63.7
🥈 2	Qwen3-VL	4B	87.1	89.0	56.8	68.6	15.9	75.2	33.7	60.9
🥉 3	Qwen2.5-VL	32B	85.5	91.4	31.5	61.4	46.7	61.8	37.4	59.4
4	GLM-4.6V	9B	86.5	83.5	55.9	68.2	24.1	72.9	21.1	58.9
5	Qwen3-VL	2B	83.1	89.1	34.9	63.5	12.3	66.6	29.6	54.2
6	Qwen2.5-VL	7B	85.0	85.6	27.6	64.5	15.9	61.8	28.1	52.6
7	MiniCPM-V	8B	78.8	84.5	33.5	54.9	24.6	53.5	11.8	48.8
8	Grounding-Dino	—	61.3	52.7	31.5	61.4	8.2	58.3	25.2	42.7
9	InternVL3.5	8B	72.5	77.0	2.9	52.1	35.4	36.0	9.4	40.8
10	Keye-VL-1.5	8B	67.9	77.0	12.4	64.4	23.6	26.4	13.3	40.7
11	MiMo-VL-RL	7B	60.7	70.5	8.9	65.5	15.1	27.7	15.0	37.6
12	LLaVA-OV-1.5	8B	70.5	75.7	7.6	59.1	6.7	19.1	8.7	35.3
13	Qwen2-VL	7B	68.4	59.3	21.7	46.8	9.2	16.7	16.8	34.1
14	Mistral-3	8B	8.2	24.8	0.7	4.6	2.3	0.7	0.2	5.9

Rank	Model	Size	Landmark	Celebrity	Logo	Ordinal	Complex Scenarios	Avg.
🥇 1	Qwen2-VL	7B	96.4	100	87.7	84.6	85.6	90.9
🥈 2	Qwen3-VL	8B	100	92.1	90.1	58.0	82.6	84.5
🥉 3	GLM-4.6V	9B	100	90.3	94.0	41.4	67.8	78.7
4	LLaVA-OV-1.5	8B	100	94.5	74.7	27.8	78.6	75.1
5	MiniCPM-V-4.5	8B	98.5	94.0	83.8	22.8	68.8	73.6
6	Mistral-3	8B	98.5	82.3	87.3	7.5	79.0	70.9
7	Keye-VL-1.5	8B	96.4	74.0	83.6	28.3	69.3	70.3
8	Qwen3-VL	2B	92.7	60.9	23.1	4.5	9.2	38.1
9	Qwen3-VL	4B	92.2	60.8	23.4	4.3	8.6	37.9
10	Qwen2.5-VL	32B	79.8	45.7	44.8	6.9	12.5	37.9
11	InternVL3.5	8B	68.9	52.3	42.8	56.6	34.6	51.0
12	Grounding-Dino	—	40.0	36.8	38.3	70.6	50.0	47.2
13	Qwen2.5-VL	7B	29.0	13.8	4.4	4.2	19.0	14.1
14	MiMo-VL-RL	7B	22.1	20.5	15.6	27.7	17.8	20.7

Multi-task Consistency Checker (MCC)

To alleviate logical hallucinations in open-world visual grounding, we unveil a fundamental task conflict between Referring Expression Comprehension (REC) and referring counting within Multimodal Large Language Models (MLLMs). Driven by this insight, we propose the Multi-task Consistency Checker (MCC), a training-free framework designed to tap into the self-verification capabilities of MLLMs. By enforcing strict logical agreement between referring counting and object detection during inference, MCC substantially boosts both grounding accuracy and negative sample rejection reliability.

(a) Motivation of MCC

(b) Overall Pipeline of MCC

Experimental Results

Extensive evaluations demonstrate that our training-free MCC significantly alleviates logical hallucinations and boosts grounding performance across multiple state-of-the-art vision-language models.

Grounding Accuracy (OpenRef)
Rejection Reliability (N3R)
Generalization (gRefCOCO)

Model	Landmark	Celebrity	Logo	Polysemy	Ordinal	Single	Multi	Avg.
Qwen3-VL-2B	83.1	89.1	35.0	63.5	12.3	66.7	29.6	54.2
+ MCC	89.6	86.6	35.6	68.1	11.8	66.2	37.7	56.5 (↑2.3)
GLM-4.6V	86.5	83.5	55.9	68.2	24.1	72.9	21.1	58.9
+ MCC	85.3	81.6	60.0	84.9	23.6	72.6	60.0	66.9 (↑8.0)
InternVL-3.5	72.5	77.0	2.9	52.1	35.4	36.0	9.4	40.8
+ MCC	78.2	77.0	14.8	62.1	37.5	49.5	17.0	48.0 (↑7.2)
LLaVA-OV-1.5	70.5	75.7	7.6	59.1	6.7	19.1	8.7	35.3
+ MCC	69.4	69.9	8.1	63.2	5.1	20.2	8.5	34.9 (↓0.4)
Keye-VL-1.5	67.9	77.0	12.4	64.4	23.6	26.4	13.3	40.7
+ MCC	73.6	79.7	9.8	68.0	29.7	27.0	16.0	43.4 (↑2.7)

Model	Landmark		Celebrity		Logo		Ordinal		Complex scenarios		Avg.
Model	N_abs	N3R	N_abs	N3R	N_abs	N3R	N_abs	N3R	N_abs	N3R	N_abs	N3R
Qwen3-VL-2B	91.7	92.7	56.1	60.9	12.1	23.1	0.0	4.5	4.0	9.2	32.8	38.1
+ MCC	100.0	100.0	96.7	97.0	90.2	91.5	92.3	94.4	75.0	76.4	90.8 (↑58.0)	91.9 (↑53.8)
GLM-4.6V	100.0	100.0	86.2	90.3	90.8	94.0	17.4	41.4	58.8	67.8	70.6	78.7
+ MCC	100.0	100.0	83.3	87.8	88.0	90.4	4.1	23.1	63.1	63.7	67.7 (↓2.9)	73.0 (↓5.7)
InternVL-3.5	66.3	68.9	36.4	52.3	38.1	42.8	48.7	56.6	29.2	34.6	43.8	51.0
+ MCC	100.0	100.0	72.4	84.2	66.4	74.1	14.9	28.4	48.6	59.3	60.5 (↑16.7)	69.2 (↑18.2)
LLaVA-OV-1.5	100.0	100.0	94.14	94.53	73.4	74.7	22.1	27.8	77.9	78.6	73.5	75.1
+ MCC	99.5	99.7	98.7	99.4	84.5	91.7	95.4	97.7	90.6	95.3	93.7 (↑20.2)	96.8 (↑21.7)
Keye-VL-1.5	93.3	96.4	59.0	74.0	78.0	83.6	4.1	1.0	28.3	55.6	52.5	62.1
+ MCC	94.8	96.5	78.2	83.3	84.0	85.8	44.6	52.2	71.6	77.6	74.6 (↑22.1)	79.1 (↑17.0)

Model	val		testA		testB		Avg.
Model	single	multi	single	multi	single	multi	Avg.
Qwen3-VL	-	62.4	90.2	60.3	83.1	57.8	70.8
+ MCC	-	68.8	87.4	65.6	76.1	64.4	72.5 (↑1.7)
GLM-4.6V	-	18.9	88.2	14.2	84.0	12.0	43.5
+ MCC	-	73.7	86.6	77.9	82.7	72.5	78.7 (↑35.2)
LLaVA-OV-1.5	-	27.7	63.1	21.5	48.6	17.9	35.8
+ MCC	-	44.3	74.2	36.7	60.0	36.5	50.3 (↑14.5)
InternVL-3.5	-	53.0	94.6	38.3	88.6	33.2	61.5
+ MCC	-	54.8	93.8	52.2	87.5	50.4	67.7 (↑6.2)
Keye-VL-1.5	-	54.6	76.8	48.3	71.5	44.4	59.1
+ MCC	-	58.5	81.1	56.7	74.3	53.6	64.8 (↑5.7)

BibTeX

@article{park2021nerfies,
  author    = {Park, Keunhong and Sinha, Utkarsh batting, Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Seitz, Steven M. and Martin-Brualla, Ricardo},
  title     = {Nerfies: Deformable Neural Radiance Fields},
  journal   = {ICCV},
  year      = {2021},
}