Caption Quality Test for Cross-View Geo-Localization

Validate whether generated text captions improve retrieval R@1 in cross-view geo-localization (drone-to-satellite). Uses GeoRSCLIP ViT-B/32 dual encoder with GatedFusion on the query branch.

Architecture

Query:   drone_img + caption -> GatedFusion -> proj -> query_emb
Gallery: sat_img -> proj -> gallery_emb
Loss:    InfoNCE(query, gallery)

Baseline: fusion gate = 1.0 (text ignored).

Structure

caption_test/
├── conf/
│   ├── balanced.gin          # Primary: gate init 0.7 (30% text)
│   ├── baseline_no_text.gin  # Reference: gate = 1.0 (no text)
│   └── text_heavy.gin        # Stress: gate init 0.3 (70% text)
├── scripts/
│   ├── generate_captions.py  # Offline caption generation
│   └── compare_runs.py       # Delta R@1 comparison report
├── src/
│   ├── datasets/
│   │   └── visloc_with_captions.py  # UAV-GeoLoc loader + template captions
│   ├── models/
│   │   └── dual_encoder.py   # GeoRSCLIP + GatedFusion + projection heads
│   ├── losses/
│   │   └── multi_infonce.py  # InfoNCE with cosine temperature
│   ├── training/
│   │   └── train.py          # Main training loop
│   └── eval/
│       └── evaluate.py       # R@K metrics, Delta R@1
└── checkpoints/              # RS5M_ViT-B-32.pt (user-provided)

Prerequisites

torch>=2.0
open_clip_torch
gin-config
Pillow
numpy

GeoRSCLIP checkpoint: download RS5M_ViT-B-32.pt from github.com/om-ai-lab/RS5M and place under checkpoints/.

Workflow

1. Train baseline (no text)

python -m src.training.train --config conf/baseline_no_text.gin

2. Train with captions

python -m src.training.train --config conf/balanced.gin

3. Compare and get verdict

python -m scripts.compare_runs \
    --baseline_report out/caption_test/baseline_no_text/eval_report.json \
    --full_report out/caption_test/balanced/eval_report.json \
    --output out/caption_test/comparison.md

Decision rule

Delta R@1 (query->gallery)	Verdict
>= +3%	PASS -- captions informative, proceed to production
+1% to +3%	MARGINAL -- add VLM refinement, re-run
0 to +1%	WEAK -- redesign caption pipeline
< 0	HARMFUL -- critical bug

Expected runtime (RTX 4090, 24 GB)

Phase	Time
Single training run (10 epochs, batch 128, 206K queries)	~15-30 min
Full test (baseline + balanced + text_heavy)	~1-1.5 h
Evaluation	~2-5 min per run

Dataset

UAV-GeoLoc Terrain split (from /mnt/data1tb/cvgl_datasets/UAV-GeoLoc/):

• Train: 206,108 queries, 94,709 DB crops (140 scenes)
• Val: 62,368 queries, 26,597 DB crops (40 scenes)
• Test: 33,472 queries, 11,684 DB crops (20 scenes)

Template captions generated automatically from path metadata:

"Aerial view at 100m facing northwest over volcanic terrain near KilaueaVolcano.
 Plan-view features: lava flows, crater edges, volcanic rock."

Code style

• from __future__ import annotations everywhere
• Type hints on all signatures
• Google-style docstrings
• @gin.configurable on top-level classes
• No emojis in code, English-only comments