# Caption Quality Test for Cross-View Geo-Localization Validate whether generated text captions improve retrieval R@1 in cross-view geo-localization (drone-to-satellite). ## Architecture ### Overview Asymmetric dual-encoder with domain-specific frozen backbones and hierarchical text fusion for drone-to-satellite image retrieval. ``` ┌──────────────────────────── QUERY BRANCH ────────────────────────────┐ │ │ │ drone_img ──► DINOv3 ViT-L/16 LVD ──► CLS token │ │ [B,3,256,256] (frozen, 303M) d_img [B,1024] │ │ │ │ │ L1 (overview) ──► DGTRS-CLIP ──► z₁ [B,768] ─┐ │ │ L2 (full desc) ──► DGTRS-CLIP ──► z₂ [B,768] ─┼─ cat ──► [B,2304]│ │ L3 (fingerprint) ──► DGTRS-CLIP ──► z₃ [B,768] ─┘ │ │ │ (248 tokens, KPS pos. emb.) MLP(2304→1024→1024) │ │ │ │ │ d_txt [B,1024] │ │ │ │ │ q = σ(α_q)·d_img + (1−σ(α_q))·d_txt GatedFusion_q │ │ │ │ │ q̂ = q / ‖q‖₂ ──► query [B,1024] │ └───────────────────────────────────────────────────────────────────────┘ ┌──────────────────────────── GALLERY BRANCH ──────────────────────────┐ │ │ │ sat_img ──► DINOv3 ViT-L/16 SAT ──► CLS token │ │ [B,3,256,256] (frozen, 303M) s_img [B,1024] │ │ │ │ │ sat_L1 ──► DGTRS-CLIP ──► z₁ [768] ─┐ │ │ sat_L2 ──► DGTRS-CLIP ──► z₂ [768] ─┼─ cat ──► MLP ──► s_txt [1024]│ │ sat_L3 ──► DGTRS-CLIP ──► z₃ [768] ─┘ (shared MLP) │ │ │ │ │ g = σ(α_g)·s_img + (1−σ(α_g))·s_txt GatedFusion_g │ │ │ │ │ ĝ = g / ‖g‖₂ ──► gallery [B,1024]│ └───────────────────────────────────────────────────────────────────────┘ Retrieval space: 1024-dim (DINOv3 native, no projection layers) TextFusionMLP shared between query and gallery branches For sat images without captions: s_txt=None → g = s_img (gate passthrough) BASELINE: σ(α) = 1.0 for both branches (text disabled, DGTRS not loaded) ``` ### Text hierarchy (L1 / L2 / L3) Each drone image has a VLM-generated caption (Qwen3-VL) split into 3 levels: | Level | Name | Content | Typical length | |-------|------|---------|----------------| | **L1** | Overview | First sentence of P1: land-cover summary with class percentages | 15–30 tokens | | **L2** | Full description | Complete P1 (inventory) + P2 (spatial layout with 5+ zones) | 100–200 tokens | | **L3** | Fingerprint | P3: unique landmarks and spatial signature for matching | 20–50 tokens | All three levels are encoded by a **single DGTRS-CLIP ViT-L-14** text encoder (248-token context via KPS positional embedding, 768-dim output) adapted with **LoRA** (rank=4 on Q/V in all 12 blocks). ### Text fusion (shared MLP for both branches) $$\mathbf{z}_{\text{text}} = \text{MLP}\bigl([\mathbf{z}_1 \;;\; \mathbf{z}_2 \;;\; \mathbf{z}_3]\bigr)$$ where $[\mathbf{z}_1 ; \mathbf{z}_2 ; \mathbf{z}_3] \in \mathbb{R}^{B \times 2304}$ is the concatenation of three 768-dim DGTRS-CLIP embeddings, and $$\text{MLP}: \text{Linear}(2304, 1024) \to \text{GELU} \to \text{Linear}(1024, 1024), \quad \mathbf{z}_{\text{text}} \in \mathbb{R}^{B \times 1024}$$ ### Gated fusion (separate gates for query and gallery) $$\mathbf{q} = \sigma(\alpha_q) \cdot \mathbf{d}_{\text{img}} + \bigl(1 - \sigma(\alpha_q)\bigr) \cdot \mathbf{d}_{\text{txt}} \qquad \text{(query branch)}$$ $$\mathbf{g} = \sigma(\alpha_g) \cdot \mathbf{s}_{\text{img}} + \bigl(1 - \sigma(\alpha_g)\bigr) \cdot \mathbf{s}_{\text{txt}} \qquad \text{(gallery branch)}$$ - $\alpha_q, \alpha_g$ — separate learnable scalars in logit-space, init $\sigma(\alpha) \approx 0.7$ - $\sigma$ — sigmoid function - $\mathbf{d}_{\text{img}}, \mathbf{s}_{\text{img}} \in \mathbb{R}^{B \times 1024}$ — DINOv3+MONA image embeddings - $\mathbf{d}_{\text{txt}}, \mathbf{s}_{\text{txt}} \in \mathbb{R}^{B \times 1024}$ — fused text embeddings - For satellite images without captions: $\mathbf{s}_{\text{txt}} = \text{None} \Rightarrow \mathbf{g} = \mathbf{s}_{\text{img}}$ ### Adaptation methods | Method | Applied to | Where | Params | |--------|-----------|-------|--------| | **MONA** (CVPR 2025) | DINOv3 ViT-L/16 (drone + sat) | After MSA and MLP in each of 24 blocks | 7.0M per encoder | | **LoRA** (rank=4) | DGTRS-CLIP text encoder | Q and V projections in all 12 blocks | 147K | **MONA adapter** (per block): $$\mathbf{x} \leftarrow \mathbf{x} + \text{Up}_{64 \to 1024}\!\Bigl(\text{GELU}\bigl(\text{MonaOp}\bigl(\text{Down}_{1024 \to 64}(\hat{\mathbf{x}})\bigr)\bigr)\Bigr)$$ where $\hat{\mathbf{x}} = \gamma \cdot \text{LN}(\mathbf{x}) + \gamma_x \cdot \mathbf{x}$ (scaled LayerNorm, $\gamma$ init $10^{-6}$, $\gamma_x$ init $1$) $$\text{MonaOp}(\mathbf{x}) = \frac{\text{DWConv}_{3 \times 3}(\mathbf{x}) + \text{DWConv}_{5 \times 5}(\mathbf{x}) + \text{DWConv}_{7 \times 7}(\mathbf{x})}{3} + \mathbf{x}$$ **LoRA** (per attention layer): $$\mathbf{Q}' = \mathbf{Q} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_Q^T \mathbf{B}_Q^T, \qquad \mathbf{V}' = \mathbf{V} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_V^T \mathbf{B}_V^T$$ where $\mathbf{A} \in \mathbb{R}^{r \times d}$, $\mathbf{B} \in \mathbb{R}^{d \times r}$, $r = 4$ ### Loss function Symmetric InfoNCE with learnable temperature (CLIP-style `logit_scale`): $$\mathcal{L} = w_{q \to g} \cdot \mathcal{L}_{q \to g} + w_{g \to q} \cdot \mathcal{L}_{g \to q}$$ $$\mathcal{L}_{q \to g} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{q}} \cdot \hat{\mathbf{g}}^T}{\tau},\; \text{targets}\right), \qquad \mathcal{L}_{g \to q} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{g}} \cdot \hat{\mathbf{q}}^T}{\tau},\; \text{targets}\right)$$ - $\tau = 1 / \exp(\text{logit\_scale})$ — learnable scalar, clamped $\tau \in [0.01, 0.5]$, init $\tau_0 = 0.07$ - $w_{q \to g} = 0.6$, $w_{g \to q} = 0.4$ - $\text{targets} = [0, 1, 2, \ldots, B-1]$ — positives on diagonal - label smoothing $= 0.1$ Loss and adapters run in **fp32** (AMP autocast disabled) to prevent gradient overflow. ### Metrics | Metric | Formula | Direction | |--------|---------|-----------| | **R@K** (Recall at K) | fraction of queries where correct gallery is in top-K | drone → satellite (primary) | | **Delta R@1** | R@1(with_text) − R@1(baseline) | higher = text helps | Reported: R@1, R@5, R@10 for both q→g and g→q directions. ### Optimizer & scheduler ``` Optimizer: AdamW - MONA adapters, TextFusionMLP, gate α_q, gate α_g, logit_scale: lr = 1e-4, weight_decay = 1e-4 - LoRA adapters (DGTRS-CLIP text encoder): lr = 1e-5 (10× lower, --text-lr-factor 0.1) Scheduler: Linear warmup (2 epochs) + cosine annealing - Per-step (not per-epoch) - warmup: lr linearly ramps from 0 to lr_max over warmup_steps - cosine: lr decays from lr_max to 0 over remaining steps Gradient clipping: max_norm = 1.0 Mixed precision: AMP fp16 for model forward, fp32 for loss ``` ### Augmentations | Transform | Drone (train) | Satellite (train) | Eval | |-----------|:---:|:---:|:---:| | RandomResizedCrop(256, scale=0.7–1.0) | ✓ | ✓ | — | | Resize(256) + CenterCrop(256) | — | — | ✓ | | RandomHorizontalFlip(0.5) | ✓ | ✓ | — | | RandomRotation(15°) | ✓ | — | — | | ColorJitter(0.3, 0.3, 0.2, 0.1) | ✓ | ✓ | — | | RandomGrayscale(0.05) | ✓ | ✓ | — | | GaussianBlur(k=3, σ=0.1–2.0) | ✓ | — | — | | ImageNet Normalize | ✓ | ✓ | ✓ | ### Model summary | Component | Params | Trainable | Notes | |-----------|--------|-----------|-------| | DINOv3 ViT-L/16 LVD (drone) | 303M | frozen | backbone weights frozen | | MONA adapters (drone) | 7.0M | 7.0M | 2 per block × 24 blocks, bottleneck=64 | | DINOv3 ViT-L/16 SAT (satellite) | 303M | frozen | backbone weights frozen | | MONA adapters (satellite) | 7.0M | 7.0M | 2 per block × 24 blocks, bottleneck=64 | | DGTRS-CLIP ViT-L-14 (text) | 124M | frozen | backbone weights frozen | | LoRA adapters (text) | 147K | 147K | Q+V, rank=4, 12 blocks | | TextFusionMLP (shared) | 3.4M | 3.4M | Linear(2304,1024) + GELU + Linear(1024,1024) | | GatedFusion α_q + α_g | 2 | 2 | separate gate scalars | | logit_scale | 1 | 1 | learnable temperature | | **Total** | **748M** | **17.6M (2.35%)** | retrieval dim = 1024 | ## Experiments ### V3 — GTA-UAV + DINOv3 + DGTRS-CLIP (active) **Dataset:** GTA-UAV-LR (33K drone + 14K satellite, GTA V synthetic) - RGB: `/home/servml/Документы/datasets/GTA-UAV-LR/` - Captions: `/home/servml/Документы/datasets/GTA-UAV-LR-captions/` (40K JSON, 3-paragraph VLM) - Segmentation: `/home/servml/Документы/datasets/GTA-UAV-LR-aug/` (17 classes) - Seg filter: 37,498 passed / 10,905 excluded (>=90% background+water) - Split: 80/20 random (26,966 train / 6,742 test → 24,891/6,252 after seg filter) ### V2 — UAV-GeoLoc + GeoRSCLIP (legacy) Single backbone with template captions. ``` Query: drone_img + caption -> GeoRSCLIP -> GatedFusion -> query Gallery: sat_img -> GeoRSCLIP -> gallery ``` **Dataset:** UAV-GeoLoc Terrain split (206K train queries) ## Structure ``` caption-test/ ├── conf/ # Gin configs (v2) │ ├── balanced.gin │ ├── baseline_no_text.gin │ └── text_heavy.gin ├── nn_models/ # Pre-trained checkpoints (v3, gitignored) │ ├── DINO_WEB/ # DINOv3 ViT-L/16 LVD-1689M (.pth) │ ├── DINO_SAT/ # DINOv3 ViT-L/16 SAT-493M (.safetensors) │ └── LRSCLIP/ # DGTRS-CLIP ViT-L-14 (.pt) ├── meta/ # Generated metadata │ ├── train_80.json # 80% train split (26,966 pairs) │ ├── test_20.json # 20% test split (6,742 pairs) │ └── seg_filter.json # Segmentation filter results ├── scripts/ │ ├── make_split.py # Create 80/20 train/test split │ ├── filter_segmentation.py # Exclude 90%+ background/water images │ ├── compare_runs.py # Delta R@1 comparison report │ └── generate_captions.py # Offline caption generation ├── src/ │ ├── datasets/ │ │ ├── gtauav_dataset.py # GTA-UAV-LR loader + L1/L2/L3 parsing (v3) │ │ └── visloc_with_captions.py # UAV-GeoLoc loader (v2) │ ├── models/ │ │ ├── dgtrs/ # Official DGTRS-CLIP text encoder (Apache-2.0) │ │ │ ├── model.py # DGTRSTextEncoder, build_model, tokenize │ │ │ ├── simple_tokenizer.py # BPE tokenizer (248 tokens) │ │ │ └── bpe_simple_vocab_16e6.txt.gz │ │ ├── asymmetric_encoder.py # DINOv3 + DGTRS + GatedFusion (v3) │ │ └── dual_encoder.py # GeoRSCLIP + GatedFusion (v2) │ ├── losses/ │ │ └── multi_infonce.py # InfoNCE with learnable temperature │ ├── training/ │ │ ├── train_gtauav.py # Training loop GTA-UAV (v3) │ │ └── train.py # Training loop UAV-GeoLoc (v2) │ └── eval/ │ └── evaluate.py # R@K metrics, Delta R@1 └── checkpoints/ # GeoRSCLIP RS5M_ViT-B-32.pt (v2) ``` ## Prerequisites ``` torch>=2.0 safetensors coloredlogs tqdm ftfy regex gin-config Pillow numpy ``` ## Workflow (V3 — GTA-UAV) ### 1. Create 80/20 split and filter segmentation ```bash python -m scripts.make_split --output-dir meta python -m scripts.filter_segmentation --output meta/seg_filter.json ``` ### 2. Train baseline (no text) ```bash python -m src.training.train_gtauav --baseline --filter-meta meta/seg_filter.json ``` ### 3. Train with captions (L1/L2/L3) ```bash python -m src.training.train_gtauav --filter-meta meta/seg_filter.json ``` ### 4. Resume from checkpoint ```bash python -m src.training.train_gtauav --resume out/gtauav/with_text/ckpt_epoch004.pt \ --filter-meta meta/seg_filter.json ``` ### 5. Compare and get verdict ```bash python -m scripts.compare_runs \ --baseline_report out/gtauav/baseline/eval_report.json \ --full_report out/gtauav/with_text/eval_report.json \ --output out/gtauav/comparison.md ``` ## Decision rule | Delta R@1 (drone→satellite) | Verdict | |---|---| | >= +3% | **PASS** — captions informative, proceed to NADEZHDA teacher | | +1% to +3% | MARGINAL — add VLM refinement, re-run | | 0 to +1% | WEAK — redesign caption pipeline | | < 0 | HARMFUL — critical bug | ## Code style - `from __future__ import annotations` everywhere - Type hints on all signatures - Google-style docstrings - English-only comments