# Caption Quality Test for Cross-View Geo-Localization

Validate whether generated text captions improve retrieval R@1 in cross-view
geo-localization (drone-to-satellite).

## Experiments

### V3 — GTA-UAV + DINOv3 + LRSCLIP (active)

Asymmetric architecture with domain-specific image encoders and hierarchical text.

```
┌─────────────────────────── QUERY BRANCH ───────────────────────────┐
│                                                                     │
│  drone_img ──► DINOv3 ViT-L/16 LVD ──► CLS [1024]                 │
│                  (frozen, 303M)            │                        │
│                                      proj_drone [1024→512]         │
│                                            │                        │
│  L1 (overview)    ──► DGTRS-CLIP ──► [768] ─┐                      │
│  L2 (full desc)   ──► DGTRS-CLIP ──► [768] ─┼─ concat [2304]       │
│  L3 (fingerprint) ──► DGTRS-CLIP ──► [768] ─┘      │               │
│                       (248 tok)       MLP [2304→768→512]            │
│                                            │                        │
│                  GatedFusion(img_512, text_512)                     │
│                          │                                          │
│                     L2-norm ──► query [512]                         │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────── GALLERY BRANCH ─────────────────────────┐
│                                                                     │
│  sat_img ──► DINOv3 ViT-L/16 SAT ──► CLS [1024]                   │
│                (frozen, 303M)            │                          │
│                                    proj_sat [1024→512]             │
│                                          │                          │
│                                     L2-norm ──► gallery [512]      │
└─────────────────────────────────────────────────────────────────────┘

LOSS: InfoNCE(query, gallery) — symmetric, learnable temperature
      weights: 0.6 × q→g + 0.4 × g→q

BASELINE: gate = 1.0 (text branch disabled, no DGTRS loaded)
```

**Models:**
- Drone: DINOv3 ViT-L/16 (LVD-1689M, web pretrained) — 1024-dim, 303M params, frozen
- Satellite: DINOv3 ViT-L/16 (SAT-493M, satellite pretrained) — 1024-dim, 303M params, frozen
- Text: DGTRS-CLIP ViT-L-14 (LRSCLIP, 248 tokens) — 768-dim, partial unfreeze
- Total: 733M params, 10.9M trainable (1.49%)

**Input:** 256x256, ImageNet normalization
**Training:** learnable temperature (CLIP logit_scale), per-group LR (proj 1e-4 / text 1e-5), warmup 2 epochs + cosine, augmentations (drone: crop+flip+rot+jitter+blur, sat: crop+flip+jitter)

**Dataset:** GTA-UAV-LR (33K drone + 14K satellite, GTA V synthetic)
- RGB: `/home/servml/Документы/datasets/GTA-UAV-LR/`
- Captions: `/home/servml/Документы/datasets/GTA-UAV-LR-captions/` (40K JSON, 3-paragraph VLM)
- Segmentation: `/home/servml/Документы/datasets/GTA-UAV-LR-aug/` (17 classes)
- Seg filter: 37,498 passed / 10,905 excluded (>=90% background+water)
- Split: 80/20 random (26,966 train / 6,742 test → 24,891/6,252 after seg filter)

### V2 — UAV-GeoLoc + GeoRSCLIP (legacy)

Single backbone with template captions.

```
Query:   drone_img + caption -> GeoRSCLIP -> GatedFusion -> query
Gallery: sat_img -> GeoRSCLIP -> gallery
```

**Dataset:** UAV-GeoLoc Terrain split (206K train queries)

## Structure

```
caption-test/
├── conf/                         # Gin configs (v2)
│   ├── balanced.gin
│   ├── baseline_no_text.gin
│   └── text_heavy.gin
├── nn_models/                    # Pre-trained checkpoints (v3)
│   ├── DINO_WEB/                 # DINOv3 ViT-L/16 LVD-1689M
│   ├── DINO_SAT/                 # DINOv3 ViT-L/16 SAT-493M
│   └── LRSCLIP/                  # DGTRS-CLIP ViT-L-14
├── scripts/
│   ├── filter_segmentation.py    # Meta-file: exclude 90%+ background/water
│   ├── compare_runs.py           # Delta R@1 comparison report
│   └── generate_captions.py      # Offline caption generation
├── src/
│   ├── datasets/
│   │   ├── gtauav_dataset.py     # GTA-UAV-LR loader + L1/L2/L3 captions (v3)
│   │   └── visloc_with_captions.py  # UAV-GeoLoc loader (v2)
│   ├── models/
│   │   ├── asymmetric_encoder.py # DINOv3 + LRSCLIP + GatedFusion (v3)
│   │   └── dual_encoder.py       # GeoRSCLIP + GatedFusion (v2)
│   ├── losses/
│   │   └── multi_infonce.py      # InfoNCE with cosine temperature
│   ├── training/
│   │   ├── train_gtauav.py       # Training loop GTA-UAV (v3)
│   │   └── train.py              # Training loop UAV-GeoLoc (v2)
│   └── eval/
│       └── evaluate.py           # R@K metrics, Delta R@1
└── checkpoints/                  # GeoRSCLIP RS5M_ViT-B-32.pt (v2)
```

## Prerequisites

```
torch>=2.0
open_clip_torch
safetensors
timm
gin-config
Pillow
numpy
```

## Workflow (V3 — GTA-UAV)

### 1. Create 80/20 split and filter segmentation

```bash
python -m scripts.make_split --output-dir meta
python -m scripts.filter_segmentation --output meta/seg_filter.json
```

### 2. Train baseline (no text)

```bash
python -m src.training.train_gtauav --baseline --filter-meta meta/seg_filter.json
```

### 3. Train with captions (L1/L2/L3)

```bash
python -m src.training.train_gtauav --filter-meta meta/seg_filter.json
```

### 4. Compare and get verdict

```bash
python -m scripts.compare_runs \
    --baseline_report out/gtauav/baseline/eval_report.json \
    --full_report out/gtauav/with_text/eval_report.json \
    --output out/gtauav/comparison.md
```

## Decision rule

| Delta R@1 (drone->satellite) | Verdict |
|---|---|
| >= +3% | PASS -- captions informative, proceed to NADEZHDA teacher |
| +1% to +3% | MARGINAL -- add VLM refinement, re-run |
| 0 to +1% | WEAK -- redesign caption pipeline |
| < 0 | HARMFUL -- critical bug |

## Code style

- `from __future__ import annotations` everywhere
- Type hints on all signatures
- Google-style docstrings
- No emojis in code, English-only comments
