161 lines
6.7 KiB
Markdown
161 lines
6.7 KiB
Markdown
# Caption Quality Test for Cross-View Geo-Localization
|
||
|
||
Validate whether generated text captions improve retrieval R@1 in cross-view
|
||
geo-localization (drone-to-satellite).
|
||
|
||
## Experiments
|
||
|
||
### V3 — GTA-UAV + DINOv3 + LRSCLIP (active)
|
||
|
||
Asymmetric architecture with domain-specific image encoders and hierarchical text.
|
||
|
||
```
|
||
┌─────────────────────────── QUERY BRANCH ───────────────────────────┐
|
||
│ │
|
||
│ drone_img ──► DINOv3 ViT-L/16 LVD ──► CLS [1024] │
|
||
│ (frozen, 303M) │ │
|
||
│ proj_drone [1024→512] │
|
||
│ │ │
|
||
│ L1 (overview) ──► DGTRS-CLIP ──► [768] ─┐ │
|
||
│ L2 (full desc) ──► DGTRS-CLIP ──► [768] ─┼─ concat [2304] │
|
||
│ L3 (fingerprint) ──► DGTRS-CLIP ──► [768] ─┘ │ │
|
||
│ (248 tok) MLP [2304→768→512] │
|
||
│ │ │
|
||
│ GatedFusion(img_512, text_512) │
|
||
│ │ │
|
||
│ L2-norm ──► query [512] │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────── GALLERY BRANCH ─────────────────────────┐
|
||
│ │
|
||
│ sat_img ──► DINOv3 ViT-L/16 SAT ──► CLS [1024] │
|
||
│ (frozen, 303M) │ │
|
||
│ proj_sat [1024→512] │
|
||
│ │ │
|
||
│ L2-norm ──► gallery [512] │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
|
||
LOSS: InfoNCE(query, gallery) — symmetric, learnable temperature
|
||
weights: 0.6 × q→g + 0.4 × g→q
|
||
|
||
BASELINE: gate = 1.0 (text branch disabled, no DGTRS loaded)
|
||
```
|
||
|
||
**Models:**
|
||
- Drone: DINOv3 ViT-L/16 (LVD-1689M, web pretrained) — 1024-dim, 303M params, frozen
|
||
- Satellite: DINOv3 ViT-L/16 (SAT-493M, satellite pretrained) — 1024-dim, 303M params, frozen
|
||
- Text: DGTRS-CLIP ViT-L-14 (LRSCLIP, 248 tokens) — 768-dim, partial unfreeze
|
||
- Total: 733M params, 10.9M trainable (1.49%)
|
||
|
||
**Input:** 256x256, ImageNet normalization
|
||
**Training:** learnable temperature (CLIP logit_scale), per-group LR (proj 1e-4 / text 1e-5), warmup 2 epochs + cosine, augmentations (drone: crop+flip+rot+jitter+blur, sat: crop+flip+jitter)
|
||
|
||
**Dataset:** GTA-UAV-LR (33K drone + 14K satellite, GTA V synthetic)
|
||
- RGB: `/home/servml/Документы/datasets/GTA-UAV-LR/`
|
||
- Captions: `/home/servml/Документы/datasets/GTA-UAV-LR-captions/` (40K JSON, 3-paragraph VLM)
|
||
- Segmentation: `/home/servml/Документы/datasets/GTA-UAV-LR-aug/` (17 classes)
|
||
- Seg filter: 37,498 passed / 10,905 excluded (>=90% background+water)
|
||
- Split: 80/20 random (26,966 train / 6,742 test → 24,891/6,252 after seg filter)
|
||
|
||
### V2 — UAV-GeoLoc + GeoRSCLIP (legacy)
|
||
|
||
Single backbone with template captions.
|
||
|
||
```
|
||
Query: drone_img + caption -> GeoRSCLIP -> GatedFusion -> query
|
||
Gallery: sat_img -> GeoRSCLIP -> gallery
|
||
```
|
||
|
||
**Dataset:** UAV-GeoLoc Terrain split (206K train queries)
|
||
|
||
## Structure
|
||
|
||
```
|
||
caption-test/
|
||
├── conf/ # Gin configs (v2)
|
||
│ ├── balanced.gin
|
||
│ ├── baseline_no_text.gin
|
||
│ └── text_heavy.gin
|
||
├── nn_models/ # Pre-trained checkpoints (v3)
|
||
│ ├── DINO_WEB/ # DINOv3 ViT-L/16 LVD-1689M
|
||
│ ├── DINO_SAT/ # DINOv3 ViT-L/16 SAT-493M
|
||
│ └── LRSCLIP/ # DGTRS-CLIP ViT-L-14
|
||
├── scripts/
|
||
│ ├── filter_segmentation.py # Meta-file: exclude 90%+ background/water
|
||
│ ├── compare_runs.py # Delta R@1 comparison report
|
||
│ └── generate_captions.py # Offline caption generation
|
||
├── src/
|
||
│ ├── datasets/
|
||
│ │ ├── gtauav_dataset.py # GTA-UAV-LR loader + L1/L2/L3 captions (v3)
|
||
│ │ └── visloc_with_captions.py # UAV-GeoLoc loader (v2)
|
||
│ ├── models/
|
||
│ │ ├── asymmetric_encoder.py # DINOv3 + LRSCLIP + GatedFusion (v3)
|
||
│ │ └── dual_encoder.py # GeoRSCLIP + GatedFusion (v2)
|
||
│ ├── losses/
|
||
│ │ └── multi_infonce.py # InfoNCE with cosine temperature
|
||
│ ├── training/
|
||
│ │ ├── train_gtauav.py # Training loop GTA-UAV (v3)
|
||
│ │ └── train.py # Training loop UAV-GeoLoc (v2)
|
||
│ └── eval/
|
||
│ └── evaluate.py # R@K metrics, Delta R@1
|
||
└── checkpoints/ # GeoRSCLIP RS5M_ViT-B-32.pt (v2)
|
||
```
|
||
|
||
## Prerequisites
|
||
|
||
```
|
||
torch>=2.0
|
||
open_clip_torch
|
||
safetensors
|
||
timm
|
||
gin-config
|
||
Pillow
|
||
numpy
|
||
```
|
||
|
||
## Workflow (V3 — GTA-UAV)
|
||
|
||
### 1. Create 80/20 split and filter segmentation
|
||
|
||
```bash
|
||
python -m scripts.make_split --output-dir meta
|
||
python -m scripts.filter_segmentation --output meta/seg_filter.json
|
||
```
|
||
|
||
### 2. Train baseline (no text)
|
||
|
||
```bash
|
||
python -m src.training.train_gtauav --baseline --filter-meta meta/seg_filter.json
|
||
```
|
||
|
||
### 3. Train with captions (L1/L2/L3)
|
||
|
||
```bash
|
||
python -m src.training.train_gtauav --filter-meta meta/seg_filter.json
|
||
```
|
||
|
||
### 4. Compare and get verdict
|
||
|
||
```bash
|
||
python -m scripts.compare_runs \
|
||
--baseline_report out/gtauav/baseline/eval_report.json \
|
||
--full_report out/gtauav/with_text/eval_report.json \
|
||
--output out/gtauav/comparison.md
|
||
```
|
||
|
||
## Decision rule
|
||
|
||
| Delta R@1 (drone->satellite) | Verdict |
|
||
|---|---|
|
||
| >= +3% | PASS -- captions informative, proceed to NADEZHDA teacher |
|
||
| +1% to +3% | MARGINAL -- add VLM refinement, re-run |
|
||
| 0 to +1% | WEAK -- redesign caption pipeline |
|
||
| < 0 | HARMFUL -- critical bug |
|
||
|
||
## Code style
|
||
|
||
- `from __future__ import annotations` everywhere
|
||
- Type hints on all signatures
|
||
- Google-style docstrings
|
||
- No emojis in code, English-only comments
|