abb3337f8d
Architecture v2: - Query branch: drone + text -> GatedFusion -> proj -> query_emb - Gallery branch: satellite -> proj -> gallery_emb - Single InfoNCE loss (asymmetric 0.6/0.4) - GatedFusion: learnable gated addition (sigma(alpha)*img + (1-sigma(alpha))*text) - Baseline mode: gate=1.0 (text ignored) Dataset: - UAV-GeoLoc loader with template captions from path metadata - 27 terrain types with predefined features - Random positive crop sampling per epoch Configs: balanced (gate=0.7), baseline (no text), text_heavy (gate=0.3) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
115 lines
3.2 KiB
Markdown
115 lines
3.2 KiB
Markdown
# Caption Quality Test for Cross-View Geo-Localization
|
|
|
|
Validate whether generated text captions improve retrieval R@1 in cross-view
|
|
geo-localization (drone-to-satellite). Uses GeoRSCLIP ViT-B/32 dual encoder
|
|
with GatedFusion on the query branch.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Query: drone_img + caption -> GatedFusion -> proj -> query_emb
|
|
Gallery: sat_img -> proj -> gallery_emb
|
|
Loss: InfoNCE(query, gallery)
|
|
```
|
|
|
|
Baseline: fusion gate = 1.0 (text ignored).
|
|
|
|
## Structure
|
|
|
|
```
|
|
caption_test/
|
|
├── conf/
|
|
│ ├── balanced.gin # Primary: gate init 0.7 (30% text)
|
|
│ ├── baseline_no_text.gin # Reference: gate = 1.0 (no text)
|
|
│ └── text_heavy.gin # Stress: gate init 0.3 (70% text)
|
|
├── scripts/
|
|
│ ├── generate_captions.py # Offline caption generation
|
|
│ └── compare_runs.py # Delta R@1 comparison report
|
|
├── src/
|
|
│ ├── datasets/
|
|
│ │ └── visloc_with_captions.py # UAV-GeoLoc loader + template captions
|
|
│ ├── models/
|
|
│ │ └── dual_encoder.py # GeoRSCLIP + GatedFusion + projection heads
|
|
│ ├── losses/
|
|
│ │ └── multi_infonce.py # InfoNCE with cosine temperature
|
|
│ ├── training/
|
|
│ │ └── train.py # Main training loop
|
|
│ └── eval/
|
|
│ └── evaluate.py # R@K metrics, Delta R@1
|
|
└── checkpoints/ # RS5M_ViT-B-32.pt (user-provided)
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
```
|
|
torch>=2.0
|
|
open_clip_torch
|
|
gin-config
|
|
Pillow
|
|
numpy
|
|
```
|
|
|
|
GeoRSCLIP checkpoint: download `RS5M_ViT-B-32.pt` from
|
|
`github.com/om-ai-lab/RS5M` and place under `checkpoints/`.
|
|
|
|
## Workflow
|
|
|
|
### 1. Train baseline (no text)
|
|
|
|
```bash
|
|
python -m src.training.train --config conf/baseline_no_text.gin
|
|
```
|
|
|
|
### 2. Train with captions
|
|
|
|
```bash
|
|
python -m src.training.train --config conf/balanced.gin
|
|
```
|
|
|
|
### 3. Compare and get verdict
|
|
|
|
```bash
|
|
python -m scripts.compare_runs \
|
|
--baseline_report out/caption_test/baseline_no_text/eval_report.json \
|
|
--full_report out/caption_test/balanced/eval_report.json \
|
|
--output out/caption_test/comparison.md
|
|
```
|
|
|
|
## Decision rule
|
|
|
|
| Delta R@1 (query->gallery) | Verdict |
|
|
|---|---|
|
|
| >= +3% | PASS -- captions informative, proceed to production |
|
|
| +1% to +3% | MARGINAL -- add VLM refinement, re-run |
|
|
| 0 to +1% | WEAK -- redesign caption pipeline |
|
|
| < 0 | HARMFUL -- critical bug |
|
|
|
|
## Expected runtime (RTX 4090, 24 GB)
|
|
|
|
| Phase | Time |
|
|
|---|---|
|
|
| Single training run (10 epochs, batch 128, 206K queries) | ~15-30 min |
|
|
| Full test (baseline + balanced + text_heavy) | ~1-1.5 h |
|
|
| Evaluation | ~2-5 min per run |
|
|
|
|
## Dataset
|
|
|
|
UAV-GeoLoc Terrain split (from `/mnt/data1tb/cvgl_datasets/UAV-GeoLoc/`):
|
|
- Train: 206,108 queries, 94,709 DB crops (140 scenes)
|
|
- Val: 62,368 queries, 26,597 DB crops (40 scenes)
|
|
- Test: 33,472 queries, 11,684 DB crops (20 scenes)
|
|
|
|
Template captions generated automatically from path metadata:
|
|
```
|
|
"Aerial view at 100m facing northwest over volcanic terrain near KilaueaVolcano.
|
|
Plan-view features: lava flows, crater edges, volcanic rock."
|
|
```
|
|
|
|
## Code style
|
|
|
|
- `from __future__ import annotations` everywhere
|
|
- Type hints on all signatures
|
|
- Google-style docstrings
|
|
- `@gin.configurable` on top-level classes
|
|
- No emojis in code, English-only comments
|