Add GTA-UAV experiment: asymmetric DINOv3 + LRSCLIP text encoder
V3 architecture for CVGL caption validation on GTA-UAV-LR dataset: - AsymmetricEncoder: DINOv3 ViT-L/16 (LVD drone + SAT satellite, frozen) + LRSCLIP/DGTRS-CLIP ViT-L-14 text encoder (248 tok, partial unfreeze) - L1/L2/L3 hierarchical captions from VLM-generated descriptions - TextFusionMLP (concat 3x768 -> MLP -> 512) + GatedFusion - Segmentation filter: exclude images with >=90% background+water - 10.9M trainable / 733M total params, 256x256 input - coloredlogs + tqdm + emoji for training UX - Baseline mode (--baseline): image-only, no text encoder loaded Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
128
README.md
128
README.md
@@ -1,42 +1,77 @@
|
||||
# Caption Quality Test for Cross-View Geo-Localization
|
||||
|
||||
Validate whether generated text captions improve retrieval R@1 in cross-view
|
||||
geo-localization (drone-to-satellite). Uses GeoRSCLIP ViT-B/32 dual encoder
|
||||
with GatedFusion on the query branch.
|
||||
geo-localization (drone-to-satellite).
|
||||
|
||||
## Architecture
|
||||
## Experiments
|
||||
|
||||
### V3 — GTA-UAV + DINOv3 + LRSCLIP (active)
|
||||
|
||||
Asymmetric architecture with domain-specific image encoders and hierarchical text.
|
||||
|
||||
```
|
||||
Query: drone_img + caption -> GatedFusion -> proj -> query_emb
|
||||
Gallery: sat_img -> proj -> gallery_emb
|
||||
Query: drone_img (DINOv3 LVD) + L1/L2/L3 captions (LRSCLIP) -> GatedFusion -> query
|
||||
Gallery: sat_img (DINOv3 SAT) -> gallery
|
||||
Loss: InfoNCE(query, gallery)
|
||||
```
|
||||
|
||||
Baseline: fusion gate = 1.0 (text ignored).
|
||||
**Models:**
|
||||
- Drone: DINOv3 ViT-L/16 (LVD-1689M, web pretrained) — 1024-dim, 303M params, frozen
|
||||
- Satellite: DINOv3 ViT-L/16 (SAT-493M, satellite pretrained) — 1024-dim, 303M params, frozen
|
||||
- Text: DGTRS-CLIP ViT-L-14 (LRSCLIP, 248 tokens) — 768-dim, partial unfreeze
|
||||
- Total: 733M params, 10.9M trainable (1.49%)
|
||||
|
||||
**Input:** 256x256, ImageNet normalization
|
||||
|
||||
**Dataset:** GTA-UAV-LR (33K drone + 14K satellite, GTA V synthetic)
|
||||
- RGB: `/home/servml/Документы/datasets/GTA-UAV-LR/`
|
||||
- Captions: `/home/servml/Документы/datasets/GTA-UAV-LR-captions/` (40K JSON, 3-paragraph VLM)
|
||||
- Segmentation: `/home/servml/Документы/datasets/GTA-UAV-LR-aug/` (17 classes)
|
||||
- Seg filter: 37,498 passed / 10,905 excluded (>=90% background+water)
|
||||
- Train: 15,693 pairs, Test: 18,015 pairs (cross-area split)
|
||||
|
||||
### V2 — UAV-GeoLoc + GeoRSCLIP (legacy)
|
||||
|
||||
Single backbone with template captions.
|
||||
|
||||
```
|
||||
Query: drone_img + caption -> GeoRSCLIP -> GatedFusion -> query
|
||||
Gallery: sat_img -> GeoRSCLIP -> gallery
|
||||
```
|
||||
|
||||
**Dataset:** UAV-GeoLoc Terrain split (206K train queries)
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
caption_test/
|
||||
├── conf/
|
||||
│ ├── balanced.gin # Primary: gate init 0.7 (30% text)
|
||||
│ ├── baseline_no_text.gin # Reference: gate = 1.0 (no text)
|
||||
│ └── text_heavy.gin # Stress: gate init 0.3 (70% text)
|
||||
caption-test/
|
||||
├── conf/ # Gin configs (v2)
|
||||
│ ├── balanced.gin
|
||||
│ ├── baseline_no_text.gin
|
||||
│ └── text_heavy.gin
|
||||
├── nn_models/ # Pre-trained checkpoints (v3)
|
||||
│ ├── DINO_WEB/ # DINOv3 ViT-L/16 LVD-1689M
|
||||
│ ├── DINO_SAT/ # DINOv3 ViT-L/16 SAT-493M
|
||||
│ └── LRSCLIP/ # DGTRS-CLIP ViT-L-14
|
||||
├── scripts/
|
||||
│ ├── generate_captions.py # Offline caption generation
|
||||
│ └── compare_runs.py # Delta R@1 comparison report
|
||||
│ ├── filter_segmentation.py # Meta-file: exclude 90%+ background/water
|
||||
│ ├── compare_runs.py # Delta R@1 comparison report
|
||||
│ └── generate_captions.py # Offline caption generation
|
||||
├── src/
|
||||
│ ├── datasets/
|
||||
│ │ └── visloc_with_captions.py # UAV-GeoLoc loader + template captions
|
||||
│ │ ├── gtauav_dataset.py # GTA-UAV-LR loader + L1/L2/L3 captions (v3)
|
||||
│ │ └── visloc_with_captions.py # UAV-GeoLoc loader (v2)
|
||||
│ ├── models/
|
||||
│ │ └── dual_encoder.py # GeoRSCLIP + GatedFusion + projection heads
|
||||
│ │ ├── asymmetric_encoder.py # DINOv3 + LRSCLIP + GatedFusion (v3)
|
||||
│ │ └── dual_encoder.py # GeoRSCLIP + GatedFusion (v2)
|
||||
│ ├── losses/
|
||||
│ │ └── multi_infonce.py # InfoNCE with cosine temperature
|
||||
│ │ └── multi_infonce.py # InfoNCE with cosine temperature
|
||||
│ ├── training/
|
||||
│ │ └── train.py # Main training loop
|
||||
│ │ ├── train_gtauav.py # Training loop GTA-UAV (v3)
|
||||
│ │ └── train.py # Training loop UAV-GeoLoc (v2)
|
||||
│ └── eval/
|
||||
│ └── evaluate.py # R@K metrics, Delta R@1
|
||||
└── checkpoints/ # RS5M_ViT-B-32.pt (user-provided)
|
||||
│ └── evaluate.py # R@K metrics, Delta R@1
|
||||
└── checkpoints/ # GeoRSCLIP RS5M_ViT-B-32.pt (v2)
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
@@ -44,71 +79,54 @@ caption_test/
|
||||
```
|
||||
torch>=2.0
|
||||
open_clip_torch
|
||||
safetensors
|
||||
timm
|
||||
gin-config
|
||||
Pillow
|
||||
numpy
|
||||
```
|
||||
|
||||
GeoRSCLIP checkpoint: download `RS5M_ViT-B-32.pt` from
|
||||
`github.com/om-ai-lab/RS5M` and place under `checkpoints/`.
|
||||
## Workflow (V3 — GTA-UAV)
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Train baseline (no text)
|
||||
### 1. Filter segmentation (exclude uninformative images)
|
||||
|
||||
```bash
|
||||
python -m src.training.train --config conf/baseline_no_text.gin
|
||||
python -m scripts.filter_segmentation --output meta/seg_filter.json
|
||||
```
|
||||
|
||||
### 2. Train with captions
|
||||
### 2. Train baseline (no text)
|
||||
|
||||
```bash
|
||||
python -m src.training.train --config conf/balanced.gin
|
||||
python -m src.training.train_gtauav --baseline --filter-meta meta/seg_filter.json
|
||||
```
|
||||
|
||||
### 3. Compare and get verdict
|
||||
### 3. Train with captions (L1/L2/L3)
|
||||
|
||||
```bash
|
||||
python -m src.training.train_gtauav --filter-meta meta/seg_filter.json
|
||||
```
|
||||
|
||||
### 4. Compare and get verdict
|
||||
|
||||
```bash
|
||||
python -m scripts.compare_runs \
|
||||
--baseline_report out/caption_test/baseline_no_text/eval_report.json \
|
||||
--full_report out/caption_test/balanced/eval_report.json \
|
||||
--output out/caption_test/comparison.md
|
||||
--baseline_report out/gtauav/baseline/eval_report.json \
|
||||
--full_report out/gtauav/with_text/eval_report.json \
|
||||
--output out/gtauav/comparison.md
|
||||
```
|
||||
|
||||
## Decision rule
|
||||
|
||||
| Delta R@1 (query->gallery) | Verdict |
|
||||
| Delta R@1 (drone->satellite) | Verdict |
|
||||
|---|---|
|
||||
| >= +3% | PASS -- captions informative, proceed to production |
|
||||
| >= +3% | PASS -- captions informative, proceed to NADEZHDA teacher |
|
||||
| +1% to +3% | MARGINAL -- add VLM refinement, re-run |
|
||||
| 0 to +1% | WEAK -- redesign caption pipeline |
|
||||
| < 0 | HARMFUL -- critical bug |
|
||||
|
||||
## Expected runtime (RTX 4090, 24 GB)
|
||||
|
||||
| Phase | Time |
|
||||
|---|---|
|
||||
| Single training run (10 epochs, batch 128, 206K queries) | ~15-30 min |
|
||||
| Full test (baseline + balanced + text_heavy) | ~1-1.5 h |
|
||||
| Evaluation | ~2-5 min per run |
|
||||
|
||||
## Dataset
|
||||
|
||||
UAV-GeoLoc Terrain split (from `/mnt/data1tb/cvgl_datasets/UAV-GeoLoc/`):
|
||||
- Train: 206,108 queries, 94,709 DB crops (140 scenes)
|
||||
- Val: 62,368 queries, 26,597 DB crops (40 scenes)
|
||||
- Test: 33,472 queries, 11,684 DB crops (20 scenes)
|
||||
|
||||
Template captions generated automatically from path metadata:
|
||||
```
|
||||
"Aerial view at 100m facing northwest over volcanic terrain near KilaueaVolcano.
|
||||
Plan-view features: lava flows, crater edges, volcanic rock."
|
||||
```
|
||||
|
||||
## Code style
|
||||
|
||||
- `from __future__ import annotations` everywhere
|
||||
- Type hints on all signatures
|
||||
- Google-style docstrings
|
||||
- `@gin.configurable` on top-level classes
|
||||
- No emojis in code, English-only comments
|
||||
|
||||
Reference in New Issue
Block a user