Add GTA-UAV experiment: asymmetric DINOv3 + LRSCLIP text encoder

V3 architecture for CVGL caption validation on GTA-UAV-LR dataset: - AsymmetricEncoder: DINOv3 ViT-L/16 (LVD drone + SAT satellite, frozen) + LRSCLIP/DGTRS-CLIP ViT-L-14 text encoder (248 tok, partial unfreeze) - L1/L2/L3 hierarchical captions from VLM-generated descriptions - TextFusionMLP (concat 3x768 -> MLP -> 512) + GatedFusion - Segmentation filter: exclude images with >=90% background+water - 10.9M trainable / 733M total params, 256x256 input - coloredlogs + tqdm + emoji for training UX - Baseline mode (--baseline): image-only, no text encoder loaded Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-21 17:54:27 +03:00
parent 5da791801c
commit 6ad9c4d149
10 changed files with 50043 additions and 101 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1,29 +1,49 @@
 # Caption Quality Test for Cross-View Geo-Localization

-## Архитектура системы (v2, 2026-04-17)
+## Архитектура системы (v3, 2026-04-21) — GTA-UAV эксперимент

 ```
-QUERY BRANCH (drone + caption):
-  drone_img --> GeoRSCLIP ViT-B/32 (frozen) --> drone_feat [B,512]
-  caption   --> GeoRSCLIP Text (partial unfreeze) --> text_feat [B,512]
-                     |
-              GatedFusion: q = sigma(alpha)*drone + (1-sigma(alpha))*text
-                     |
-              proj_query (Linear 512->512) --> L2-norm --> query [B,512]
+QUERY BRANCH (drone + L1/L2/L3 captions):
+  drone_img --> DINOv3 ViT-L/16 LVD-1689M (frozen) --> CLS [B,1024]
+                        |
+                 proj_drone (1024->512)
+                        |
+  L1 (overview)    --> LRSCLIP (248 tok) --> z_L1 [768] --\
+  L2 (full desc)   --> LRSCLIP (248 tok) --> z_L2 [768] ---+-- concat [B,2304]
+  L3 (fingerprint) --> LRSCLIP (248 tok) --> z_L3 [768] --/        |
+                                                          MLP(2304->768->512)
+                                                                   |
+                              GatedFusion(drone_512, text_512) --> L2-norm --> query [B,512]

 GALLERY BRANCH (satellite only):
-  sat_img --> GeoRSCLIP ViT-B/32 (frozen) --> sat_feat [B,512]
-                     |
-              proj_gallery (Linear 512->512) --> L2-norm --> gallery [B,512]
+  sat_img --> DINOv3 ViT-L/16 SAT-493M (frozen) --> CLS [B,1024]
+                      |
+               proj_sat (1024->512) --> L2-norm --> gallery [B,512]

 LOSS: InfoNCE(query, gallery) — symmetric, asymmetric weights (0.6 q->g, 0.4 g->q)
 BASELINE: gate = 1.0 (text ignored)
 ```

-### Trainable parameters: ~1.2M из ~151M (proj_query + proj_gallery + fusion alpha + text last_block)
+### Trainable parameters: 10.9M из 733M (1.49%)
+- proj_drone: 1024x512 = ~524K
+- proj_sat: 1024x512 = ~524K
+- TextFusionMLP: 2304->768->512 = ~2.2M
+- gate alpha: 1 scalar
+- LRSCLIP partial unfreeze (last block + ln_final + text_projection): ~7.6M
+- Backbones DINOv3 x2 (303M each): frozen
+
+### Image input: 256x256
+DINOv3 ViT-L/16 с patch_size=16 → 16x16=256 patches на 256x256.
+Resize(256) + CenterCrop(256) + ImageNet normalization.
+
+### Предыдущая архитектура (v2) — UAV-GeoLoc эксперимент
+
+Использовала GeoRSCLIP ViT-B/32 (512-dim) для обеих веток + template captions.
+Код в `src/models/dual_encoder.py`, `src/datasets/visloc_with_captions.py`.

 ## Ключевые файлы

+### V2 (UAV-GeoLoc, GeoRSCLIP)
 | Файл | Назначение |
 |------|-----------|
 | `src/models/dual_encoder.py` | GeoRSCLIP + GatedFusion + projection heads |
@@ -34,13 +54,42 @@ BASELINE: gate = 1.0 (text ignored)
 | `scripts/compare_runs.py` | Markdown/JSON сравнение baseline vs caption runs |
 | `scripts/generate_captions.py` | Offline caption generation (template/VLM/hybrid) |

-## Backbone: GeoRSCLIP ViT-B/32
+### V3 (GTA-UAV, DINOv3 + LRSCLIP) — DONE
+| Файл | Назначение |
+|------|-----------|
+| `src/models/asymmetric_encoder.py` | DINOv3ViT + LRSCLIPTextEncoder + TextFusionMLP + AsymmetricEncoder + GatedFusion |
+| `src/datasets/gtauav_dataset.py` | GTA-UAV-LR loader + L1/L2/L3 caption parsing из VLM JSON |
+| `src/training/train_gtauav.py` | Training loop с eval, AMP, CLI args (--baseline, --filter-meta) |
+| `scripts/filter_segmentation.py` | Scan segm masks, output meta JSON (exclude >=90% bg+water) |

- **Checkpoint:** `checkpoints/RS5M_ViT-B-32.pt` (скачать с github.com/om-ai-lab/RS5M)
+## Backbones (v3)
+
+### DINOv3 ViT-L/16 — Drone (web pretrained)
+- **Checkpoint:** `nn_models/DINO_WEB/dinov3-vitl16-pretrain-lvd1689m.pth`
+- **Arch:** ViT-L/16, 24 layers, 16 heads, hidden=1024, MLP=4096, 303M params
+- **Input:** 256x256, ImageNet normalization, patch=16 → 256 patches
+- **Register tokens:** 4, RoPE theta=100.0
+- **Status:** frozen
+
+### DINOv3 ViT-L/16 — Satellite (sat pretrained)
+- **Checkpoint:** `nn_models/DINO_SAT/model.safetensors`
+- **HuggingFace:** `facebook/dinov3-vitl16-pretrain-sat493m`
+- **Arch:** идентична DINO_WEB (ViT-L/16, hidden=1024, 303M params)
+- **Input:** 256x256
+- **Config:** `nn_models/DINO_SAT/config.json` — BROKEN (auth error), используем конфиг от DINO_WEB
+- **Status:** frozen
+
+### DGTRS-CLIP ViT-L-14 (LRSCLIP) — Text encoder
+- **Checkpoint:** `nn_models/LRSCLIP/DGTRS-CLIP-ViT-L-14.pt`
+- **Text dim:** 768, max tokens: 248 (KPS stretched from CLIP's 77)
+- **Содержит:** полную CLIP модель (visual + text), используем только text encoder
+- **GitHub:** `github.com/MitsuiChen14/DGTRS`
+- **Status:** partial unfreeze (last block + text_projection)
+
+### GeoRSCLIP ViT-B/32 (v2, legacy)
+- **Checkpoint:** `checkpoints/RS5M_ViT-B-32.pt`
 - **Image encoder:** ViT-B/32, 224x224, 512-dim, ~86M params — frozen
- **Text encoder:** CLIP text transformer, 77 tokens, 512-dim — partial unfreeze (last_block + text_projection)
- **Throughput:** ~4000 img/s на RTX 4090 (AMP, batch 128)
- **Выбран вместо SigLIP 2** (ViT-SO400M, 384px, ~400M): в 7-10x быстрее, domain-specific (обучен на 5M RS-изображений), больше batch = больше негативов в InfoNCE
+- **Text encoder:** 77 tokens, 512-dim — partial unfreeze

 ## GatedFusion

@@ -50,49 +99,91 @@ BASELINE: gate = 1.0 (text ignored)
 - `baseline_mode = True` → gate = 1.0, text полностью игнорируется
 - Gate value логируется каждую эпоху для интерпретации вклада текста

-## Датасет: UAV-GeoLoc
+## Text Hierarchy (L1/L2/L3)
+
+Три уровня описаний из VLM-generated captions:
+
+| Уровень | Контент | Длина | Источник |
+|---|---|---|---|
+| L1 overview | Краткое описание сцены | <=30 tok | Конденсация P1 |
+| L2 full description | Детальное описание через Qwen3-VL | <=200 tok | Полный P1+P2 |
+| L3 fingerprint | Ключевые landmark'ы | <=30 tok | Конденсация P3 |
+
+Все три уровня кодируются одним LRSCLIP (248 tok max).
+Альтернатива (Stage 2): RemoteCLIP для L1/L3 + LRSCLIP для L2.
+
+## Датасет: GTA-UAV-LR
+
+- **RGB:** `/home/servml/Документы/datasets/GTA-UAV-LR/`
+  - Drone: 33,763 PNG (512x384), altitudes 100-600m
+  - Satellite: 14,640 PNG (256x256 RGBA)
+  - Pairs: `cross-area-drone2sate-{train,test}.json` (primary split)
+  - Metadata: `*_drone_meta.csv` (height, yaw, roll, pitch)
+  - Origin: GTA V simulation (Los Santos)
+
+- **Captions:** `/home/servml/Документы/datasets/GTA-UAV-LR-captions/`
+  - Drone: 33,411 JSON (32,635 multi-paragraph P1/P2/P3 + 776 short water-only)
+  - Satellite: 6,546 JSON (все multi-paragraph)
+  - Формат: 3 абзаца (P1 Inventory + P2 Spatial + P3 Fingerprint)
+  - Token counts: ~430 output tokens per caption
+
+- **Segmentation:** `/home/servml/Документы/datasets/GTA-UAV-LR-aug/`
+  - 48,403 images, 17 классов (background, building, road, vegetation, water, ...)
+  - Modalities: segm/, depth/, edge/, chm/, safetensors/
+  - Query: 512x512, DB: 256x256
+
+### Фильтрация сегментации
+Meta-файл `meta/seg_filter.json`: исключение изображений с >=90% background(class 0) + water(class 4).
+- **Total:** 48,403 → **Passed:** 37,498 (77.5%) / **Excluded:** 10,905 (22.5%)
+- Drone: 31,188 passed / 2,575 excluded
+- Satellite: 6,310 passed / 8,330 excluded (преимущественно open water tiles)
+
+## Датасет: UAV-GeoLoc (v2, legacy)

 - **Путь:** `/mnt/data1tb/cvgl_datasets/UAV-GeoLoc/`
 - **Train:** 206,108 queries, 94,709 DB crops (140 scenes, Terrain split)
 - **Val:** 62,368 queries, 26,597 DB crops (40 scenes)
 - **Test:** 33,472 queries, 11,684 DB crops (20 scenes)
- **Структура:** `Terrain/{type}/{scene}/query/height{N}_rot{M}/footage/{file}.jpeg`
 - **Index:** `Index/train_query.txt` — `query_path 0 pos_crop1 pos_crop2 ...`

-### Template captions (из path metadata)
+## Конфигурации

-Формат: `"Aerial view at {height}m facing {heading} over {terrain} terrain near {scene}. Plan-view features: {features}."`
-
-Пример: `"Aerial view at 100m facing northwest over volcanic terrain near KilaueaVolcano. Plan-view features: lava flows, crater edges, volcanic rock."`
-
-Metadata извлекается из пути:
- `Terrain/Volcano/KilaueaVolcano/query/height100_rot315/...` → terrain=Volcano, scene=KilaueaVolcano, height=100, heading=northwest
- 27 terrain типов с predefined features (Volcano, Mountain, Hill, Desert, Plain, ...)
- Country subset: features = "buildings, roads, urban blocks, rooftops, intersections"
-
-## Конфигурации (gin)
+### V3 (GTA-UAV)
+Параметры: 10 epochs, batch 64, lr=1e-4, AMP, cosine LR, image 256x256.
+Train: 15,693 pairs (cross-area), Test: 18,015 pairs.
+Eval на cross-area split (primary).

+### V2 (UAV-GeoLoc, gin)
 | Конфиг | Gate init | Описание |
 |--------|-----------|----------|
 | `conf/balanced.gin` | 0.7 (30% text) | **Primary test** |
 | `conf/baseline_no_text.gin` | 1.0 (no text) | Reference baseline |
 | `conf/text_heavy.gin` | 0.3 (70% text) | Stress test |

-Общие параметры: 10 epochs, batch 128, lr=1e-4, AMP, cosine LR schedule, eval every 2 epochs.
-
 ## Запуск

+### V3 (GTA-UAV)
 ```bash
-# 1. Baseline (no text)
-python -m src.training.train --config conf/baseline_no_text.gin
+# 1. Filter segmentation (exclude 90%+ background/water)
+python -m scripts.filter_segmentation --output meta/seg_filter.json

-# 2. With captions (primary test)
-python -m src.training.train --config conf/balanced.gin
+# 2. Baseline (no text)
+python -m src.training.train_gtauav --baseline --filter-meta meta/seg_filter.json

-# 3. Text-heavy (stress test)
-python -m src.training.train --config conf/text_heavy.gin
+# 3. With captions (L1/L2/L3)
+python -m src.training.train_gtauav --filter-meta meta/seg_filter.json

 # 4. Compare
+python -m scripts.compare_runs \
+    --baseline_report out/gtauav/baseline/eval_report.json \
+    --full_report out/gtauav/with_text/eval_report.json \
+    --output out/gtauav/comparison.md
+```
+
+### V2 (UAV-GeoLoc)
+```bash
+python -m src.training.train --config conf/baseline_no_text.gin
+python -m src.training.train --config conf/balanced.gin
 python -m scripts.compare_runs \
    --baseline_report out/caption_test/baseline_no_text/eval_report.json \
    --full_report out/caption_test/balanced/eval_report.json \
@@ -101,7 +192,7 @@ python -m scripts.compare_runs \

 ## Метрики и Decision rule

-**Primary metric:** Delta R@1 (query -> gallery)
+**Primary metric:** Delta R@1 (drone -> satellite)

 | Delta R@1 | Verdict |
 |-----------|---------|
@@ -110,11 +201,21 @@ python -m scripts.compare_runs \
 | 0 to +1% | WEAK — redesign caption pipeline |
 | < 0 | HARMFUL — critical bug |

+**Eval metrics:** R@1, R@5, R@10 для drone->satellite и satellite->drone
+**Splits (GTA-UAV):** cross-area (primary, harder) и same-area (sanity check)
 **Logged per epoch:** loss, temperature (tau), gate value (sigma(alpha)), lr
-**Eval metrics:** R@1, R@5, R@10 для query->gallery и gallery->query

 ## Бюджет времени (RTX 4090, 24 GB)

+### V3 (GTA-UAV, DINOv3 ViT-L/16, 256x256)
+| Фаза | Оценка |
+|------|--------|
+| VRAM: 2x DINOv3-L + LRSCLIP + batch 64 | ~18-22 GB |
+| GPU mem (smoke test, batch 4) | 3.1 GB |
+| Batch size | 64 (default) |
+| Total params | 733M (10.9M trainable, 1.49%) |
+
+### V2 (UAV-GeoLoc, GeoRSCLIP)
 | Фаза | Время |
 |------|-------|
 | Один training run (10 epochs, 206K queries, batch 128) | ~15-30 мин |
@@ -138,6 +239,42 @@ python -m scripts.compare_runs \

 ---

+## Downstream: NADEZHDA Teacher (DINOv3 + Multi-FiLM)
+
+caption-test — первый этап валидации. При Δ R@1 >= +3% переход к полному teacher'у:
+
+```
+Этап 1 (caption-test): GeoRSCLIP + GatedFusion(text) → валидация текста
+Этап 2 (teacher):      DINOv3-L + Multi-FiLM(depth, seg, CHM, normals, text)
+Этап 3 (distillation): teacher ~300M → student ~5M → Jetson Orin NX
+```
+
+### Auxiliary modalities (предвычисляются из 512x512 офлайн)
+
+| Модальность | Модель | Формат для teacher | Каналы |
+|---|---|---|---|
+| Depth | DepthAnything V2 | continuous, log(1+d) | 1 |
+| Normals | Sobel от depth | continuous | 3 |
+| Segmentation | SegFormer-B5 | binary per-class masks (top-K) | 16-17 |
+| Canopy Height | Meta HRCH | binary bins (1-5m, 5-15m, >15m) + occupancy | 4-5 |
+| Text | Qwen3-VL-8B / MobileCLIP2 | embedding | - |
+
+### Асимметрия sat/drone
+- CHM: только satellite (модель обучена на nadir, на oblique drone не работает)
+- Satellite: ~27 aux каналов, Drone: ~21 aux каналов
+
+### Fusion: Multi-FiLM
+```
+aux_features → FiLM(γ, β) → γ * DINOv3_tokens + β
+```
+Binary masks — natural FiLM gates. Modality dropout: text 0.3, CHM 0.5, seg 0.15, depth 0.1.
+
+### Планируемый эксперимент H5
+Сравнение T_bin (binary masks) vs T_pyr (native feature pyramids) vs T_hybrid.
+Прогноз: binary masks лучше на cross-domain из-за робастности к aux-model artifacts.
+
+---
+
 ## Датасеты (справочник)

 ### UAV-VisLoc
@@ -197,7 +334,7 @@ python scripts/prepare_dataset.py \
 - Размер на диске: 25 GB
 - Median distance drone->crop: 25.9m, P99: 45.7m
 - Память при генерации: до ~8.7 GB RAM (маршрут 09 stitched 44800x33280)
- Разрешение 512 для downstream задач (сегментация, depth, normals); resize до 224/256 в dataloader
+- Разрешение 512 для downstream задач (сегментация, depth, normals); resize до 256x256 в dataloader

 ### Ревью и исправления (2026-04-17)
 1. `train_db.txt`/`test_db.txt` содержали только matched кропы -> теперь все ~74K (полная gallery)