Files

pikaliov 6ad9c4d149 Add GTA-UAV experiment: asymmetric DINOv3 + LRSCLIP text encoder

V3 architecture for CVGL caption validation on GTA-UAV-LR dataset:
- AsymmetricEncoder: DINOv3 ViT-L/16 (LVD drone + SAT satellite, frozen)
  + LRSCLIP/DGTRS-CLIP ViT-L-14 text encoder (248 tok, partial unfreeze)
- L1/L2/L3 hierarchical captions from VLM-generated descriptions
- TextFusionMLP (concat 3x768 -> MLP -> 512) + GatedFusion
- Segmentation filter: exclude images with >=90% background+water
- 10.9M trainable / 733M total params, 256x256 input
- coloredlogs + tqdm + emoji for training UX
- Baseline mode (--baseline): image-only, no text encoder loaded

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-21 17:54:27 +03:00

4.2 KiB

Raw Blame History

Caption Quality Test for Cross-View Geo-Localization

Validate whether generated text captions improve retrieval R@1 in cross-view geo-localization (drone-to-satellite).

Experiments

V3 — GTA-UAV + DINOv3 + LRSCLIP (active)

Asymmetric architecture with domain-specific image encoders and hierarchical text.

Query:   drone_img (DINOv3 LVD) + L1/L2/L3 captions (LRSCLIP) -> GatedFusion -> query
Gallery: sat_img (DINOv3 SAT) -> gallery
Loss:    InfoNCE(query, gallery)

Models:

Drone: DINOv3 ViT-L/16 (LVD-1689M, web pretrained) — 1024-dim, 303M params, frozen
Satellite: DINOv3 ViT-L/16 (SAT-493M, satellite pretrained) — 1024-dim, 303M params, frozen
Text: DGTRS-CLIP ViT-L-14 (LRSCLIP, 248 tokens) — 768-dim, partial unfreeze
Total: 733M params, 10.9M trainable (1.49%)

Input: 256x256, ImageNet normalization

Dataset: GTA-UAV-LR (33K drone + 14K satellite, GTA V synthetic)

RGB: /home/servml/Документы/datasets/GTA-UAV-LR/
Captions: /home/servml/Документы/datasets/GTA-UAV-LR-captions/ (40K JSON, 3-paragraph VLM)
Segmentation: /home/servml/Документы/datasets/GTA-UAV-LR-aug/ (17 classes)
Seg filter: 37,498 passed / 10,905 excluded (>=90% background+water)
Train: 15,693 pairs, Test: 18,015 pairs (cross-area split)

V2 — UAV-GeoLoc + GeoRSCLIP (legacy)

Single backbone with template captions.

Query:   drone_img + caption -> GeoRSCLIP -> GatedFusion -> query
Gallery: sat_img -> GeoRSCLIP -> gallery

Dataset: UAV-GeoLoc Terrain split (206K train queries)

Structure

caption-test/
├── conf/                         # Gin configs (v2)
│   ├── balanced.gin
│   ├── baseline_no_text.gin
│   └── text_heavy.gin
├── nn_models/                    # Pre-trained checkpoints (v3)
│   ├── DINO_WEB/                 # DINOv3 ViT-L/16 LVD-1689M
│   ├── DINO_SAT/                 # DINOv3 ViT-L/16 SAT-493M
│   └── LRSCLIP/                  # DGTRS-CLIP ViT-L-14
├── scripts/
│   ├── filter_segmentation.py    # Meta-file: exclude 90%+ background/water
│   ├── compare_runs.py           # Delta R@1 comparison report
│   └── generate_captions.py      # Offline caption generation
├── src/
│   ├── datasets/
│   │   ├── gtauav_dataset.py     # GTA-UAV-LR loader + L1/L2/L3 captions (v3)
│   │   └── visloc_with_captions.py  # UAV-GeoLoc loader (v2)
│   ├── models/
│   │   ├── asymmetric_encoder.py # DINOv3 + LRSCLIP + GatedFusion (v3)
│   │   └── dual_encoder.py       # GeoRSCLIP + GatedFusion (v2)
│   ├── losses/
│   │   └── multi_infonce.py      # InfoNCE with cosine temperature
│   ├── training/
│   │   ├── train_gtauav.py       # Training loop GTA-UAV (v3)
│   │   └── train.py              # Training loop UAV-GeoLoc (v2)
│   └── eval/
│       └── evaluate.py           # R@K metrics, Delta R@1
└── checkpoints/                  # GeoRSCLIP RS5M_ViT-B-32.pt (v2)

Prerequisites

torch>=2.0
open_clip_torch
safetensors
timm
gin-config
Pillow
numpy

Workflow (V3 — GTA-UAV)

1. Filter segmentation (exclude uninformative images)

python -m scripts.filter_segmentation --output meta/seg_filter.json

2. Train baseline (no text)

python -m src.training.train_gtauav --baseline --filter-meta meta/seg_filter.json

3. Train with captions (L1/L2/L3)

python -m src.training.train_gtauav --filter-meta meta/seg_filter.json

4. Compare and get verdict

python -m scripts.compare_runs \
    --baseline_report out/gtauav/baseline/eval_report.json \
    --full_report out/gtauav/with_text/eval_report.json \
    --output out/gtauav/comparison.md

Decision rule

Delta R@1 (drone->satellite)	Verdict
>= +3%	PASS -- captions informative, proceed to NADEZHDA teacher
+1% to +3%	MARGINAL -- add VLM refinement, re-run
0 to +1%	WEAK -- redesign caption pipeline
< 0	HARMFUL -- critical bug

Code style

from __future__ import annotations everywhere
Type hints on all signatures
Google-style docstrings
No emojis in code, English-only comments

4.2 KiB Raw Blame History