Update README architecture diagram with adapters, loss, and diagnostics pipeline
- Show MONA adapters (7M) and LoRA (147K) in branch diagrams - Add retrieval/loss block with temperature and CE weights - Add diagnostics pipeline block (per-batch CSV, Grad-CAM, profiler, grad norms) - Add gtauav_image_heavy.gin to structure - Split CSV row in diagnostics table into per-batch and per-epoch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
89
README.md
89
README.md
@@ -11,42 +11,61 @@ Asymmetric dual-encoder with domain-specific frozen backbones and hierarchical
|
||||
text fusion for drone-to-satellite image retrieval.
|
||||
|
||||
```
|
||||
┌──────────────────────────── QUERY BRANCH ────────────────────────────┐
|
||||
│ │
|
||||
│ drone_img ──► DINOv3 ViT-L/16 LVD ──► CLS token │
|
||||
│ [B,3,256,256] (frozen, 303M) d_img [B,1024] │
|
||||
│ │ │
|
||||
│ L1 (overview) ──► DGTRS-CLIP ──► z₁ [B,768] ─┐ │
|
||||
│ L2 (full desc) ──► DGTRS-CLIP ──► z₂ [B,768] ─┼─ cat ──► [B,2304]│
|
||||
│ L3 (fingerprint) ──► DGTRS-CLIP ──► z₃ [B,768] ─┘ │ │
|
||||
│ (248 tokens, KPS pos. emb.) MLP(2304→1024→1024) │
|
||||
│ │ │
|
||||
│ d_txt [B,1024] │
|
||||
│ │ │
|
||||
│ q = σ(α_q)·d_img + (1−σ(α_q))·d_txt GatedFusion_q │
|
||||
│ │ │
|
||||
│ q̂ = q / ‖q‖₂ ──► query [B,1024] │
|
||||
└───────────────────────────────────────────────────────────────────────┘
|
||||
┌─────────────────────────────── QUERY BRANCH ─────────────────────────────────┐
|
||||
│ │
|
||||
│ drone_img ──► DINOv3 ViT-L/16 LVD (frozen, 303M) ──► CLS token │
|
||||
│ [B,3,256,256] + MONA adapters (7M trainable) d_img [B,1024] │
|
||||
│ (2 per block × 24 blocks, bn=64) │ │
|
||||
│ │ │
|
||||
│ L1 (overview) ──► DGTRS-CLIP ViT-L-14 ──► z₁ [B,768] ─┐ │
|
||||
│ L2 (full desc) ──► (frozen, 124M) ──► z₂ [B,768] ─┼─ cat ──[B,2304] │
|
||||
│ L3 (fingerprint) ──► + LoRA r=4 (147K) ──► z₃ [B,768] ─┘ │ │
|
||||
│ (248 tok, KPS pos.) TextFusionMLP (shared, 3.4M) │
|
||||
│ Linear→GELU→Linear │
|
||||
│ │ │
|
||||
│ d_txt [B,1024] │
|
||||
│ │ │
|
||||
│ q = σ(α_q)·d_img + (1−σ(α_q))·d_txt GatedFusion_q │
|
||||
│ │ │
|
||||
│ q̂ = q / ‖q‖₂ ──► query [B,1024] │
|
||||
└───────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌──────────────────────────── GALLERY BRANCH ──────────────────────────┐
|
||||
│ │
|
||||
│ sat_img ──► DINOv3 ViT-L/16 SAT ──► CLS token │
|
||||
│ [B,3,256,256] (frozen, 303M) s_img [B,1024] │
|
||||
│ │ │
|
||||
│ sat_L1 ──► DGTRS-CLIP ──► z₁ [768] ─┐ │
|
||||
│ sat_L2 ──► DGTRS-CLIP ──► z₂ [768] ─┼─ cat ──► MLP ──► s_txt [1024]│
|
||||
│ sat_L3 ──► DGTRS-CLIP ──► z₃ [768] ─┘ (shared MLP) │
|
||||
│ │ │
|
||||
│ g = σ(α_g)·s_img + (1−σ(α_g))·s_txt GatedFusion_g │
|
||||
│ │ │
|
||||
│ ĝ = g / ‖g‖₂ ──► gallery [B,1024]│
|
||||
└───────────────────────────────────────────────────────────────────────┘
|
||||
┌────────────────────────────── GALLERY BRANCH ────────────────────────────────┐
|
||||
│ │
|
||||
│ sat_img ──► DINOv3 ViT-L/16 SAT (frozen, 303M) ──► CLS token │
|
||||
│ [B,3,256,256] + MONA adapters (7M trainable) s_img [B,1024] │
|
||||
│ (2 per block × 24 blocks, bn=64) │ │
|
||||
│ │ │
|
||||
│ sat_L1 ──► DGTRS-CLIP ──► z₁ [768] ─┐ │
|
||||
│ sat_L2 ──► (shared encoder)──► z₂ [768] ─┼─ cat ──► TextFusionMLP (shared) │
|
||||
│ sat_L3 ──► + LoRA ──► z₃ [768] ─┘ │ │
|
||||
│ s_txt [B,1024] │
|
||||
│ │ │
|
||||
│ g = σ(α_g)·s_img + (1−σ(α_g))·s_txt GatedFusion_g │
|
||||
│ │ │
|
||||
│ ĝ = g / ‖g‖₂ ──► gallery [B,1024] │
|
||||
└───────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Retrieval space: 1024-dim (DINOv3 native, no projection layers)
|
||||
TextFusionMLP shared between query and gallery branches
|
||||
For sat images without captions: s_txt=None → g = s_img (gate passthrough)
|
||||
┌────────────────────────────── RETRIEVAL ──────────────────────────────────────┐
|
||||
│ │
|
||||
│ similarity = q̂ · ĝᵀ / τ (τ learnable, init=0.07, clamp [0.01, 0.5]) │
|
||||
│ loss = 0.6·CE(q→g) + 0.4·CE(g→q) (label_smoothing=0.1) │
|
||||
│ │
|
||||
│ Retrieval space: 1024-dim (DINOv3 native, no projection layers) │
|
||||
│ TextFusionMLP shared between query and gallery branches │
|
||||
│ For sat images without captions: s_txt=None → g = s_img (gate passthrough) │
|
||||
│ BASELINE: σ(α) = 1.0 for both branches (text disabled, DGTRS not loaded) │
|
||||
└───────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
BASELINE: σ(α) = 1.0 for both branches (text disabled, DGTRS not loaded)
|
||||
┌──────────────────────────── DIAGNOSTICS PIPELINE ────────────────────────────┐
|
||||
│ │
|
||||
│ Per-batch ──► CSV (train_batches.csv) ──► TensorBoard / W&B scalars │
|
||||
│ Per-epoch ──► CSV (train.csv, val.csv) ──► Seaborn plots (PNG) │
|
||||
│ Every N epochs ──► Grad-CAM heatmaps (drone + satellite DINOv3 last block) │
|
||||
│ Epoch 0 (opt) ──► PyTorch Profiler (Chrome trace + CUDA timeline) │
|
||||
│ Per-50-batch ──► Gradient norms per group (MONA, LoRA, MLP, gates, τ) │
|
||||
│ On init ──► torchinfo model summary (model_summary.txt) │
|
||||
└───────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Text hierarchy (L1 / L2 / L3)
|
||||
@@ -222,6 +241,7 @@ caption-test/
|
||||
│ ├── gtauav_balanced.gin # GTA-UAV with text (10 epochs, v3)
|
||||
│ ├── gtauav_baseline.gin # GTA-UAV baseline, no text (v3)
|
||||
│ ├── gtauav_text_heavy.gin # GTA-UAV text-heavy gate=0.3 (v3)
|
||||
│ ├── gtauav_image_heavy.gin # GTA-UAV image-heavy gate=0.9 (v3)
|
||||
│ ├── balanced.gin # UAV-GeoLoc with text (v2)
|
||||
│ ├── baseline_no_text.gin # UAV-GeoLoc baseline (v2)
|
||||
│ └── text_heavy.gin # UAV-GeoLoc text-heavy (v2)
|
||||
@@ -368,7 +388,8 @@ tensorboard --logdir out/gtauav/with_text/tb_logs
|
||||
| **PyTorch Profiler** | `--profile` | `{out}/profiler/` | Chrome trace, CUDA timeline, memory |
|
||||
| **torchinfo** | auto | `{out}/model_summary.txt` | Layer-by-layer parameter table |
|
||||
| **Gradient norms** | `--log-grad-norms` (default on) | TB/W&B | Per-group: MONA, LoRA, MLP, gates, tau |
|
||||
| **CSV + plots** | auto | `{out}/logs/` | train.csv, val.csv, PNG plots every epoch |
|
||||
| **CSV (per-batch)** | auto | `{out}/logs/train_batches.csv` | Loss, tau, gates, lr for every batch |
|
||||
| **CSV (per-epoch)** | auto | `{out}/logs/train.csv, val.csv` | Epoch averages + seaborn PNG plots |
|
||||
|
||||
## Decision rule
|
||||
|
||||
|
||||
Reference in New Issue
Block a user