diff --git a/README.md b/README.md index 67a3116..b978638 100644 --- a/README.md +++ b/README.md @@ -11,42 +11,61 @@ Asymmetric dual-encoder with domain-specific frozen backbones and hierarchical text fusion for drone-to-satellite image retrieval. ``` -┌──────────────────────────── QUERY BRANCH ────────────────────────────┐ -│ │ -│ drone_img ──► DINOv3 ViT-L/16 LVD ──► CLS token │ -│ [B,3,256,256] (frozen, 303M) d_img [B,1024] │ -│ │ │ -│ L1 (overview) ──► DGTRS-CLIP ──► z₁ [B,768] ─┐ │ -│ L2 (full desc) ──► DGTRS-CLIP ──► z₂ [B,768] ─┼─ cat ──► [B,2304]│ -│ L3 (fingerprint) ──► DGTRS-CLIP ──► z₃ [B,768] ─┘ │ │ -│ (248 tokens, KPS pos. emb.) MLP(2304→1024→1024) │ -│ │ │ -│ d_txt [B,1024] │ -│ │ │ -│ q = σ(α_q)·d_img + (1−σ(α_q))·d_txt GatedFusion_q │ -│ │ │ -│ q̂ = q / ‖q‖₂ ──► query [B,1024] │ -└───────────────────────────────────────────────────────────────────────┘ +┌─────────────────────────────── QUERY BRANCH ─────────────────────────────────┐ +│ │ +│ drone_img ──► DINOv3 ViT-L/16 LVD (frozen, 303M) ──► CLS token │ +│ [B,3,256,256] + MONA adapters (7M trainable) d_img [B,1024] │ +│ (2 per block × 24 blocks, bn=64) │ │ +│ │ │ +│ L1 (overview) ──► DGTRS-CLIP ViT-L-14 ──► z₁ [B,768] ─┐ │ +│ L2 (full desc) ──► (frozen, 124M) ──► z₂ [B,768] ─┼─ cat ──[B,2304] │ +│ L3 (fingerprint) ──► + LoRA r=4 (147K) ──► z₃ [B,768] ─┘ │ │ +│ (248 tok, KPS pos.) TextFusionMLP (shared, 3.4M) │ +│ Linear→GELU→Linear │ +│ │ │ +│ d_txt [B,1024] │ +│ │ │ +│ q = σ(α_q)·d_img + (1−σ(α_q))·d_txt GatedFusion_q │ +│ │ │ +│ q̂ = q / ‖q‖₂ ──► query [B,1024] │ +└───────────────────────────────────────────────────────────────────────────────┘ -┌──────────────────────────── GALLERY BRANCH ──────────────────────────┐ -│ │ -│ sat_img ──► DINOv3 ViT-L/16 SAT ──► CLS token │ -│ [B,3,256,256] (frozen, 303M) s_img [B,1024] │ -│ │ │ -│ sat_L1 ──► DGTRS-CLIP ──► z₁ [768] ─┐ │ -│ sat_L2 ──► DGTRS-CLIP ──► z₂ [768] ─┼─ cat ──► MLP ──► s_txt [1024]│ -│ sat_L3 ──► DGTRS-CLIP ──► z₃ [768] ─┘ (shared MLP) │ -│ │ │ -│ g = σ(α_g)·s_img + (1−σ(α_g))·s_txt GatedFusion_g │ -│ │ │ -│ ĝ = g / ‖g‖₂ ──► gallery [B,1024]│ -└───────────────────────────────────────────────────────────────────────┘ +┌────────────────────────────── GALLERY BRANCH ────────────────────────────────┐ +│ │ +│ sat_img ──► DINOv3 ViT-L/16 SAT (frozen, 303M) ──► CLS token │ +│ [B,3,256,256] + MONA adapters (7M trainable) s_img [B,1024] │ +│ (2 per block × 24 blocks, bn=64) │ │ +│ │ │ +│ sat_L1 ──► DGTRS-CLIP ──► z₁ [768] ─┐ │ +│ sat_L2 ──► (shared encoder)──► z₂ [768] ─┼─ cat ──► TextFusionMLP (shared) │ +│ sat_L3 ──► + LoRA ──► z₃ [768] ─┘ │ │ +│ s_txt [B,1024] │ +│ │ │ +│ g = σ(α_g)·s_img + (1−σ(α_g))·s_txt GatedFusion_g │ +│ │ │ +│ ĝ = g / ‖g‖₂ ──► gallery [B,1024] │ +└───────────────────────────────────────────────────────────────────────────────┘ -Retrieval space: 1024-dim (DINOv3 native, no projection layers) -TextFusionMLP shared between query and gallery branches -For sat images without captions: s_txt=None → g = s_img (gate passthrough) +┌────────────────────────────── RETRIEVAL ──────────────────────────────────────┐ +│ │ +│ similarity = q̂ · ĝᵀ / τ (τ learnable, init=0.07, clamp [0.01, 0.5]) │ +│ loss = 0.6·CE(q→g) + 0.4·CE(g→q) (label_smoothing=0.1) │ +│ │ +│ Retrieval space: 1024-dim (DINOv3 native, no projection layers) │ +│ TextFusionMLP shared between query and gallery branches │ +│ For sat images without captions: s_txt=None → g = s_img (gate passthrough) │ +│ BASELINE: σ(α) = 1.0 for both branches (text disabled, DGTRS not loaded) │ +└───────────────────────────────────────────────────────────────────────────────┘ -BASELINE: σ(α) = 1.0 for both branches (text disabled, DGTRS not loaded) +┌──────────────────────────── DIAGNOSTICS PIPELINE ────────────────────────────┐ +│ │ +│ Per-batch ──► CSV (train_batches.csv) ──► TensorBoard / W&B scalars │ +│ Per-epoch ──► CSV (train.csv, val.csv) ──► Seaborn plots (PNG) │ +│ Every N epochs ──► Grad-CAM heatmaps (drone + satellite DINOv3 last block) │ +│ Epoch 0 (opt) ──► PyTorch Profiler (Chrome trace + CUDA timeline) │ +│ Per-50-batch ──► Gradient norms per group (MONA, LoRA, MLP, gates, τ) │ +│ On init ──► torchinfo model summary (model_summary.txt) │ +└───────────────────────────────────────────────────────────────────────────────┘ ``` ### Text hierarchy (L1 / L2 / L3) @@ -222,6 +241,7 @@ caption-test/ │ ├── gtauav_balanced.gin # GTA-UAV with text (10 epochs, v3) │ ├── gtauav_baseline.gin # GTA-UAV baseline, no text (v3) │ ├── gtauav_text_heavy.gin # GTA-UAV text-heavy gate=0.3 (v3) +│ ├── gtauav_image_heavy.gin # GTA-UAV image-heavy gate=0.9 (v3) │ ├── balanced.gin # UAV-GeoLoc with text (v2) │ ├── baseline_no_text.gin # UAV-GeoLoc baseline (v2) │ └── text_heavy.gin # UAV-GeoLoc text-heavy (v2) @@ -368,7 +388,8 @@ tensorboard --logdir out/gtauav/with_text/tb_logs | **PyTorch Profiler** | `--profile` | `{out}/profiler/` | Chrome trace, CUDA timeline, memory | | **torchinfo** | auto | `{out}/model_summary.txt` | Layer-by-layer parameter table | | **Gradient norms** | `--log-grad-norms` (default on) | TB/W&B | Per-group: MONA, LoRA, MLP, gates, tau | -| **CSV + plots** | auto | `{out}/logs/` | train.csv, val.csv, PNG plots every epoch | +| **CSV (per-batch)** | auto | `{out}/logs/train_batches.csv` | Loss, tau, gates, lr for every batch | +| **CSV (per-epoch)** | auto | `{out}/logs/train.csv, val.csv` | Epoch averages + seaborn PNG plots | ## Decision rule