bogdanpavl
ec7586b68d
П. 1.1. Конспект по пункту П. 1.2. Разбор personal_package, заметки, Evidence matrix (≥8 источников)
9.3 KiB
9.3 KiB
1. Резюме (Master)
Full multi-modal fusion в MERIDIAN — холистический pipeline для Triple-Teacher (DINOv3-L SAT/Web-LVD/7B, frozen, ~356M+) объединяющий 5 modalities × 2 views = 10 channels через 10 категорий fusion-парадигм. Master synthesis 4 sub-pair reviews + general fusion review.
DINOv3-L backbone (frozen): ~356M
Per-modality adapters (×5): ~500K (5 × 100K, light)
Multi-FiLM γ,β projections (×5): ~50K (5 × ~10K)
KARMMA tokens (5 modalities): ~100K (5 × 20K)
Θ-Average: 0 (parameter-free)
─────────────────────────────────────────
Total trainable params: ~650K (~0.2% backbone)
Master Outcomes
- Primary fusion mechanism: Multi-FiLM-Fusion (F43 SSF TPAMI 2024 anchor + F47 zero-init β) — <1% params overhead, INT8-compatible
- Secondary mechanism: ACF Condition Token (F39 CAFuser RA-L 2025) — -54% params vs separate backbones
- Parallel-arm research: Fusion-Mamba (F44 TMM 2025) — +5.9% mAP detection benchmark
- Critical ablations (F88 KARMMA):
- Two-token missing-modality strategy
- Θ-Average FB reduction (-81.45% memory)
- Modality dropout 50% canonical
- Missing modality handling convergent evidence (4 sources): F45 + F88 + F89 + F90 → dropout p=0.5 canonical
- Production-ready INT8 stack confirmed (AUDIT_N6 v2: ~0-2 ms fusion overhead)
Per-Pair Contribution Summary
| Pair | Modality | Expected R@1 gain | Status |
|---|---|---|---|
| A | Segmentation | +2-4% (L_seg aux) | Primary aux |
| B | Depth + Normals | +2-4% (geometric) | Primary geometric |
| C | CHM | +0.5-1.5% (vegetation scenes) | Optional niche |
| D | Text | +0.5-1% (visually-ambiguous) | Secondary |
| Edges | Edges | +1-2% (with depth synergy) | Supporting |
| Combined | Full 5-modal | +3-5% R@1 total | Triple-Teacher |
Top-5 Critical Reads (Master)
| # | Paper | Venue | Year | Role |
|---|---|---|---|---|
| 1 | F43 SSF (Robust PEFT) | IEEE TPAMI | 2024 | Multi-FiLM anchor (<1% overhead) |
| 2 | F39 CAFuser | IEEE RA-L | 2025 | ACF canonical (-54% params) |
| 3 | F88 KARMMA | arXiv | 2026 | 3 critical ablations (two-token, Θ-Avg, dropout) |
| 4 | F44 Fusion-Mamba | IEEE TMM | 2025 | Parallel-arm (+5.9% mAP) |
| 5 | F45 Flex-MoE | NeurIPS Spotlight | 2024 | Missing modality bank |
2. MERIDIAN Triple-Teacher Architecture
INPUT (5 modalities × 2 views = 10 channels):
┌──────────────────────────────────────────────────────────┐
│ ├── RGB sat ┌── RGB UAV │
│ ├── Depth (DepthAny v2) ├── Depth │
│ ├── Edges (Canny/HED) ├── Edges │
│ ├── Segmentation (SAM) ├── Segmentation │
│ ├── CHM (Lidar/M11 ML) ├── CHM │
│ └── Text caption (VLM) └── Text caption │
└──────────────────────────────────────────────────────────┘
↓
DINOv3-L BACKBONE (frozen, ~356M+):
┌──────────────────────────────────────────────────────────┐
│ ├── DINOv3-L SAT-493M (satellite-specialized) │
│ ├── DINOv3-L Web-LVD (web-scale) │
│ └── DINOv3-L ViT-7B (large general) │
└──────────────────────────────────────────────────────────┘
↓
FUSION MECHANISM (10 categories, primary: Multi-FiLM-Fusion):
┌──────────────────────────────────────────────────────────┐
│ Per-modality adapters (F39 CAFuser pattern, light): │
│ ├── edge_adapter (light Conv → FiLM) │
│ ├── depth_adapter (light Conv → FiLM) │
│ ├── seg_adapter (light Conv → FiLM) │
│ ├── chm_adapter (light Conv → FiLM) │
│ └── text_adapter (CLIP encoder → FiLM) │
│ │
│ Multi-FiLM-Fusion (F43 SSF pattern): │
│ F_fused = F_rgb │
│ F_fused = film_edge(F_fused, F_edge) │
│ F_fused = film_depth(F_fused, F_depth) │
│ F_fused = film_seg(F_fused, F_seg) │
│ F_fused = film_chm(F_fused, F_chm) ← when avail │
│ F_fused = film_text(F_fused, F_text) ← when avail │
│ │
│ Modality dropout p=0.5 (F88+F45+F89+F90 convergent): │
│ - Two-token KARMMA для каждой modality │
│ - Gradual schedule (F90 sigmoid warmup) │
│ │
│ Θ-Average FB reduction (F88, -81.45% memory): │
│ Output Teacher embedding compressed │
└──────────────────────────────────────────────────────────┘
↓
OUTPUT: 512-dim Teacher embedding per view (+ optional 64-token queries)
↓
KD signal (E2-E primary, см. ОБЗОР_KD_detailed_v1)
↓
STUDENT SOFIA v7.6 (edge, ~5M Tiny):
┌──────────────────────────────────────────────────────────┐
│ ├── Input: RGB sat + RGB UAV (always) │
│ ├── Backbone (Variant-A/E/Q) │
│ ├── Asymmetric Heads (SatHead GGeM + UAVHead CHP) │
│ └── Optional TextFiLM caption-aware │
│ │
│ Latency target: <50 ms Jetson Orin NX INT8 │
└──────────────────────────────────────────────────────────┘
Hybrid Pattern (Multi-FiLM + ACF combined)
MERIDIAN architecture combines:
- Shared backbone (F39 CAFuser pattern): DINOv3-L processes RGB primary
- Per-modality lightweight adapters (per F39): edge_adapter, depth_adapter, seg_adapter, chm_adapter, text_adapter
- Multi-FiLM modulation per stage (F43 SSF): Each modality contributes γ⊙F + β
- Modality dropout p=0.5 training (F88+F45 convergent)
- F88 KARMMA two-token для missing-modality (each modality)
- Θ-Average FB reduction (F88, -81.45% memory)
Critical Design Choices
| Decision | Rationale | Source |
|---|---|---|
| Shared backbone + adapters | -54% params vs separate | F39 CAFuser |
| Multi-FiLM modulation | <1% overhead PEFT | F43 SSF TPAMI |
| Zero-init β identity при init | Graceful warmup | F47 TacFiLM |
| Two-token missing-modality | +43% Epic-Kitchens evidence | F88 KARMMA |
| Θ-Average FB reduction | -81.45% memory parameter-free | F88 KARMMA |
| Modality dropout p=0.5 | 4-source convergent | F45+F88+F89+F90 |
| Element-wise gating only | INT8 compatible | F44 DSSF |
| Cached Tensors Era | No on-device modality compute | F8 SegEarth-R1 |
| Per-modality adapters light (~100K each) | Param budget | F43 PEFT principle |
Tier-1 (immediate — E1 Teacher fusion benchmark)
- Multi-FiLM-Fusion (F43 + F47) — primary mechanism
- ACF (F39 CAFuser) Condition Token — secondary for ablation
- Fusion-Mamba (F44) — parallel-arm benchmark
- Per-modality light adapters — F39 pattern (-54% params vs separate)
- Modality dropout p=0.5 — canonical (4-source convergent)
- Θ-Average FB reduction (F88) — -81.45% memory, INT8-trivial