forked from Pikaliov/fuze_task
Загрузить файлы в «reports»
П. 1.1. Конспект по пункту П. 1.2. Разбор personal_package, заметки, Evidence matrix (≥8 источников)
This commit is contained in:
138
reports/Fusion RGB MM full v1.md
Normal file
138
reports/Fusion RGB MM full v1.md
Normal file
@@ -0,0 +1,138 @@
|
||||
|
||||
# 1. Резюме (Master)
|
||||
|
||||
Full multi-modal fusion в MERIDIAN — **холистический pipeline** для Triple-Teacher (DINOv3-L SAT/Web-LVD/7B, frozen, ~356M+) объединяющий **5 modalities × 2 views = 10 channels** через 10 категорий fusion-парадигм. Master synthesis 4 sub-pair reviews + general fusion review.
|
||||
|
||||
```
|
||||
DINOv3-L backbone (frozen): ~356M
|
||||
Per-modality adapters (×5): ~500K (5 × 100K, light)
|
||||
Multi-FiLM γ,β projections (×5): ~50K (5 × ~10K)
|
||||
KARMMA tokens (5 modalities): ~100K (5 × 20K)
|
||||
Θ-Average: 0 (parameter-free)
|
||||
─────────────────────────────────────────
|
||||
Total trainable params: ~650K (~0.2% backbone)
|
||||
```
|
||||
|
||||
### Master Outcomes
|
||||
|
||||
1. **Primary fusion mechanism:** Multi-FiLM-Fusion (F43 SSF TPAMI 2024 anchor + F47 zero-init β) — **<1% params overhead, INT8-compatible**
|
||||
2. **Secondary mechanism:** ACF Condition Token (F39 CAFuser RA-L 2025) — **-54% params vs separate backbones**
|
||||
3. **Parallel-arm research:** Fusion-Mamba (F44 TMM 2025) — **+5.9% mAP detection benchmark**
|
||||
4. **Critical ablations (F88 KARMMA):**
|
||||
- Two-token missing-modality strategy
|
||||
- Θ-Average FB reduction (-81.45% memory)
|
||||
- Modality dropout 50% canonical
|
||||
5. **Missing modality handling** convergent evidence (4 sources): F45 + F88 + F89 + F90 → **dropout p=0.5 canonical**
|
||||
6. **Production-ready INT8 stack** confirmed (AUDIT_N6 v2: ~0-2 ms fusion overhead)
|
||||
|
||||
### Per-Pair Contribution Summary
|
||||
|
||||
|Pair|Modality|Expected R@1 gain|Status|
|
||||
|:-:|:--|:-:|:--|
|
||||
|**A**|Segmentation|+2-4% (L_seg aux)|Primary aux|
|
||||
|**B**|Depth + Normals|+2-4% (geometric)|Primary geometric|
|
||||
|**C**|CHM|+0.5-1.5% (vegetation scenes)|Optional niche|
|
||||
|**D**|Text|+0.5-1% (visually-ambiguous)|Secondary|
|
||||
|**Edges**|Edges|+1-2% (with depth synergy)|Supporting|
|
||||
|**Combined**|Full 5-modal|**+3-5% R@1 total**|Triple-Teacher|
|
||||
### Top-5 Critical Reads (Master)
|
||||
|
||||
| # | Paper | Venue | Year | Role |
|
||||
| :-: | :------------------------ | :-------------------- | :--: | :----------------------------------------------- |
|
||||
| 1 | **F43 SSF (Robust PEFT)** | **IEEE TPAMI** | 2024 | Multi-FiLM anchor (<1% overhead) |
|
||||
| 2 | **F39 CAFuser** | **IEEE RA-L** | 2025 | ACF canonical (-54% params) |
|
||||
| 3 | **F88 KARMMA** | arXiv | 2026 | 3 critical ablations (two-token, Θ-Avg, dropout) |
|
||||
| 4 | **F44 Fusion-Mamba** | **IEEE TMM** | 2025 | Parallel-arm (+5.9% mAP) |
|
||||
| 5 | **F45 Flex-MoE** | **NeurIPS Spotlight** | 2024 | Missing modality bank |
|
||||
# 2. MERIDIAN Triple-Teacher Architecture
|
||||
|
||||
```
|
||||
INPUT (5 modalities × 2 views = 10 channels):
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ ├── RGB sat ┌── RGB UAV │
|
||||
│ ├── Depth (DepthAny v2) ├── Depth │
|
||||
│ ├── Edges (Canny/HED) ├── Edges │
|
||||
│ ├── Segmentation (SAM) ├── Segmentation │
|
||||
│ ├── CHM (Lidar/M11 ML) ├── CHM │
|
||||
│ └── Text caption (VLM) └── Text caption │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
DINOv3-L BACKBONE (frozen, ~356M+):
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ ├── DINOv3-L SAT-493M (satellite-specialized) │
|
||||
│ ├── DINOv3-L Web-LVD (web-scale) │
|
||||
│ └── DINOv3-L ViT-7B (large general) │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
FUSION MECHANISM (10 categories, primary: Multi-FiLM-Fusion):
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Per-modality adapters (F39 CAFuser pattern, light): │
|
||||
│ ├── edge_adapter (light Conv → FiLM) │
|
||||
│ ├── depth_adapter (light Conv → FiLM) │
|
||||
│ ├── seg_adapter (light Conv → FiLM) │
|
||||
│ ├── chm_adapter (light Conv → FiLM) │
|
||||
│ └── text_adapter (CLIP encoder → FiLM) │
|
||||
│ │
|
||||
│ Multi-FiLM-Fusion (F43 SSF pattern): │
|
||||
│ F_fused = F_rgb │
|
||||
│ F_fused = film_edge(F_fused, F_edge) │
|
||||
│ F_fused = film_depth(F_fused, F_depth) │
|
||||
│ F_fused = film_seg(F_fused, F_seg) │
|
||||
│ F_fused = film_chm(F_fused, F_chm) ← when avail │
|
||||
│ F_fused = film_text(F_fused, F_text) ← when avail │
|
||||
│ │
|
||||
│ Modality dropout p=0.5 (F88+F45+F89+F90 convergent): │
|
||||
│ - Two-token KARMMA для каждой modality │
|
||||
│ - Gradual schedule (F90 sigmoid warmup) │
|
||||
│ │
|
||||
│ Θ-Average FB reduction (F88, -81.45% memory): │
|
||||
│ Output Teacher embedding compressed │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
OUTPUT: 512-dim Teacher embedding per view (+ optional 64-token queries)
|
||||
↓
|
||||
KD signal (E2-E primary, см. ОБЗОР_KD_detailed_v1)
|
||||
↓
|
||||
STUDENT SOFIA v7.6 (edge, ~5M Tiny):
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ ├── Input: RGB sat + RGB UAV (always) │
|
||||
│ ├── Backbone (Variant-A/E/Q) │
|
||||
│ ├── Asymmetric Heads (SatHead GGeM + UAVHead CHP) │
|
||||
│ └── Optional TextFiLM caption-aware │
|
||||
│ │
|
||||
│ Latency target: <50 ms Jetson Orin NX INT8 │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Hybrid Pattern (Multi-FiLM + ACF combined)
|
||||
|
||||
**MERIDIAN architecture combines:**
|
||||
|
||||
1. **Shared backbone (F39 CAFuser pattern):** DINOv3-L processes RGB primary
|
||||
2. **Per-modality lightweight adapters** (per F39): edge_adapter, depth_adapter, seg_adapter, chm_adapter, text_adapter
|
||||
3. **Multi-FiLM modulation per stage (F43 SSF):** Each modality contributes γ⊙F + β
|
||||
4. **Modality dropout p=0.5** training (F88+F45 convergent)
|
||||
5. **F88 KARMMA two-token** для missing-modality (each modality)
|
||||
6. **Θ-Average FB reduction** (F88, -81.45% memory)
|
||||
|
||||
### Critical Design Choices
|
||||
|
||||
| Decision | Rationale | Source |
|
||||
| :--------------------------------------- | :---------------------------- | :--------------------- |
|
||||
| Shared backbone + adapters | -54% params vs separate | **F39** CAFuser |
|
||||
| Multi-FiLM modulation | <1% overhead PEFT | **F43** SSF TPAMI |
|
||||
| Zero-init β identity при init | Graceful warmup | **F47** TacFiLM |
|
||||
| Two-token missing-modality | +43% Epic-Kitchens evidence | **F88** KARMMA |
|
||||
| Θ-Average FB reduction | -81.45% memory parameter-free | **F88** KARMMA |
|
||||
| Modality dropout p=0.5 | 4-source convergent | **F45+F88+F89+F90** |
|
||||
| Element-wise gating only | INT8 compatible | **F44** DSSF |
|
||||
| Cached Tensors Era | No on-device modality compute | **F8** SegEarth-R1 |
|
||||
| Per-modality adapters light (~100K each) | Param budget | **F43** PEFT principle |
|
||||
### Tier-1 (immediate — E1 Teacher fusion benchmark)
|
||||
|
||||
1. **Multi-FiLM-Fusion (F43 + F47)** — primary mechanism
|
||||
2. **ACF (F39 CAFuser) Condition Token** — secondary for ablation
|
||||
3. **Fusion-Mamba (F44)** — parallel-arm benchmark
|
||||
4. **Per-modality light adapters** — F39 pattern (-54% params vs separate)
|
||||
5. **Modality dropout p=0.5** — canonical (4-source convergent)
|
||||
6. **Θ-Average FB reduction (F88)** — -81.45% memory, INT8-trivial
|
||||
Reference in New Issue
Block a user