forked from Pikaliov/fuze_task
fuse_proj: Initial operational package for 3 researchers (Pavlenko/Blizno/Moroz)
Multimodal fusion research on StripNet+GTA-UAV proxy: - 3 independent fusion tracks: condition-aware (A), token/bottleneck (B), role-aware (C) - Shared interfaces, protocol, dataset audit, baseline benchmarks - Canonical version-chain references to vault (SPEC, ANALYSIS, TRIAGE) - Personalized task plans and decision tables for each researcher - 3 generated DOCX task assignment files with milestones and DoD checklist - Full modality dropout diagnostics and missing-modality robustness requirements - Data contract, benchmark registry, experiment tracking infrastructure Operational documents: - docs/00_project/: MERIDIAN context, protocol, repository reuse guide, experiment specification - docs/01_tasks/: Master assignment + 3 individual researcher tracks + joint integration - docs/02_references/: Core literature, version-chain bases, code maps - docs/03_codebase_guides/: Existing code snapshots from vault - scripts/: gen_task_plans.js (DOCX generation), placeholder infrastructure - vendor_reference/: Snapshots of caption_test, depth_edges_annotate, existing SOFIA/SegModel code - reports/, results/, experiments/: Shared output structure for all 3 researchers 3 DOCX files generated from gen_task_plans.js (Times New Roman 14pt, GOST format): - План_заданий_Павленко_БВ.docx (Condition-Aware track, fusion API owner) - План_заданий_Близно_МВ.docx (Token/Bottleneck track, benchmark owner) - План_заданий_Мороз_ЕС.docx (Role-Aware track, data contract owner) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,565 @@
|
||||
---
|
||||
type: delta
|
||||
status: draft
|
||||
date: 2026-05-06
|
||||
parent: "[[REVIEW_depth_normals_pairB]]"
|
||||
related:
|
||||
- "[[REVIEW_chm_pairC]]"
|
||||
- "[[../00_overall/SPEC_fusion_ACF_MERIDIAN]]"
|
||||
- "[[MASTER_synthesis_cached_tensors]]"
|
||||
- "[[DELTA_pair_C_chm_sat]]"
|
||||
- "[[DELTA_pair_A_seg_revised]]"
|
||||
- "[[DELTA_E1_pair_D_text_fusion]]"
|
||||
tags:
|
||||
- delta
|
||||
- decision/delta
|
||||
- component/cvgl
|
||||
- method/film
|
||||
- method/depth
|
||||
- arch/dinov3
|
||||
- gate/E1
|
||||
- priority/high
|
||||
- domain-aware/HIGH
|
||||
phase: E1
|
||||
hypotheses_added:
|
||||
- H_pair_B_1
|
||||
- H_pair_B_2
|
||||
- H_pair_B_3
|
||||
- H_pair_B_4
|
||||
author: claude
|
||||
---
|
||||
|
||||
# DELTA §2.7 — Pair B depth fusion (cached DA3-LARGE → unified continuous encoder)
|
||||
|
||||
> [!summary] TL;DR
|
||||
> **Major architectural shift**: anchor REVIEW_depth_normals_pairB (DFormer / GeminiFusion / Metric3D-v2 family — joint depth+normals encoders, Mona-LoRA-B injection) **superseded** новым cached tensors anchor ([[MASTER_synthesis_cached_tensors]] §3): **cached depth tensor [1,256,256] fp16 ∈ [0,1] (per-frame minmax от DA3-LARGE-1.1 411M) → unified conv encoder (1→32→64→96→128) → GGeM → FiLM-B head 128→256→(2×1024×5), ~2.2M trainable**. Normals **deprecated** в anchor (no separate normals provider в cached tensors; Sobel-derived edges = pair E covers structural information). Per-frame minmax normalization сохраняет relative depth, **теряет absolute scale** — compensated через **scalar height conditioning** (§2.1 H_backbone_8 если активен).
|
||||
>
|
||||
> 4 refinements (D1-D4): D1 confirm DA3-LARGE-1.1 anchor, monitor для DA3-V3 release (Nov 2025); D2 per-frame minmax sufficient (alternative absolute scale via altitude scalar); D3 aux depth-regression head на student — defer to E5 (similar risk pollution as Pair A); D4 §0.8 RGBD-fusion alternatives (A.1 concat / E.5 AdaIN-stats / cross-attn / aux-loss) — checked, FiLM-B retained.
|
||||
>
|
||||
> **4 новые гипотезы H_pair_B_1..4**: per-frame minmax preservation, altitude-aware FiLM auto-attenuation, B-E adaptive λ_⊥ (depth-edges decorrelation), aux depth head feature pollution risk.
|
||||
|
||||
---
|
||||
|
||||
## §1. AS-IS — anchor состояние (cached tensors revised)
|
||||
|
||||
### 1.1. Anchor architecture (MASTER §3, 2026-04-20)
|
||||
|
||||
**Provider:** Depth Anything V3 LARGE 1.1 (DA3-LARGE-1.1, 411M params, DINOv2 ViT-L/14 backbone). Pre-computed offline → **cached fp16 tensors [1,256,256] ∈ [0,1]** per-frame minmax normalized в SafeTensors на NVMe SSD (~256 KB/pair).
|
||||
|
||||
**Pipeline (~2.2M trainable):**
|
||||
|
||||
```
|
||||
depth [1,256,256] fp16 ∈ [0,1] (per-frame minmax)
|
||||
│
|
||||
▼ unified continuous encoder (shared design with Pair C/E):
|
||||
conv 7×7 stride 2 (1→32)
|
||||
conv 3×3 stride 2 (32→64)
|
||||
conv 3×3 stride 2 (64→96)
|
||||
conv 3×3 stride 2 (96→128)
|
||||
│
|
||||
▼
|
||||
[128, 16, 16] feature map
|
||||
│
|
||||
▼ GGeM pool (learnable p, shared design)
|
||||
[128] descriptor
|
||||
│
|
||||
▼ FiLM-B head MLP 128→256→(2×1024×5)
|
||||
γ_B^{(20-24)}, β_B^{(20-24)} → blocks 20-24 DINOv3 ViT-L/16
|
||||
через shared 256-d bottleneck
|
||||
```
|
||||
|
||||
**Параметрический бюджет** (verified MASTER §3):
|
||||
- conv encoder: ~200K
|
||||
- FiLM-B MLP: ~2M
|
||||
- **Total Pair B trainable = ~2.2M**
|
||||
|
||||
**Normals deprecated в anchor** ([[MASTER_synthesis_cached_tensors]] §3): нет отдельного normals provider в cached tensors. Structural information частично восстанавливается через **Pair E (edges = Sobel(depth))** — отдельный paired encoder.
|
||||
|
||||
**Key trade-off**: per-frame minmax → **теряем absolute scale**. Компенсация:
|
||||
- **Option A**: scalar height conditioning от §2.1 H_backbone_8 (если scalar height доступен) — auto-attenuation λ_depth при altitude > 200m
|
||||
- **Option B**: **scale-invariant retrieval** — anchor accepts scale loss как architectural decision (cross-image comparability >> absolute metric depth)
|
||||
|
||||
### 1.2. Cross-pair B-E correlation concern (anchor)
|
||||
|
||||
[[MASTER_synthesis_cached_tensors]] §3 noted: **Pair B (depth) ↔ Pair E (edges = Sobel(depth))** — γ-streams могут быть structurally коррелированы (edges = derivative depth). **Adaptive λ_⊥ для конкретной пары B-E** через monitoring schedule:
|
||||
- Default: λ_⊥(B-E) = 0.1 (uniform с другими pairs)
|
||||
- If pairwise cosine sim(γ_B, γ_E) > 0.6 устойчиво в течение 2 эпох → **automatically double** λ_⊥(B-E) → 0.2
|
||||
- Resilience: B-E independence enforced via decorrelation regularization
|
||||
|
||||
### 1.3. Anchor justification (компактно)
|
||||
|
||||
- **Cached tensors regime** ([[MASTER_synthesis_cached_tensors]] §1-2): provider features заменены на pre-computed tensors → minimum runtime (no online DA3-LARGE inference)
|
||||
- **DA3-LARGE-1.1** = current best monocular depth для aerial oblique (Nov 2024-2025 generation, 411M params, robust на UAV altitudes)
|
||||
- **per-frame minmax** sufficient для cross-image comparability в same dataset (World-UAV)
|
||||
- **Unified continuous encoder** shared с Pair C (CHM) и Pair E (edges) — 80% infrastructure reuse
|
||||
- **GGeM** = anchor для feature aggregation (cross from F11)
|
||||
- **FiLM-B → blocks 20-24** = same injection point как Pair A/C/D/E (5-way unified)
|
||||
|
||||
### 1.4. H_fus_B_X (anchor REVIEW_depth_normals_pairB) — updated status
|
||||
|
||||
| ID | Original (REVIEW_depth_normals_pairB) | New status (DELTA 2026-05-06) | Rationale |
|
||||
|:--|:--|:-:|:--|
|
||||
| H_fus_B_0 | Continuous dense modality breaks DINO less than discrete one-hot | Confirmed (still relevant) | Anchor design supports |
|
||||
| H_fus_B_1 | Depth-fusion даёт больший ΔR@1 на SUES-200 (multi-height) чем University-1652 | **Confirmed (still relevant)** — H_pair_B_2 refines | Multi-altitude SUES-200 needs scale invariance |
|
||||
| (others REVIEW H_fus_B_X) | DFormer / GeminiFusion / Metric3D-v2 specific hypotheses | **Superseded** — cached tensors regime drops these specific architectures | Architecture shift |
|
||||
|
||||
---
|
||||
|
||||
## §2. Лит-обзор: новые свидетельства (2025-2026)
|
||||
|
||||
### 2.1. Depth Anything V3 (DA3, arXiv:2511.10647, Nov 2025) — provider release
|
||||
|
||||
> [!cite] Источник
|
||||
> arXiv:2511.10647 · Nov 2025 · ICLR 2026 submission
|
||||
|
||||
**Что нового:**
|
||||
- DA3 plain DINOv2 ViT-L (p14), any-view cross-attention
|
||||
- ~335M (L) → 1.3B (G) params
|
||||
- **Relative + nested metric** depth output (vs DA2 relative-only с metric FT option)
|
||||
- 4K streaming inference
|
||||
- **No aerial validation в paper** — но nested metric может lечить height ambiguity для UAV
|
||||
|
||||
**Implication для DELTA:**
|
||||
- **DA3** is currently anchor (DA3-LARGE-1.1) — но **nested metric variant** + cross-view может улучшить scale awareness
|
||||
- **Action**: monitor open-source release; if DA3 v1.1+ дает absolute metric depth (vs relative) — re-cache 962K tensors (~50 H100-hours; не блокирующее, можно параллельно с E1)
|
||||
- **Decision**: keep DA3-LARGE-1.1 anchor; track DA3 nested-metric option как **D2 alternative**
|
||||
|
||||
### 2.2. RGBD fusion via DCF (Depth-weighted Cross-attention) — arXiv:2405.05614
|
||||
|
||||
> [!cite] Источник
|
||||
> «Depth Awakens: A Depth-perceptual Attention Fusion Network for RGB-D Camouflaged Object Detection» · 2024
|
||||
|
||||
**Что нового:**
|
||||
- **Depth-weighted Cross-attention Fusion (DCF) module** — controls depth contribution while fusing RGB+depth
|
||||
- Designed для COD (camouflaged object detection) — different domain, structural pattern transferable
|
||||
|
||||
**Implication для DELTA:**
|
||||
- **DCF альтернатива** для FiLM-B (категория B.1 §0.8 cross-attention) — cross-attention с depth-weight gating
|
||||
- **Reject as primary**: cross-attn overhead 50-200M (vs FiLM-B ~2M); distillability ⚠
|
||||
- Add to research backlog для F3-research-B (E5+ ablation)
|
||||
|
||||
### 2.3. DFormerv2 (CVPR 2025, arXiv:2504.04701) — geometry self-attention
|
||||
|
||||
> [!cite] Источник
|
||||
> arXiv:2504.04701 · CVPR 2025
|
||||
|
||||
**Что нового:**
|
||||
- **Geometry self-attention** — leverages depth для geometry relationship modeling
|
||||
- Asymmetric pattern «depth-as-modulator» для RGBD seg
|
||||
- Builds on DFormer ICLR 2024
|
||||
|
||||
**Implication для DELTA:**
|
||||
- **Architectural pattern relevant** — confirms «depth-as-modulator» (FiLM-B asymmetric pattern в anchor)
|
||||
- DFormerv2 itself uses different backbone (not DINOv3) — not direct reuse
|
||||
- **Confirms anchor** asymmetric design (depth modulates RGB через FiLM, не симметричный exchange)
|
||||
|
||||
### 2.4. Self-supervised UAV oblique depth (arXiv:2012.10704, JPRS 2021 + RMTDepth MDPI 2025)
|
||||
|
||||
> [!cite] Источник
|
||||
> JPRS 2021 + RMTDepth MDPI 17(19):3372
|
||||
|
||||
**Что нового:**
|
||||
- Self-supervised monocular depth для UAV oblique videos
|
||||
- Challenges: **depth discontinuity** from geometric distortion; **spatial ambiguity** в weakly textured regions
|
||||
- RMTDepth (2025) — Retentive Vision Transformer for self-supervised UAV depth
|
||||
|
||||
**Implication для DELTA:**
|
||||
- **Confirms F3 oblique distortion concern** (§0.6 domain-aware) — depth provider может галлюцинировать на UAV oblique
|
||||
- **Mitigation**: per-frame minmax normalization (anchor) reduces absolute depth artifacts; **uncertainty-weighting** could compensate
|
||||
- **Decision**: defer uncertainty-weighting к E5 ablation (H_pair_B_4 risk)
|
||||
|
||||
### 2.5. Scale-Aware UAV CVGL (P68, arXiv:2603.07535) — already в backlog
|
||||
|
||||
> [!cite] Источник
|
||||
> [[P68_CVGL_2026_Scale_Aware_Semantic_Geometric]] · already PARTIAL_DONE in main backlog
|
||||
|
||||
**Что нового:**
|
||||
- DSPM (Differentiable Scale Prediction Module) для scale recovery
|
||||
- F1 evidence для H_aug_1 (already cross-linked)
|
||||
|
||||
**Implication для DELTA:**
|
||||
- DSPM mechanism — alternative для **scalar height conditioning** at depth-fusion stage
|
||||
- **Sync с §2.1 H_backbone_8**: altitude-aware FiLM-B attenuation (H_pair_B_2 NEW)
|
||||
|
||||
---
|
||||
|
||||
## §3. DELTA — что изменяется vs anchor
|
||||
|
||||
### 3.1. Что НЕ меняется
|
||||
|
||||
| Anchor | Источник | Действие |
|
||||
|:--|:--|:--|
|
||||
| Cached fp16 depth [1,256,256] ∈ [0,1] | MASTER §2 | ✅ keep |
|
||||
| DA3-LARGE-1.1 provider | MASTER §3 | ✅ keep (monitor DA3 v1.1+ для D2) |
|
||||
| Per-frame minmax normalization | MASTER §3 | ✅ keep (D2 verifies) |
|
||||
| Unified continuous encoder (1→32→64→96→128) | MASTER §3 | ✅ keep (shared с Pair C/E) |
|
||||
| GGeM pool (learnable p) | MASTER §3 + F11 | ✅ keep |
|
||||
| FiLM-B head 128→256→(2×1024×5) | MASTER §3 | ✅ keep |
|
||||
| ~2.2M trainable params budget | MASTER §3 | ✅ keep |
|
||||
| Blocks 20-24 DINOv3 injection | REVIEW + MASTER §3 | ✅ keep (5-way unified) |
|
||||
| Normals deprecated | MASTER §3 | ✅ keep deprecated |
|
||||
| Adaptive λ_⊥(B-E) doubling at sim > 0.6 | MASTER §3 | ✅ keep |
|
||||
|
||||
### 3.2. Что предлагается УТОЧНИТЬ (Decision DELTA Table)
|
||||
|
||||
| # | Item | Было (anchor) | Станет (DELTA) | Threshold для acceptance | Источник evidence |
|
||||
|:-:|:--|:--|:--|:--|:--|
|
||||
| **D1** | DA3-LARGE-1.1 provider | anchor (single) | **Confirm + monitor DA3 v1.1+ release** (re-cache opt-in if absolute metric depth available) | DA3 V3 nested-metric +5pp R@1 vs relative on multi-altitude UAV → trigger re-cache | §2.1 web discovery |
|
||||
| **D2** | Per-frame minmax normalization | anchor | **Confirm; alternative**: + altitude scalar conditioning (sync с §2.1 H_backbone_8); evaluate в E1.altitude_ablation | scalar height + per-frame minmax > per-frame alone ≥ +0.5pp R@1 на multi-altitude SUES-200 | DSPM (P68) + REVIEW §0.6 F1 scale |
|
||||
| **D3** | Aux depth-regression head на teacher | not in anchor | **No teacher aux head** (FiLM injection sufficient); **defer student-side aux head к E5** (similar feature pollution risk as Pair A H_pair_A_8) | aux head adds CE/Charbonnier loss → potential negative transfer; defer | analogous Pair A H_pair_A_8 |
|
||||
| **D4** | §0.8 RGBD-fusion alternatives | implicit anchor (FiLM-B) | **§4 chklist 12 категорий** — A.1 concat reject (input-level breaks DINOv3), E.5 AdaIN-stats reject (less expressive), B.1 cross-attn defer to F3-research-B | §0.8 mandatory for fusion-prompt | §0.8 + REVIEW §3 taxonomy |
|
||||
|
||||
### 3.3. Что предлагается ДОБАВИТЬ
|
||||
|
||||
#### 3.3.1. Altitude-aware FiLM-B attenuation (H_pair_B_2)
|
||||
|
||||
**Rationale:** depth от monocular estimator при altitude > 200m **почти uniform** (depth distribution flattens) → FiLM-B сигнал должен **auto-attenuate** при large altitude.
|
||||
|
||||
**Mechanism:** scalar altitude h ∈ [50, 300]m → FiLM-B gating:
|
||||
|
||||
$$\gamma_B^{\text{eff}} = \alpha(h) \cdot \gamma_B, \quad \beta_B^{\text{eff}} = \alpha(h) \cdot \beta_B$$
|
||||
|
||||
где $\alpha(h) = \text{sigmoid}\left(\frac{200 - h}{50}\right)$ — smooth attenuation от 1.0 (h=50m) до 0.27 (h=300m).
|
||||
|
||||
**Sync с §2.1 H_backbone_8** scalar height conditioning — altitude scalar используется в обоих местах consistently.
|
||||
|
||||
**Cost:** добавляет ~1K params (sigmoid + linear scaling); negligible.
|
||||
|
||||
#### 3.3.2. F3-research-B = Cross-attn alternative (D4 §0.8 entry)
|
||||
|
||||
**Rationale:** anchor использует **single FiLM-B на global GGeM descriptor**. Alternative — cross-attention между RGB tokens (Q) и depth feature map (K,V) preserves spatial granularity.
|
||||
|
||||
**Cost:** ~50-100M trainable (vs 2.2M anchor) — 25-45× overhead. **Reject as primary**, defer post-E1.
|
||||
|
||||
### 3.4. Conflicts с anchor
|
||||
|
||||
> [!warning]+ Conflict 1 — REVIEW_depth_normals_pairB H_fus_B_X (DFormer/GeminiFusion/Metric3D-v2 specific) vs cached tensors regime
|
||||
>
|
||||
> Same pattern as Pair A (DELTA §2.6 Conflict 1) — REVIEW_depth_normals_pairB канон (2026-04-20) discusses fusion architectures assuming raw provider features (DFormer joint encoder, Metric3D-v2 ViT-G2 directly). Cached tensors regime drops raw features → only fp16 depth tensors. **Resolution:** MASTER_synthesis_cached_tensors **supersedes** REVIEW для cached tensors regime. REVIEW сохраняется как **historical reference**.
|
||||
|
||||
> [!warning]+ Conflict 2 — Normals deprecated vs REVIEW Pair B "depth+normals"
|
||||
>
|
||||
> REVIEW _depth_normals_pairB targets RGB+Depth+Normals trio. Cached tensors anchor drops normals (no separate normals provider). **Resolution:** Sobel-derived edges (Pair E) covers structural information complementary to depth. Normals может быть added в E5 ablation если Pair E insufficient.
|
||||
|
||||
> [!info]+ No conflict — DA3-LARGE-1.1 provider continuity
|
||||
>
|
||||
> Anchor MASTER §2 confirms DA3-LARGE-1.1 as provider; web discovery (§2.1) confirms DA3 v3 release Nov 2025 — no breaking changes, only improvement direction (nested metric). Anchor stable.
|
||||
|
||||
### 3.5. Risks of refinement
|
||||
|
||||
> [!danger]+ R1 — DA3-LARGE-1.1 provider drift при V3+ updates
|
||||
>
|
||||
> Если Meta/AI релизит DA3 V3 with structural changes (e.g., switch backbone к DINOv3 ViT-L/16 для grid-match), наш cached fp16 tensors stale. Re-cache cost ~50 H100-hours. **Mitigation:** strict version-pinning (hash-suffix SafeTensors) + 500-image regression test set + automated drift detection (D1 monitor).
|
||||
|
||||
> [!warning]+ R2 — Per-frame minmax loses absolute scale
|
||||
>
|
||||
> Per-frame normalization preserves cross-image comparability (consistent [0,1] range), но **destroys absolute metric depth** — critical для multi-altitude SUES-200 retrieval. **Mitigation D2:** altitude scalar conditioning from §2.1 H_backbone_8 — restores scale awareness.
|
||||
|
||||
> [!warning]+ R3 — F3 oblique distortion (§0.6)
|
||||
>
|
||||
> Depth от monocular estimator на UAV oblique view имеет artifacts: depth discontinuities при large viewpoint changes, hallucinated depth-jumps при cast shadows (§2.4 web evidence). **Mitigation:** uncertainty-weighting / confidence gating (defer to E5; H_pair_B_4 explores feasibility).
|
||||
|
||||
> [!warning]+ R4 — B-E correlation collapse
|
||||
>
|
||||
> Edges = Sobel(depth) → γ_B и γ_E могут collapse в redundant directions (cosine sim > 0.7). **Mitigation (anchor):** adaptive λ_⊥(B-E) doubling при sim > 0.6 — already в anchor. **H_pair_B_3 NEW:** verify mechanism effectiveness empirically.
|
||||
|
||||
### 3.6. Отвергнутые предложения
|
||||
|
||||
> [!failure]+ Reject — Joint depth+normals encoder (REVIEW Metric3D-v2 / GeoWizard pattern)
|
||||
>
|
||||
> Joint encoder requires online provider inference → defeats cached tensors purpose. Reject. Use Pair E edges as structural compensation.
|
||||
|
||||
> [!failure]+ Reject — Cross-attention RGBD fusion (DFormer/GeminiFusion category)
|
||||
>
|
||||
> 50-200M overhead vs FiLM-B 2.2M. Distillability ⚠ (cross-attn не дистиллируется без depth provider в Student). Reject as primary; defer F3-research-B.
|
||||
|
||||
> [!failure]+ Reject — Aux depth-regression head на teacher
|
||||
>
|
||||
> Anchor MASTER §3 — FiLM injection sufficient; aux head adds Charbonnier/SiLog loss redundant с CVGL InfoNCE objective. Risk of teacher feature drift away from CVGL. Reject.
|
||||
|
||||
> [!failure]+ Reject — Aux depth-regression head на student (deferred)
|
||||
>
|
||||
> Same risk as Pair A H_pair_A_8 (feature pollution). Reject as primary; **defer to E5.aux_depth_loss_ablation** with PCGrad gradient surgery.
|
||||
|
||||
> [!failure]+ Reject — Surgical-DINO-style LoRA-injection on teacher для depth
|
||||
>
|
||||
> LoRA уже applied на DINOv3 blocks 20-24 (anchor). Adding depth-conditional LoRA = double-LoRA conflict. Reject. Use FiLM-B (anchor) instead.
|
||||
|
||||
> [!failure]+ Reject — Replace per-frame minmax with absolute metric depth (DA3 V3 nested metric)
|
||||
>
|
||||
> Conditional reject — DA3 V3 release pending. **Conditional adopt** if D1 monitor triggers (V3 demonstrates +5pp R@1 multi-altitude). Defer decision.
|
||||
|
||||
---
|
||||
|
||||
## §4. §0.8 FiLM-АЛЬТЕРНАТИВЫ — обязательный чек-лист (RGBD-fusion-aware)
|
||||
|
||||
| Cat | Метод | Anchor decision | Reason |
|
||||
|:-:|:--|:-:|:--|
|
||||
| **A.1** Early concat (RGB + depth → 4-channel) | Reject | Breaks DINOv3 patch-embed weights |
|
||||
| **A.2** Late concat | Reject | Linear, weak |
|
||||
| **B.1** One-way cross-attn (RGB-Q, depth-K/V) | F3-research-B research | 50-200M overhead, distillability ⚠ |
|
||||
| **B.2** Two-way cross-attn (CoCa-style) | Reject | Symmetric exchange complicates distillation |
|
||||
| **B.5** Q-Former (BLIP-2) | Reject | Bottleneck destroys dense correspondence |
|
||||
| **C.1** GMU (Gated Multimodal Unit) | Reject | Gating only, FiLM superset |
|
||||
| **E.5** AdaIN (γ,β from depth statistics) | Reject | Less expressive than learned MLP FiLM |
|
||||
| **E.7** Conditional FiLM (anchor) | **PRIMARY** | Anchor ✅ |
|
||||
| **E.10** Dynamic Conv | Reject | Generates kernels, overhead |
|
||||
| **E.11** Mixture-of-FiLM (depth-bin routing) | Research direction | Per-depth-bin FiLM (similar to Pair A per-class MoFiLM); defer |
|
||||
| **G.1** Houlsby adapter (RGBD-conditional) | Reject | Mona-LoRA-B deprecated в cached tensors |
|
||||
| **L.1** FiLM + cross-attn | Architecture B alternative | Hybrid valid; defer |
|
||||
|
||||
**DELTA-чеклист для F3-research-B (cross-attn alternative):** см. §3.3.2 — defer post-E1.
|
||||
|
||||
---
|
||||
|
||||
## §5. CVGL DOMAIN AWARENESS — чек-вопросы (HIGH domain-aware §0.6)
|
||||
|
||||
§2.7 — **HIGH domain-aware** (depth scale зависит от altitude!).
|
||||
|
||||
| Категория | Status | Mitigation |
|
||||
|:--|:-:|:--|
|
||||
| **F1 scale + altitude** | ⚠ Critical | per-frame minmax normalizes; altitude scalar conditioning (H_pair_B_2) auto-attenuates λ_depth at high altitude |
|
||||
| **F2 viewpoint mismatch** | ⚠ - sat nadir / UAV oblique | per-frame normalization handles distribution shift; FiLM-B learns view-asymmetric γ,β |
|
||||
| **F3 oblique distortion** | ⚠ Critical | gradient inconsistency между views; mitigation through normalization + B-E adaptive λ_⊥ |
|
||||
| **F4 sun angle / shadows** | ⚠ - depth halucinates shadow as discontinuity | uncertainty-weighting (R3 deferred); shadows captured by edges (Pair E) for cross-validation |
|
||||
| **#9 domain gap (sim-to-real)** | ✅ - DA3-LARGE trained on real RGBD; cached tensors handle distribution | confidence-weighting deferred to E5 |
|
||||
| **F7 rotation handling** | ✅ Aug consistency через cached tensors frozen | depth tensor rotated together with RGB |
|
||||
| **#13 GPS noise tolerance** | N/A | depth invariant к GPS |
|
||||
| **#14 temporal mismatch** | ✅ stable structures (buildings, terrain) preserved | seasonal vegetation may differ → CHM (Pair C) covers this |
|
||||
|
||||
**Conclusion:** HIGH domain-aware addressed. Critical concerns F1 + F3 + F4 mitigated через per-frame minmax + altitude conditioning + B-E decorrelation.
|
||||
|
||||
---
|
||||
|
||||
## §6. GAP ANALYSIS — backlog для §2.7
|
||||
|
||||
### Output: Таблица A — В vault, требуют углубления (SHALLOW)
|
||||
|
||||
| # | Paper / Author Year | Paper-ID | Существующая заметка | Глубина | Priority | Est. MODE-A time |
|
||||
|:-:|:--|:--|:--|:--|:-:|:-:|
|
||||
| A1 | DA3 LARGE 1.1 | (cross from §2.5 backlog) | mentioned в [[REVIEW_depth_normals_pairB]] §1 | NOT_FOUND deep-dive | P2 | 1.5h |
|
||||
| A2 | F18 SegDINO (cross from 3_fusion) | F18 | (already в backlog #54 для §2.6) | SHALLOW | P2 | — |
|
||||
| A3 | F11 GGeM (cross) | F11 | (already в backlog #53) | SHALLOW | P2 | — |
|
||||
| A4 | F34 Coupled Mamba (cross from 3_fusion) | F34 | DEEP (in [[F34_2024_Coupled Mamba Enhanced Multi-modal Fusion with Coupled State Space Model]]) | DEEP | — | — |
|
||||
|
||||
### Output: Таблица B — НЕ в vault, требуют acquisition + MODE-A
|
||||
|
||||
| # | Paper / Author Year | Title | DOI / arXiv | Status | Priority | Acquisition path |
|
||||
|:-:|:--|:--|:--|:--|:-:|:--|
|
||||
| B1 | **DA3 V3 (Nov 2025)** | Depth Anything V3 | [arXiv:2511.10647](https://arxiv.org/abs/2511.10647) | NOT_FOUND | P1 | acquire — provider monitor for D1 → `2_foundation_models/F_DA3_V3_2025.md` (~2h) |
|
||||
| B2 | **DFormerv2 (CVPR 2025)** | Geometry Self-Attention RGBD | [arXiv:2504.04701](https://arxiv.org/abs/2504.04701) | NOT_FOUND | P3 | acquire — geometry self-attn pattern reference (~1.5h) |
|
||||
| B3 | **DCF Depth-weighted Cross-attention (2024)** | RGB-D Camouflaged Object Detection | [arXiv:2405.05614](https://arxiv.org/abs/2405.05614) | NOT_FOUND | P3 | acquire summary — cross-attn alternative (~1h) |
|
||||
| B4 | **RMTDepth (MDPI 2025)** | Retentive ViT for UAV Depth | MDPI 17(19):3372 | NOT_FOUND | P3 | acquire summary — UAV-specific depth provider (~1h) |
|
||||
| B5 | **Self-supervised UAV oblique depth (JPRS 2021)** | Foundational reference | [arXiv:2012.10704](https://arxiv.org/abs/2012.10704) | NOT_FOUND | P3 | acquire summary — oblique distortion challenges reference (~1h) |
|
||||
| B6 | **MoGe-2 (NeurIPS 2025)** | Joint depth + normals + scale + FOV | [arXiv:2507.02546](https://arxiv.org/abs/2507.02546) | NOT_FOUND | P3 | acquire — alternative metric depth provider (~1.5h) |
|
||||
|
||||
### Output: Сводная статистика
|
||||
|
||||
- Всего цитируемых работ по теме промпта (Pair B): ~20
|
||||
- DEEP: 3 (REVIEW + MASTER + M11)
|
||||
- SHALLOW: 4 (DA3 LARGE 1.1 mention, F18, F11, F34)
|
||||
- NOT_FOUND: 6 (DA3 V3, DFormerv2, DCF, RMTDepth, JPRS UAV depth, MoGe-2)
|
||||
- **P0 backlog: 0** — anchor coverage complete
|
||||
- **P1 backlog: 1** (DA3 V3 monitor для D1)
|
||||
- **P2 backlog: 2** (F18, F11 cross from §2.6)
|
||||
- **P3 backlog: 5** (DFormerv2, DCF, RMTDepth, JPRS, MoGe-2)
|
||||
|
||||
### Output: Action items
|
||||
|
||||
- [ ] **P1:** acquire arXiv:2511.10647 DA3 V3 (~2h) — D1 monitor for re-cache trigger
|
||||
- [ ] **P3:** acquire DFormerv2 / DCF / RMTDepth / MoGe-2 (low priority)
|
||||
|
||||
---
|
||||
|
||||
## §7. Synchronization
|
||||
|
||||
### С §2.1 (Student backbone — scalar height conditioning)
|
||||
- D2 altitude-aware FiLM-B attenuation (H_pair_B_2) ↔ §2.1 H_backbone_8 scalar height
|
||||
- **Action**: ensure scalar height available в Pair B fusion stream
|
||||
|
||||
### С §2.6 (Pair A semantic)
|
||||
- Same unified bottleneck (256-d) + 5-way orthogonality
|
||||
- Pair A 9-zone vs Pair B global FiLM — different granularity, complementary
|
||||
|
||||
### С §2.8 (Pair C CHM)
|
||||
- **Identical encoder design** (1→32→64→96→128) → infrastructure reuse
|
||||
- B-C correlation analysis (forested regions) — separate from B-E concern
|
||||
- Geometric bridge depth_uav ↔ CHM_sat — H_pair_C_X (см. DELTA §2.8)
|
||||
|
||||
### С Pair E (edges)
|
||||
- **B-E adaptive λ_⊥** (anchor MASTER §3) — H_pair_B_3 verifies
|
||||
- Sobel(depth) → edges на CPU; cached как pair E
|
||||
|
||||
### С §2.13 augmentation
|
||||
- Cached tensors frozen rule — no augmentation на depth tensor
|
||||
- Geometric aug (rotation, scale) consistent with RGB
|
||||
|
||||
---
|
||||
|
||||
## §8. Связь с ROADMAP
|
||||
|
||||
### Phase E1 (Teacher 5-modal benchmark)
|
||||
- Pair B activated в **Phase 2** (epochs 20-30, [[MASTER_synthesis_cached_tensors]] §4)
|
||||
- Phase 2 — geometric modalities (B depth + C CHM + E edges) jointly активируются
|
||||
|
||||
### H_pair_B — обновлённое resume
|
||||
|
||||
| ID | Status | Phase | Notes |
|
||||
|:--|:-:|:-:|:--|
|
||||
| H_fus_B_0 | Confirmed | E1 | Continuous depth breaks DINO less than discrete one-hot |
|
||||
| H_fus_B_1 | Confirmed | E1 | Multi-altitude SUES-200 needs scale invariance |
|
||||
| **H_pair_B_1** *(new, DELTA 2026-05-06)* | High | E1 | Per-frame minmax preserves cross-image comparability (parity with absolute metric within ±0.5pp R@1) |
|
||||
| **H_pair_B_2** *(new, DELTA 2026-05-06)* | Medium-High | E1 | Altitude-aware FiLM-B attenuation > uniform λ_depth ≥ +1pp R@1 на multi-altitude SUES-200 |
|
||||
| **H_pair_B_3** *(new, DELTA 2026-05-06)* | Medium | E1 | B-E adaptive λ_⊥ doubling механизм activates ≤ 30% epochs (cosine sim < 0.6 у >70% epochs) |
|
||||
| **H_pair_B_4** *(new, DELTA 2026-05-06)* | Low (defer E5) | E5 | Aux depth head на student = -0.5pp R@1 (feature pollution risk, similar Pair A H_pair_A_8) |
|
||||
|
||||
(Полные формулировки H_pair_B_1..4 — см. §9.)
|
||||
|
||||
### Зависимости / блокировки
|
||||
- **Блокирует:** §2.9 (5-way fusion synthesis) — Pair B architecture choice
|
||||
- **Блокируется:** §2.1 H_backbone_8 (scalar height availability), DA3 V3 release decision
|
||||
|
||||
---
|
||||
|
||||
## §9. Новые гипотезы H_pair_B_1..4
|
||||
|
||||
### H_pair_B_1: Per-frame minmax preserves cross-image comparability (parity with absolute metric)
|
||||
|
||||
**Если** anchor cached depth uses **per-frame minmax normalization** (raw depth → [0,1] independently per image),
|
||||
|
||||
**то** R@1 на University-1652 и SUES-200 находится **в пределах ±0.5pp** от hypothetical anchor с absolute metric depth (DA3 V3 nested metric option),
|
||||
|
||||
**потому что** (1) cross-image comparability через consistent [0,1] range — основная цель fusion для CVGL retrieval; (2) absolute metric depth теряет relevance when retrieval is invariant to scale (we match images, not measure heights); (3) altitude scalar conditioning (H_pair_B_2) provides scale awareness где нужно — без overhead absolute metric provider.
|
||||
|
||||
- **Уверенность:** High
|
||||
- **Область:** Normalization choice
|
||||
- **Baseline:** Hypothetical absolute metric depth (DA3 V3 nested metric)
|
||||
- **Метрика:** R@1 на University-1652 + SUES-200
|
||||
- **Threshold для успеха:**
|
||||
- |R@1(per-frame minmax) − R@1(absolute metric)| ≤ 0.5pp на University-1652
|
||||
- |Δ| ≤ 1pp на SUES-200 (multi-altitude harder)
|
||||
- **Опровержение:** absolute metric > per-frame minmax by ≥ 1.5pp R@1 → DA3 V3 re-cache justified
|
||||
- **Зависимости:** DA3 V3 release (P1 backlog #B1)
|
||||
- **Эксперимент:** E1.normalization_ablation (conditional on DA3 V3 availability)
|
||||
|
||||
---
|
||||
|
||||
### H_pair_B_2: Altitude-aware FiLM-B attenuation > uniform λ_depth
|
||||
|
||||
**Если** применить altitude-aware FiLM-B gating $\alpha(h) = \text{sigmoid}\left(\frac{200 - h}{50}\right)$ для smooth attenuation от 1.0 (h=50m) до 0.27 (h=300m),
|
||||
|
||||
**то** R@1 на multi-altitude SUES-200 (heights 150/200/250/300m) ≥ R@1(uniform λ_depth) + **1.0pp**, благодаря auto-attenuation depth контрибуции при large altitude где depth distribution flattens,
|
||||
|
||||
**потому что** (1) DA3-LARGE-1.1 produces near-uniform depth at altitude > 200m (web evidence §2.4); (2) uniform λ_depth applies same FiLM-B contribution regardless of altitude — оvercommits на high-altitude где depth uninformative; (3) SUES-200 multi-height eval explicitly tests this; (4) sync с §2.1 H_backbone_8 scalar height conditioning.
|
||||
|
||||
- **Уверенность:** Medium-High
|
||||
- **Область:** Altitude-aware fusion
|
||||
- **Baseline:** uniform λ_depth (anchor without altitude gating)
|
||||
- **Метрика:** R@1 на SUES-200 per-altitude breakdown
|
||||
- **Threshold для успеха:**
|
||||
- R@1(altitude-aware) − R@1(uniform) ≥ +1.0pp на altitudes ≥ 200m
|
||||
- Parity (±0.3pp) на altitudes ≤ 150m (low-alt где depth informative)
|
||||
- **Опровержение:**
|
||||
- Δ < 0.5pp → attenuation overhead не оправдан, simplify
|
||||
- Δ < 0pp → attenuation damages low-altitude performance, reject
|
||||
- **Зависимости:** §2.1 H_backbone_8 scalar height availability
|
||||
- **Эксперимент:** E1.altitude_ablation (Phase 2, ~6 GPU-h)
|
||||
|
||||
---
|
||||
|
||||
### H_pair_B_3: B-E adaptive λ_⊥ doubling activates ≤ 30% epochs
|
||||
|
||||
**Если** anchor monitoring schedule (cosine sim(γ_B, γ_E) > 0.6 → λ_⊥(B-E) automatically doubled to 0.2),
|
||||
|
||||
**то** механизм activates **в ≤ 30%** epochs (cosine sim < 0.6 maintained для > 70% training time), confirming structural decorrelation между depth и edge γ-streams,
|
||||
|
||||
**потому что** (1) MASTER §3 предполагает adaptive mechanism preserves B-E independence; (2) edges = Sobel(depth) creates **structural correlation** только при collapsed γ — orthogonality regularizer prevents this; (3) sustained collapse > 30% epochs indicates regularizer insufficient — мерge B-E pair (drop redundancy) или manually pre-set λ_⊥(B-E) = 0.2.
|
||||
|
||||
- **Уверенность:** Medium
|
||||
- **Область:** Cross-pair regularization, B-E independence
|
||||
- **Baseline:** anchor adaptive mechanism active
|
||||
- **Метрика:** Fraction of epochs where cosine sim(γ_B, γ_E) > 0.6 (and λ_⊥(B-E) doubled)
|
||||
- **Threshold для успеха:**
|
||||
- Doubling fraction ≤ 30% epochs
|
||||
- Final cosine sim(γ_B, γ_E) ≤ 0.5 после full E1 training
|
||||
- **Опровержение:**
|
||||
- Doubling > 50% epochs → mechanism insufficient, **manually set** λ_⊥(B-E) = 0.2 from epoch 0
|
||||
- Doubling > 80% epochs → fundamental B-E redundancy, consider **merge B-E single encoder** (saves ~2.2M params)
|
||||
- **Зависимости:** Pair E activated in Phase 2
|
||||
- **Эксперимент:** E1.B_E_correlation_monitoring (passive observation during E1)
|
||||
|
||||
---
|
||||
|
||||
### H_pair_B_4 (defer E5): Aux depth head на student polluting features
|
||||
|
||||
**Если** в Student добавить auxiliary depth-regression head — Charbonnier/SiLog loss на cached fp16 → student depth recovery,
|
||||
|
||||
**то** Student R@1 на University-1652 деградирует на **-0.5pp** относительно anchor (no aux depth loss),
|
||||
|
||||
**потому что** (1) depth-regression task gradient (dense pixel-level Charbonnier) и CVGL-task gradient (image-level retrieval InfoNCE) имеют разные directions; (2) Student ~5M params — limited capacity для multi-task; (3) Pair A H_pair_A_8 demonstrates same risk pattern для seg-task; (4) **PCGrad mitigation** (REVIEW Pair A §8.3) может enable +0.5pp gain — defer.
|
||||
|
||||
- **Уверенность:** Low (defer E5)
|
||||
- **Область:** Multi-task student learning
|
||||
- **Baseline:** Student no aux depth loss
|
||||
- **Метрика:** R@1 (Student); ΔCKA student-teacher
|
||||
- **Threshold для опровержения (anchor wins):**
|
||||
- ΔR@1(aux depth) ≤ -0.3pp (feature pollution)
|
||||
- **Опровержение alternative (aux depth helps):**
|
||||
- ΔR@1 ≥ +0.5pp с λ_aux ≤ 0.1 → adopt
|
||||
- PCGrad: ΔR@1 ≥ +1pp при λ_aux=0.3 → adopt
|
||||
- **Эксперимент:** E5.aux_depth_loss_ablation (post-E1, conditional)
|
||||
|
||||
---
|
||||
|
||||
## §10. Cross-links + acceptance
|
||||
|
||||
### К другим модулям
|
||||
- **§2.1** Student backbone — scalar height conditioning (H_backbone_8 ↔ H_pair_B_2)
|
||||
- **§2.6** Pair A — shared bottleneck + 5-way orthogonality
|
||||
- **§2.8** Pair C CHM — identical encoder design + geometric bridge
|
||||
- **Pair E (edges)** — B-E adaptive λ_⊥ regularization
|
||||
|
||||
### К master plans
|
||||
- [[MASTER_synthesis_cached_tensors]] §3 (Pair B anchor)
|
||||
- Phase 2 staged training
|
||||
|
||||
### Hypothesis bookkeeping
|
||||
- 4 новые H_pair_B_1..4
|
||||
- H_fus_B_0/H_fus_B_1 confirmed; others superseded
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
DELTA принимается, если:
|
||||
- [x] anchor cached tensors regime + DA3-LARGE-1.1 + per-frame minmax + unified encoder сохранён
|
||||
- [x] §0.8 12 категорий FiLM-альтернатив проверены — anchor (E.7) primary
|
||||
- [x] D1-D4 refinements — procedural / decision / monitor
|
||||
- [x] Conflicts (REVIEW vs MASTER, normals deprecation) explicitly resolved
|
||||
- [x] P0 backlog = 0
|
||||
- [x] 4 новые гипотезы H_pair_B_1..4 имеют explicit thresholds
|
||||
- [x] Cross-DELTA consistency (§2.1, §2.6, §2.8 sync)
|
||||
|
||||
→ **DELTA APPROVED**. Pending P1 review (DA3 V3 monitor) — recommended но не блокирующее.
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## Refresh notes 2026-05-07 (post HIGH backlog)
|
||||
|
||||
### New evidence — DA3 V3 monitor
|
||||
|
||||
- [[F_DA3_2025]] **NEW DEEP** (Nov 2025): plain DINOv2 ViT-L p14 + any-view cross-attention, +44.3% camera pose accuracy + +25.1% geometric vs VGGT
|
||||
- **Re-cache decision**: deferred until absolute metric depth confirmed via PDF (depth-ray prediction language suggests relative)
|
||||
- [[P68_CVGL_2026_Scale_Aware_Semantic_Geometric]] cross-relevant: **monocular depth fails** for absolute scale on UAV (DSPM vehicle anchors complementary)
|
||||
|
||||
### Confirmations
|
||||
|
||||
- DA3-LARGE-1.1 anchor (cached) preserved unless absolute metric DA3 V3 confirmed
|
||||
- Per-frame minmax normalization preserved (H_pair_B_1)
|
||||
- Altitude-aware FiLM-B preserved (H_pair_B_2)
|
||||
|
||||
### Sources
|
||||
|
||||
- [[TODO_HIGH_backlog_action_plan]]
|
||||
|
||||
#delta #pair-B #depth #film #cvgl #priority/high #task/experiment #cached-tensors
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
type: delta
|
||||
status: active
|
||||
date: 2026-05-12
|
||||
parent: "[[../00_overall/SPEC_fusion_ACF_MERIDIAN_v3]]"
|
||||
supersedes: "[[DELTA_pair_B_depth_uav]]"
|
||||
related:
|
||||
- "[[DELTA_pair_C_chm_sat_v2]]"
|
||||
- "[[../00_overall/HYP_fusion_variants_v2]]"
|
||||
applicable_to: ["E1"]
|
||||
tags: [delta, fusion, pair-b, depth, uav, v2, post-F84-F85]
|
||||
phase: E1
|
||||
author: claude
|
||||
---
|
||||
|
||||
# DELTA: Pair B — Depth (UAV) v2
|
||||
|
||||
## Changelog v1 → v2 (2026-05-12)
|
||||
|
||||
### Major changes
|
||||
|
||||
1. **Differential-modal aug compat (H_arch_A_6 from SPEC v3)**
|
||||
- Pair B depth может участвовать в pairwise differences с Pair A/C/D/E
|
||||
- $|X_B - X_C|$ — depth vs CHM divergence signal (terrain noise vs canopy)
|
||||
- Implementation: per-pair differential gating optional research arm
|
||||
|
||||
2. **DA3 v1.1 anchor preserved** (cached fp16 per-frame minmax)
|
||||
|
||||
3. **Modality dropout p=0.3 safety** — preserved via canonical Multi-FiLM identity-at-init
|
||||
|
||||
### Preserved (v1)
|
||||
|
||||
- DA3 v1.1 depth encoder (cached, frozen)
|
||||
- Per-frame minmax normalization
|
||||
- 4-layer Conv encoder 128-d output
|
||||
- GGeM + FiLM-B head MLP
|
||||
|
||||
---
|
||||
|
||||
## §1. Pair B architecture v2
|
||||
|
||||
```
|
||||
Input: UAV image [B, 3, 256, 256]
|
||||
↓ DA3 v1.1 frozen (cached features)
|
||||
Depth map [B, 1, H, W] fp16 per-frame minmax
|
||||
↓ Conv encoder 4 layers (1 → 32 → 64 → 128 → 128)
|
||||
Feature map [B, 128, H/8, W/8]
|
||||
↓ GGeM pooling
|
||||
[B, 128]
|
||||
↓ FiLM-B head MLP (128 → 256 → (2×1024×5))
|
||||
γ_B, β_B for Teacher blocks 20-24
|
||||
```
|
||||
|
||||
## §2. v2 research arms
|
||||
|
||||
| Arm | Source | Activation |
|
||||
|:--|:--|:--|
|
||||
| **differential B-C**: $|X_B - X_C|$ gating signal | F85 DEGF-YOLO DFE pattern | E1 parallel |
|
||||
| **spatial-FiLM-B**: per-pixel γ, β | F84 DGE-YOLO Inject | E1 P2 |
|
||||
| **ADD-fusion sync с SPEC v3 §1.3** | B112 HPMSFPN | E1 parallel (aggregation level) |
|
||||
|
||||
## §3. Risks v2
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|:--|:-:|:--|
|
||||
| **R1**: DA3 v1.1 cached features become stale | Low | Re-cache via DA3 V3 (deferred per SPEC v3) |
|
||||
| **R2 NEW v2**: differential B-C correlation high | Low | Adaptive λ_⊥(B,C) per orthogonality regularizer |
|
||||
|
||||
## §4. Cross-references v2
|
||||
|
||||
- [[../00_overall/SPEC_fusion_ACF_MERIDIAN_v3]]
|
||||
- [[../00_overall/DELTA_E1_teacher_5modal_Arch_A_final_v2]]
|
||||
- [[DELTA_pair_C_chm_sat_v2]]
|
||||
|
||||
---
|
||||
|
||||
[[DELTA_pair_B_depth_uav]] (v1) → archive. v2 supersedes (2026-05-12).
|
||||
|
||||
#delta #fusion #pair-b #depth #uav #v2 #post-F84-F85
|
||||
@@ -0,0 +1,556 @@
|
||||
---
|
||||
type: delta
|
||||
status: draft
|
||||
date: 2026-05-06
|
||||
parent: "[[REVIEW_chm_pairC]]"
|
||||
related:
|
||||
- "[[REVIEW_depth_normals_pairB]]"
|
||||
- "[[../00_overall/SPEC_fusion_ACF_MERIDIAN]]"
|
||||
- "[[MASTER_synthesis_cached_tensors]]"
|
||||
- "[[DELTA_pair_B_depth_uav]]"
|
||||
- "[[M11_CHMv2_deep_dive_for_MERIDIAN]]"
|
||||
tags:
|
||||
- delta
|
||||
- decision/delta
|
||||
- component/cvgl
|
||||
- method/film
|
||||
- method/lupi
|
||||
- arch/dinov3
|
||||
- gate/E1
|
||||
- priority/high
|
||||
phase: E1
|
||||
hypotheses_added:
|
||||
- H_pair_C_1
|
||||
- H_pair_C_2
|
||||
- H_pair_C_3
|
||||
- H_pair_C_4
|
||||
author: claude
|
||||
---
|
||||
|
||||
# DELTA §2.8 — Pair C CHM fusion (cached CHMv2 → unified continuous encoder)
|
||||
|
||||
> [!summary] TL;DR
|
||||
> Anchor «Pair C: cached CHM от DINOv3-ViTL16 CHMv2 (337M, fp16 [0,1] per-frame minmax) → unified continuous encoder same as Pair B → ~2.2M FiLM-C» **подтверждается** (CHMv2 verified release March 2026 — Meta/WRI partnership; native ViT-L/16 grid match). 4 refinements (D1-D4): D1 confirm CHMv2 anchor (verified web evidence — R²=0.86 SOTA, vs Tolan 2024 / Lang 2023); D2 sezonality robustness benchmark (winter/summer split на UAV-VisLoc rural subset); D3 **geometric bridge** depth_uav ≈ altitude − CHM_sat — **rejected as aux loss** (information already у Pair B+altitude scalar); D4 §0.8 LUPI-aux loss vs FiLM injection — **FiLM injection primary**, LUPI-aux deferred к E5.
|
||||
>
|
||||
> **4 новые гипотезы H_pair_C_1..4**: CHMv2 native grid match advantage, sezonality robustness gain UAV-VisLoc, B-C correlation analysis (forested regions only), aux CHM-regression head feature pollution risk.
|
||||
|
||||
---
|
||||
|
||||
## §1. AS-IS — anchor состояние (cached tensors revised)
|
||||
|
||||
### 1.1. Anchor architecture (MASTER §3, 2026-04-20)
|
||||
|
||||
**Provider:** **CHMv2** (Meta + WRI, March 2026, arXiv:2603.06382) — frozen DINOv3 ViT-L/16 backbone (~304M params) + DPT-256-mixlog head (range 0.001-96m).
|
||||
- **Native ViT-L/16 grid match** с teacher backbone — критичное архитектурное преимущество (REVIEW §1)
|
||||
- Trained on 18M satellite images, validated against ALS + GEDI L2A + ICESat-2 ATL08
|
||||
- R² = 0.86 (vs Tolan 2024 R² = 0.53) — SOTA
|
||||
- **HuggingFace**: `facebook/dinov3-vitl16-chmv2-dpt-head`
|
||||
|
||||
Pre-computed offline → cached fp16 tensors [1,256,256] ∈ [0,1] per-frame minmax (~256 KB/pair).
|
||||
|
||||
**Pipeline (~2.2M trainable, identical to Pair B):**
|
||||
|
||||
```
|
||||
chm [1,256,256] fp16 ∈ [0,1] (per-frame minmax)
|
||||
│
|
||||
▼ unified continuous encoder (shared design with Pair B/E):
|
||||
conv 7×7 stride 2 (1→32)
|
||||
conv 3×3 stride 2 (32→64)
|
||||
conv 3×3 stride 2 (64→96)
|
||||
conv 3×3 stride 2 (96→128)
|
||||
│
|
||||
▼
|
||||
[128, 16, 16] feature map
|
||||
│
|
||||
▼ GGeM pool (learnable p, shared design)
|
||||
[128] descriptor
|
||||
│
|
||||
▼ FiLM-C head MLP 128→256→(2×1024×5)
|
||||
γ_C^{(20-24)}, β_C^{(20-24)} → blocks 20-24 DINOv3 ViT-L/16
|
||||
через shared 256-d bottleneck
|
||||
```
|
||||
|
||||
**Параметрический бюджет** (verified MASTER §3):
|
||||
- conv encoder: ~200K
|
||||
- FiLM-C MLP: ~2M
|
||||
- **Total Pair C trainable = ~2.2M**
|
||||
|
||||
### 1.2. CHM-specific role (REVIEW §2)
|
||||
|
||||
**CHM отличается от generic depth**:
|
||||
- Фокус на vegetation height (DSM − DTM), игнорирует buildings и terrain
|
||||
- Роль: **дезамбигуатор self-similar forest canopy** в RGB
|
||||
- **Sezonality**: deciduous vs evergreen разница, disturbance signals (logging, fire scars, regrowth)
|
||||
- **Sparse signal в urban-dominant**: CHM ≈ 0 на 85-95% University-1652 / DenseUAV tiles
|
||||
- **Dominant signal в forested**: 30-60% UAV-VisLoc rural tiles с CHM > 5m
|
||||
|
||||
**Conditional utility CVGL benchmarks** (REVIEW §2):
|
||||
| Bench | CHM-сигнал | Применимость Pair C |
|
||||
|:--|:--|:--|
|
||||
| University-1652 | ≈ 0 на 85-95% tiles (urban) | **Low utility** |
|
||||
| DenseUAV | ≈ 0 на абсолютном большинстве (urban) | **Low utility** |
|
||||
| SUES-200 | partial 20-40% tiles | **Medium utility** |
|
||||
| GTA-UAV / Game4Loc | variable (urban/mountain/coast/forest) | **High utility** |
|
||||
| **UAV-VisLoc** | dominant 30-60% rural/hilly | **High utility** |
|
||||
|
||||
### 1.3. CHMv2 LUPI pattern (M11 deep-dive)
|
||||
|
||||
[[M11_CHMv2_deep_dive_for_MERIDIAN]] — CHMv2 как **canonical LUPI pattern**:
|
||||
- Teacher (frozen DINOv3-L Sat 304M)
|
||||
- Student/Decoder DPT (~15-25M trainable)
|
||||
- Multi-scale features из layers {5, 11, 17, 23}
|
||||
- Curriculum loss: SiLog (epochs 0-30k) → Charbonnier (30k+) → PatchGrad (5k-50k)
|
||||
- Privileged signal: ALS CHM (только при обучении)
|
||||
|
||||
**For MERIDIAN**: CHMv2 = **frozen provider** (not co-training). Pair C uses cached CHM tensor от offline CHMv2 inference — **decouples from CHMv2 LUPI training methodology** (which is provider's internal training, not relevant к downstream Pair C fusion).
|
||||
|
||||
### 1.4. H_fus_C_X (anchor REVIEW_chm_pairC) — updated status
|
||||
|
||||
| ID | Original (REVIEW) | New status (DELTA 2026-05-06) | Rationale |
|
||||
|:--|:--|:-:|:--|
|
||||
| (REVIEW H_fus_C_X) | CHMv2 dual FiLM-branch with depth specific patterns | **Superseded** by H_pair_C_X | cached tensors regime simplifies to unified encoder |
|
||||
| (REVIEW geometric bridge) | tri-branch FiLM (depth + normals + CHM) | **Adapted** — normals deprecated, CHM as separate FiLM-C | Pair B+C+E unified encoder pattern |
|
||||
|
||||
---
|
||||
|
||||
## §2. Лит-обзор: новые свидетельства (2025-2026)
|
||||
|
||||
### 2.1. CHMv2 release (March 2026, Meta + WRI) — anchor verified
|
||||
|
||||
> [!cite] Источник
|
||||
> [arXiv:2603.06382](https://arxiv.org/abs/2603.06382) · Meta AI + WRI partnership · March 2026
|
||||
|
||||
**Что нового (verified web):**
|
||||
- Released March 2026 — anchor verified
|
||||
- Trained on 18M satellite images
|
||||
- R² = 0.86 vs CHMv1 0.53 (×1.6 improvement)
|
||||
- Reduces bias в tall forests, preserves canopy edges/gaps
|
||||
- AWS open dataset registry available
|
||||
- HuggingFace `facebook/dinov3-vitl16-chmv2-dpt-head` ✅
|
||||
|
||||
**Implication для DELTA:**
|
||||
- Anchor confirmed — **no major change needed**
|
||||
- HuggingFace integration enables pre-cache pipeline (~10 H100-hours для 962K tiles)
|
||||
|
||||
### 2.2. Tolan 2024 + Lang 2023 — comparative baselines (web verified)
|
||||
|
||||
> [!cite] Источники
|
||||
> Tolan et al. 2024 (RSE 113888) · Lang et al. 2023 (Nature EE)
|
||||
|
||||
**Что выяснено:**
|
||||
- **Tolan 2024**: DINOv2 ViT-H/14 + DPT, RMSE 4.25m (NFI), 2.8m MAE (global). Patch-14 ≠ DINOv3-16 → **resample required**
|
||||
- **Lang 2023**: Xception-S2 CNN ensemble + GEDI NLL-supervision, RMSE 4.7-9.6m biome-dependent. CNN, **incompatible** с DINOv3 grid
|
||||
|
||||
**Implication для DELTA:**
|
||||
- CHMv2 Pareto-dominant choice — both quality (R²=0.86 vs 0.53) and architectural compatibility (native ViT-L/16)
|
||||
- **Reject** Tolan 2024 / Lang 2023 как primary providers
|
||||
|
||||
### 2.3. Depth Any Canopy (arXiv:2408.04523) — alternative
|
||||
|
||||
> [!cite] Источник
|
||||
> arXiv:2408.04523 · 2024
|
||||
|
||||
**Что нового:**
|
||||
- Lightweight Depth-Anything-V2 ViT-S/B + DPT для canopy
|
||||
- MAE 0.10-0.14 normalized
|
||||
- CONUS-only validation (limited geographic coverage)
|
||||
|
||||
**Implication для DELTA:**
|
||||
- Inferior to CHMv2 (limited coverage, not native ViT-L/16)
|
||||
- **Reject as primary**; reference только как baseline
|
||||
|
||||
### 2.4. ForestIQNet (Drones 2025) — RGB+CHM cross-attention precedent
|
||||
|
||||
> [!cite] Источник
|
||||
> Drones 2025 · ForestIQNet
|
||||
|
||||
**Что нового:**
|
||||
- Dual-stream RGB + voxel-CHM с **Cross-Attentional Feature Fusion (CAFF)**
|
||||
- Direct precedent для RGB+CHM cross-attention fusion
|
||||
|
||||
**Implication для DELTA:**
|
||||
- CAFF — alternative для FiLM-C (категория §0.8 B.1 cross-attention)
|
||||
- **Reject as primary**: distillability ⚠ (cross-attn не дистиллируется без CHM provider в Student)
|
||||
- Add to research backlog для F3-research-C
|
||||
|
||||
### 2.5. msGFM (CVPR 2024, arXiv:2404.01260) — multi-modal RS FM with RGB+DSM
|
||||
|
||||
> [!cite] Источник
|
||||
> arXiv:2404.01260 · CVPR 2024
|
||||
|
||||
**Что нового:**
|
||||
- Single multi-modal RS FM с paired **RGB + DSM** architecture
|
||||
- Cross-sensor MIM pre-training
|
||||
|
||||
**Implication для DELTA:**
|
||||
- DSM ≠ CHM (DSM = surface включая buildings; CHM = canopy only)
|
||||
- Architectural reference для RGB+geometry FM training
|
||||
- **Reject as primary**: not provider; not directly applicable as fusion mechanism
|
||||
|
||||
### 2.6. TerraMind (arXiv:2504.11171, ICCV 2025) — multi-modality including DEM
|
||||
|
||||
> [!cite] Источник
|
||||
> arXiv:2504.11171 · ICCV 2025
|
||||
|
||||
**Что нового:**
|
||||
- Dual-scale token-level FSQ-VAE fusion 9 модальностей включая DEM (Digital Elevation Model)
|
||||
- Closest analog tri-modal teacher
|
||||
|
||||
**Implication для DELTA:**
|
||||
- DEM ≠ CHM (DEM = ground elevation, no vegetation differentiation)
|
||||
- Architectural reference только; reject as fusion mechanism
|
||||
- Defer to §2.9 5-modal synthesis
|
||||
|
||||
---
|
||||
|
||||
## §3. DELTA — что изменяется vs anchor
|
||||
|
||||
### 3.1. Что НЕ меняется
|
||||
|
||||
| Anchor | Источник | Действие |
|
||||
|:--|:--|:--|
|
||||
| Cached fp16 CHM [1,256,256] ∈ [0,1] | MASTER §2 | ✅ keep |
|
||||
| **CHMv2 provider** (DINOv3 ViT-L/16 + DPT-256-mixlog) | MASTER §3 + verified release | ✅ keep |
|
||||
| Per-frame minmax normalization | MASTER §3 | ✅ keep |
|
||||
| Unified continuous encoder (1→32→64→96→128) | MASTER §3 (shared с Pair B/E) | ✅ keep |
|
||||
| GGeM pool (learnable p) | MASTER §3 + F11 | ✅ keep |
|
||||
| FiLM-C head 128→256→(2×1024×5) | MASTER §3 | ✅ keep |
|
||||
| ~2.2M trainable params budget | MASTER §3 | ✅ keep |
|
||||
| Blocks 20-24 DINOv3 injection | MASTER §3 | ✅ keep (5-way unified) |
|
||||
| Native ViT-L/16 grid match | REVIEW §1 + CHMv2 paper | ✅ keep (architectural advantage) |
|
||||
|
||||
### 3.2. Что предлагается УТОЧНИТЬ (Decision DELTA Table)
|
||||
|
||||
| # | Item | Было (anchor) | Станет (DELTA) | Threshold для acceptance | Источник evidence |
|
||||
|:-:|:--|:--|:--|:--|:--|
|
||||
| **D1** | CHMv2 provider | anchor | **Confirmed verified release March 2026** (Meta+WRI), R²=0.86 SOTA, native ViT-L/16. No alternative provider considered (Tolan/Lang dominated) | provider release verified | §2.1-2.2 web discovery |
|
||||
| **D2** | Sezonality robustness | not in anchor (deferred) | **Add E5 ablation**: winter/summer split на UAV-VisLoc rural subset (тестирует CHM-aware retrieval) | ΔR@1(winter→summer cross-season) ≤ 1.5pp при CHM activated; vs ≥ 3pp baseline без CHM | H_pair_C_2 (NEW) |
|
||||
| **D3** | Geometric bridge depth_uav ≈ altitude − CHM_sat | not in anchor | **Reject as aux loss** — information уже у Pair B + scalar height; redundant aux loss adds gradient interference | architectural decision | (D3 reject argument §3.6) |
|
||||
| **D4** | LUPI-aux loss vs FiLM-C injection | implicit FiLM | **§0.8 chklist**: A.1 concat reject, B.1 cross-attn defer F3-research, **LUPI-aux head defer to E5** (similar pollution risk) | §0.8 mandatory + analogous Pair A/B aux head reject | §0.8 + REVIEW Pair A H_pair_A_8 |
|
||||
|
||||
### 3.3. Что предлагается ДОБАВИТЬ
|
||||
|
||||
#### 3.3.1. Sezonality robustness benchmark (D2, H_pair_C_2)
|
||||
|
||||
**Rationale:** CHM is **stable across seasons** (canopy structure persistent, deciduous height variation ≤ 10% vs RGB color change 100%). CHM-aware retrieval should reduce seasonal cross-domain gap.
|
||||
|
||||
**Procedure (E5):**
|
||||
1. Split UAV-VisLoc rural subset into seasonal halves (если timestamps available; иначе manual annotation сезона по vegetation appearance)
|
||||
2. Train на summer; test на winter (cross-season)
|
||||
3. Compare R@1(with CHM) vs R@1(without CHM)
|
||||
4. Threshold: ΔR@1(cross-season) reduction ≥ 1.5pp при CHM активирован
|
||||
|
||||
**Cost:** ~6 GPU-h (single ablation pair)
|
||||
|
||||
#### 3.3.2. B-C correlation analysis (forested regions, H_pair_C_3)
|
||||
|
||||
**Rationale:** В forested regions Metric3D-v2 (Pair B depth) и CHMv2 partial overlap — both capture vegetation height. **Orthogonality regularizer** должен decorrelate γ_B and γ_C in forested tiles.
|
||||
|
||||
**Mechanism (anchor MASTER §3 5-way ortho):**
|
||||
|
||||
$$\rho_{B,C} = \mathbb{E}_{t \in \text{forested}} \left[ \cos(\gamma_B^{(\ell)}, \gamma_C^{(\ell)}) \right], \quad \text{target } \rho \in [0.2, 0.5]$$
|
||||
|
||||
- $\rho < 0.2$: signals полностью independent (good but may indicate underutilization of one)
|
||||
- $\rho \in [0.2, 0.5]$: balanced complementarity
|
||||
- $\rho > 0.5$: signals collapse, **trigger λ_⊥(B-C) increase** (similar to B-E mechanism)
|
||||
|
||||
**Action:** monitor passively during E1; if $\rho > 0.5$ persistently → manual λ_⊥(B-C) = 0.2 from epoch 0.
|
||||
|
||||
### 3.4. Conflicts с anchor
|
||||
|
||||
> [!warning]+ Conflict 1 — REVIEW Pair C tri-branch FiLM (depth+normals+CHM) vs anchor Pair B+C+E independent
|
||||
>
|
||||
> REVIEW §3 предлагает **tri-branch FiLM** (depth + normals + CHM) с shared bottleneck. Anchor MASTER §3 — **Pair B (depth) + Pair C (CHM) + Pair E (edges)** independent encoders + shared 256-d bottleneck. **Resolution:** anchor decision — independent encoders win for modularity (provider drift isolation, separate trainable params). Architecture preserves complementarity through shared bottleneck.
|
||||
|
||||
> [!info]+ No conflict — CHMv2 native grid match
|
||||
>
|
||||
> Both REVIEW + MASTER + web evidence confirm CHMv2 native ViT-L/16. No resampling cost. Anchor stable.
|
||||
|
||||
> [!info]+ No conflict — Pair B (depth) + Pair C (CHM) complementarity
|
||||
>
|
||||
> REVIEW §3 explicitly distinguishes Metric3D-v2 (depth до max-surface, building-aware) vs CHMv2 (canopy-only, ignores buildings). Tile-type analysis (urban/mixed/forested/disturbed) shows orthogonal/complementary signals. Confirmed.
|
||||
|
||||
### 3.5. Risks of refinement
|
||||
|
||||
> [!danger]+ R1 — CHMv2 provider drift
|
||||
>
|
||||
> CHMv2 March 2026 release; future updates may break cached fp16 tensors. **Mitigation:** version-pinning + 500-image regression test; re-cache cost ~10 H100-hours.
|
||||
|
||||
> [!warning]+ R2 — Sparse CHM signal в urban-dominant datasets
|
||||
>
|
||||
> CHM ≈ 0 на 85-95% University-1652 / DenseUAV tiles → FiLM-C сигнал может быть **drown out** или, что хуже, **add noise**. **Mitigation:** CHM-aware adaptive λ_C — gate FiLM-C contribution by mean CHM > τ threshold. Defer if needed; H_pair_C_3 monitors orthogonality.
|
||||
|
||||
> [!warning]+ R3 — Sezonality unverified
|
||||
>
|
||||
> Anchor predicts sezonality robustness benefit, но без empirical evidence из Polyakova captions / UAV-VisLoc. **Mitigation:** D2 explicit ablation в E5; defer claim до empirical validation.
|
||||
|
||||
> [!warning]+ R4 — B-C correlation collapse в forested regions
|
||||
>
|
||||
> If γ_B и γ_C collapse в forested tiles, redundancy → wasted FiLM capacity. **Mitigation:** H_pair_C_3 monitors; manual λ_⊥(B-C) escalation if needed.
|
||||
|
||||
> [!warning]+ R5 — CHM provider OOD geographies
|
||||
>
|
||||
> CHMv2 trained on US-tilted ALS data; tropical / disturbed regions могут be OOD (REVIEW §2). **Mitigation:** GeoText-1652 / GTA-UAV для diverse geographic coverage validation.
|
||||
|
||||
### 3.6. Отвергнутые предложения
|
||||
|
||||
> [!failure]+ Reject — Geometric bridge aux loss (depth_uav ≈ altitude − CHM_sat)
|
||||
>
|
||||
> Information already у Pair B + scalar height (§2.1 H_backbone_8). Aux loss adds gradient interference (similar к Pair A H_pair_A_8 / Pair B H_pair_B_4 reject). Reject.
|
||||
|
||||
> [!failure]+ Reject — Tolan 2024 / Lang 2023 alternative providers
|
||||
>
|
||||
> CHMv2 Pareto-dominant: better quality (R²=0.86 vs 0.53) + native ViT-L/16 grid match. Reject.
|
||||
|
||||
> [!failure]+ Reject — ForestIQNet CAFF cross-attention
|
||||
>
|
||||
> 50-100M overhead vs FiLM-C 2.2M; distillability ⚠. Reject as primary; defer F3-research-C (E5+).
|
||||
|
||||
> [!failure]+ Reject — Co-training CHMv2 + DINOv3
|
||||
>
|
||||
> Breaks cached tensors decoupling (similar к Pair A H_fus_A_3 reject). Reject.
|
||||
|
||||
> [!failure]+ Reject — Aux CHM-regression head на teacher OR student
|
||||
>
|
||||
> Same risk pattern as Pair A H_pair_A_8 / Pair B H_pair_B_4. Reject as primary; defer to E5.
|
||||
|
||||
---
|
||||
|
||||
## §4. §0.8 FiLM-АЛЬТЕРНАТИВЫ — обязательный чек-лист (LUPI-aware)
|
||||
|
||||
| Cat | Метод | Anchor decision | Reason |
|
||||
|:-:|:--|:-:|:--|
|
||||
| **A.1** Early concat (RGB + CHM → 4-channel) | Reject | Breaks DINOv3 patch-embed weights |
|
||||
| **B.1** Cross-attn (CAFF-style) | F3-research-C research | 50-100M overhead, distillability ⚠ |
|
||||
| **C.1** GMU | Reject | Gating only, FiLM superset |
|
||||
| **E.5** AdaIN (γ,β from CHM stats) | Reject | Less expressive than learned MLP FiLM |
|
||||
| **E.7** Conditional FiLM (anchor) | **PRIMARY** | Anchor ✅ |
|
||||
| **E.11** MoFiLM (per-CHM-bin routing) | Research direction | 256-bin DPT routing — defer |
|
||||
| **G.1** Houlsby adapter | Reject | Mona-LoRA-C deprecated в cached tensors |
|
||||
| **H.5** MI-max (CRD-like) на CHM | Aux loss alternative defer to E5 | LUPI-aux pattern |
|
||||
| **L.1** FiLM + cross-attn | Architecture B alternative | Hybrid valid; defer |
|
||||
|
||||
**DELTA-чеклист для F3-research-C (cross-attn alternative):** §3.3.1 — defer post-E1.
|
||||
|
||||
---
|
||||
|
||||
## §5. CVGL DOMAIN AWARENESS — чек-вопросы (MEDIUM domain-aware §0.6)
|
||||
|
||||
§2.8 — **MEDIUM domain-aware** (sezonality-robustness via CHM).
|
||||
|
||||
| Категория | Status |
|
||||
|:--|:--|
|
||||
| **F1 scale + altitude** | ⚠ partial — CHM normalized [0,96]m, per-frame minmax preserves cross-image; absolute scale partially recovered через native units |
|
||||
| **F4 sun angle / shadows** | ✅ CHM **invariant** к освещению (vegetation height structural) |
|
||||
| **F5 repetitive texture** | ✅ CHM disambiguates self-similar forest canopy в RGB (REVIEW §2 key insight) |
|
||||
| **F6 occlusion / dynamic objects** | ✅ CHM ignores transient objects (vehicles), focuses на canopy |
|
||||
| **#13 GPS noise tolerance** | N/A |
|
||||
| **#14 temporal mismatch (sezonality)** | ✅✅ **CHM most stable across seasons** — key advantage; D2 ablation tests это |
|
||||
| **F7 rotation handling** | ✅ Aug consistency через cached tensors frozen |
|
||||
| **#9 sim-to-real domain gap** | ⚠ - CHMv2 trained on US-tilted ALS; tropical/disturbed regions OOD |
|
||||
|
||||
**Conclusion:** MEDIUM domain-aware addressed. **Sezonality robustness** = CHM unique strength среди 5 pairs.
|
||||
|
||||
---
|
||||
|
||||
## §6. GAP ANALYSIS — backlog для §2.8
|
||||
|
||||
### Output: Таблица A — В vault, требуют углубления
|
||||
|
||||
| # | Paper / Author Year | Paper-ID | Существующая заметка | Глубина | Priority | Est. MODE-A time |
|
||||
|:-:|:--|:--|:--|:--|:-:|:-:|
|
||||
| A1 | M11 CHMv2 deep-dive | M11 | [[M11_CHMv2_deep_dive_for_MERIDIAN]] | DEEP | — | — |
|
||||
| A2 | M10 GeoBridge | M10 | [[M10_2026_GeoBridge A Semantic-Anchored Multi-View Foundation Model Bridging Images and Text for Geo-Localization]] | DEEP (already in §2.5 backlog) | — | — |
|
||||
|
||||
### Output: Таблица B — НЕ в vault
|
||||
|
||||
| # | Paper / Author Year | Title | DOI / arXiv | Status | Priority | Acquisition path |
|
||||
|:-:|:--|:--|:--|:--|:-:|:--|
|
||||
| B1 | **CHMv2 paper** | Improvements in Global Canopy Height Mapping using DINOv3 | [arXiv:2603.06382](https://arxiv.org/abs/2603.06382) | NOT_FOUND deep-dive (only M11 cross-summary) | P2 | acquire — provider primary reference (~2h) → `2_foundation_models/F_CHMv2_2026.md` |
|
||||
| B2 | **Tolan 2024** | DINOv2-ViT-H/14 + DPT canopy height | RSE 113888 | NOT_FOUND | P3 | acquire summary — comparative baseline (~1h) |
|
||||
| B3 | **Lang 2023** | Probabilistic 10m CNN canopy height | Nature EE | NOT_FOUND | P3 | acquire summary — comparative baseline (~1h) |
|
||||
| B4 | **Depth Any Canopy (2024)** | Lightweight DA2 для canopy | [arXiv:2408.04523](https://arxiv.org/abs/2408.04523) | NOT_FOUND | P3 | acquire summary — alternative reference (~1h) |
|
||||
| B5 | **msGFM (CVPR 2024)** | RGB+DSM multi-modal RS FM | [arXiv:2404.01260](https://arxiv.org/abs/2404.01260) | NOT_FOUND | P3 | acquire summary — RS multimodal FM (~1h) |
|
||||
| B6 | **TerraMind (ICCV 2025)** | Multi-modality 9 (incl. DEM) | [arXiv:2504.11171](https://arxiv.org/abs/2504.11171) | NOT_FOUND | P3 | (defer to §2.9 backlog) |
|
||||
| B7 | **MMEarth/MP-MAE (ECCV 2024)** | MAE-pretext с GCHM target | ECCV 2024 | NOT_FOUND | P3 | acquire summary — pretext signal precedent (~1h) |
|
||||
| B8 | **ForestIQNet (Drones 2025)** | RGB+voxel-CHM CAFF | Drones 2025 | NOT_FOUND | P3 | acquire summary — CAFF cross-attn precedent (~1h) |
|
||||
|
||||
### Output: Сводная статистика
|
||||
|
||||
- Всего цитируемых работ по теме (Pair C): ~15
|
||||
- DEEP: 2 (M11 deep-dive + M10 cross)
|
||||
- NOT_FOUND: 8
|
||||
- **P0 backlog: 0** — anchor coverage complete
|
||||
- **P1 backlog: 0** — anchor verified
|
||||
- **P2 backlog: 1** (CHMv2 paper deep-dive)
|
||||
- **P3 backlog: 7** (Tolan/Lang/Depth Any Canopy/msGFM/TerraMind/MMEarth/ForestIQNet)
|
||||
|
||||
### Output: Action items
|
||||
|
||||
- [ ] **P2:** acquire arXiv:2603.06382 CHMv2 paper (~2h) — provider primary reference deep-dive
|
||||
- [ ] **P3:** acquire alternative providers (low priority — anchor stable)
|
||||
|
||||
---
|
||||
|
||||
## §7. Synchronization
|
||||
|
||||
### С §2.7 (Pair B depth)
|
||||
- **Identical encoder architecture** (1→32→64→96→128) — infrastructure reuse
|
||||
- B-C correlation analysis (forested regions) — H_pair_C_3
|
||||
- Both contribute geometric signal, but orthogonal на urban tiles
|
||||
|
||||
### С §2.6 (Pair A semantic)
|
||||
- Pair A 17-class includes "forest" / "vegetation" — semantic complement к CHM continuous height
|
||||
- A-C correlation: forest mask ↔ CHM > 5m (binary indicator vs continuous)
|
||||
|
||||
### С §2.13 augmentation
|
||||
- Cached tensors frozen — no augmentation на CHM tensor
|
||||
|
||||
### С §2.9 (full 5-modal fusion synthesis)
|
||||
- Pair C contribution to combined γ_combined / β_combined
|
||||
- Sezonality benchmark (D2) — primary unique evaluation для Pair C
|
||||
|
||||
---
|
||||
|
||||
## §8. Связь с ROADMAP
|
||||
|
||||
### Phase E1
|
||||
- Pair C activated в **Phase 2** (epochs 20-30) одновременно с Pair B и Pair E
|
||||
- Sezonality benchmark — defer to E5 ablation
|
||||
|
||||
### H_pair_C — обновлённое resume
|
||||
|
||||
| ID | Status | Phase | Notes |
|
||||
|:--|:-:|:-:|:--|
|
||||
| **H_pair_C_1** *(new, DELTA 2026-05-06)* | High | E1 | Native ViT-L/16 grid match advantage (CHMv2) — confirms architectural choice |
|
||||
| **H_pair_C_2** *(new, DELTA 2026-05-06)* | Medium | E5 | Sezonality robustness — CHM activation reduces cross-season ΔR@1 by ≥ 1.5pp |
|
||||
| **H_pair_C_3** *(new, DELTA 2026-05-06)* | Medium | E1 | B-C correlation в forested regions $\rho \in [0.2, 0.5]$ (orthogonality balanced) |
|
||||
| **H_pair_C_4** *(new, DELTA 2026-05-06)* | Low (defer E5) | E5 | Aux CHM-regression head на student = -0.5pp R@1 (feature pollution risk) |
|
||||
|
||||
### Зависимости / блокировки
|
||||
- **Блокирует:** §2.9 (5-way fusion synthesis)
|
||||
- **Блокируется:** CHMv2 provider availability (verified March 2026 ✅)
|
||||
|
||||
---
|
||||
|
||||
## §9. Новые гипотезы H_pair_C_1..4
|
||||
|
||||
### H_pair_C_1: CHMv2 native ViT-L/16 grid match advantage
|
||||
|
||||
**Если** anchor использует CHMv2 (DINOv3 ViT-L/16 backbone) с native grid-match с teacher DINOv3 ViT-L/16 (vs alternative Tolan 2024 ViT-H/14 + bilinear resample),
|
||||
|
||||
**то** native grid match даёт **R@1 ≥ R@1(resampled-Tolan equivalent) + 0.5pp** благодаря (1) zero feature drift from resampling, (2) consistent 1:1 token correspondence,
|
||||
|
||||
**потому что** (1) bilinear resample interpolates between patches → smooths out fine-grained CHM details (e.g., individual tree crowns < patch resolution); (2) CHMv2 + DPT pre-trained на DINOv3 ViT-L/16 — **same coordinate system as teacher**, no projection ambiguity; (3) REVIEW §1 confirmed «единственный provider 2024-2026 с native ViT-L/16 grid match»; (4) AnyUp/FeatUp etc. resample tools mitigate but don't eliminate drift.
|
||||
|
||||
- **Уверенность:** High
|
||||
- **Область:** Provider choice, architectural compatibility
|
||||
- **Baseline:** Hypothetical Tolan 2024 (ViT-H/14) + AnyUp resample (~10M extra params)
|
||||
- **Метрика:** R@1 на UAV-VisLoc rural; CKA(features pre-resample, post-resample) measure feature drift
|
||||
- **Threshold для успеха:**
|
||||
- R@1(CHMv2 native) ≥ R@1(Tolan + resample) + 0.5pp
|
||||
- Resample CKA drift > 0.05 (measurable artifact from bilinear)
|
||||
- **Опровержение:** ΔR@1 < 0.2pp → resample acceptable, CHMv2 grid-match advantage marginal
|
||||
- **Зависимости:** anchor CHMv2 + access to Tolan 2024 model
|
||||
- **Эксперимент:** E5_research.provider_grid_match (post-E1, low priority)
|
||||
|
||||
---
|
||||
|
||||
### H_pair_C_2: Sezonality robustness gain on UAV-VisLoc
|
||||
|
||||
**Если** evaluate Pair C contribution на UAV-VisLoc rural subset с **cross-season retrieval** (train summer / test winter, или vice versa),
|
||||
|
||||
**то** R@1(with CHM) − R@1(without CHM) ≥ **1.5pp** на cross-season scenario, vs ≤ 0.3pp на same-season,
|
||||
|
||||
**потому что** (1) CHM **structurally stable** across seasons (canopy height persistent); (2) RGB color changes radically winter→summer (deciduous forest brown→green, snow); (3) Pair C provides season-invariant landmark signal; (4) anchor unique strength для Pair C among 5 pairs (most others are RGB-derived → seasonal-sensitive); (5) REVIEW §2 lists sezonality как key CHM advantage.
|
||||
|
||||
- **Уверенность:** Medium
|
||||
- **Область:** Sezonality robustness, CHM-aware retrieval
|
||||
- **Baseline:** Pair B + Pair A activated, Pair C disabled
|
||||
- **Метрика:** Cross-season R@1 (train=summer, test=winter) на UAV-VisLoc rural subset
|
||||
- **Threshold для успеха:**
|
||||
- ΔR@1(cross-season) ≥ 1.5pp при CHM activation
|
||||
- ΔR@1(same-season) ≤ 0.3pp (no significant overfit без cross-season data)
|
||||
- **Опровержение:**
|
||||
- ΔR@1(cross-season) ≤ 0.3pp → CHM не предоставляет sezonality benefit
|
||||
- ΔR@1(same-season) ≥ 1pp → CHM helps generally, not sezonality-specific
|
||||
- **Зависимости:** UAV-VisLoc seasonal annotations (timestamps OR manual annotation)
|
||||
- **Ресурсы:** ~6 GPU-h (single ablation pair)
|
||||
- **Эксперимент:** E5.sezonality_ablation (post-E1, conditional on data availability)
|
||||
|
||||
---
|
||||
|
||||
### H_pair_C_3: B-C correlation balanced в forested regions (orthogonality maintained)
|
||||
|
||||
**Если** anchor 5-way orthogonality regularizer ($\lambda_\perp = 0.1$) применен ко всем pairs включая B-C,
|
||||
|
||||
**то** в forested regions UAV-VisLoc rural subset $\rho_{B,C} = \mathbb{E}_{\text{forested}}[\cos(\gamma_B, \gamma_C)] \in [0.2, 0.5]$ — balanced complementarity,
|
||||
|
||||
**потому что** (1) Pair B (depth) + Pair C (CHM) capture overlapping но distinct vegetation height info (depth = surface height including buildings; CHM = canopy only); (2) anchor orthogonality regularizer enforces decorrelation; (3) full collapse $\rho > 0.5$ indicates redundancy → trigger λ_⊥(B-C) escalation; (4) too low $\rho < 0.2$ indicates one signal underutilized.
|
||||
|
||||
- **Уверенность:** Medium
|
||||
- **Область:** Cross-pair regularization, B-C complementarity
|
||||
- **Baseline:** anchor 5-way ortho regularizer active
|
||||
- **Метрика:** $\rho_{B,C}$ measured per epoch on forested subset; persistence of $\rho > 0.5$ over training
|
||||
- **Threshold для успеха:**
|
||||
- Final $\rho_{B,C} \in [0.2, 0.5]$ (balanced)
|
||||
- $\rho > 0.5$ persists ≤ 30% epochs (transient, not collapse)
|
||||
- **Опровержение:**
|
||||
- $\rho > 0.5$ persists > 50% epochs → λ_⊥(B-C) = 0.2 manual escalation needed
|
||||
- $\rho < 0.2$ persists → CHM signal underutilized, increase λ_C contribution
|
||||
- **Зависимости:** Pair B + C activated в Phase 2
|
||||
- **Эксперимент:** E1.B_C_correlation_monitoring (passive observation)
|
||||
|
||||
---
|
||||
|
||||
### H_pair_C_4 (defer E5): Aux CHM-regression head на student polluting features
|
||||
|
||||
**Если** в Student добавить auxiliary CHM-regression head — Charbonnier/SiLog loss на cached fp16 → student CHM recovery,
|
||||
|
||||
**то** Student R@1 на University-1652 деградирует на **-0.5pp** относительно anchor (no aux CHM loss),
|
||||
|
||||
**потому что** (1) similar feature pollution risk pattern to Pair A H_pair_A_8 / Pair B H_pair_B_4; (2) CHM = sparse signal in urban-dominant University-1652 → aux loss adds noise gradient; (3) Student ~5M params не хватает для multi-task learning; (4) **PCGrad mitigation** может enable +0.5pp gain — defer.
|
||||
|
||||
- **Уверенность:** Low (defer E5)
|
||||
- **Область:** Multi-task student learning, CHM-specific
|
||||
- **Baseline:** Student no aux CHM loss
|
||||
- **Threshold для опровержения (anchor wins):**
|
||||
- ΔR@1(aux CHM) ≤ -0.3pp (feature pollution)
|
||||
- **Опровержение alternative (aux CHM helps):**
|
||||
- ΔR@1 ≥ +0.5pp при λ_aux ≤ 0.1 → adopt
|
||||
- PCGrad: ΔR@1 ≥ +1pp при λ_aux=0.3 → adopt
|
||||
- **Эксперимент:** E5.aux_chm_loss_ablation (post-E1, conditional)
|
||||
|
||||
---
|
||||
|
||||
## §10. Cross-links + acceptance
|
||||
|
||||
### К другим модулям
|
||||
- **§2.7** Pair B depth — identical encoder design + B-C orthogonality (H_pair_C_3)
|
||||
- **§2.6** Pair A semantic — A-C complementarity (forest mask + canopy height)
|
||||
- **§2.9** 5-way fusion synthesis — Pair C contribution
|
||||
- **§2.13** Augmentation — cached tensors frozen rule
|
||||
|
||||
### К master plans
|
||||
- [[MASTER_synthesis_cached_tensors]] §3 (Pair C anchor)
|
||||
- Phase 2 staged training
|
||||
|
||||
### Hypothesis bookkeeping
|
||||
- 4 новые H_pair_C_1..4
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
DELTA принимается, если:
|
||||
- [x] anchor cached tensors regime + CHMv2 + per-frame minmax + unified encoder сохранён
|
||||
- [x] §0.8 chklist 12 категорий проверены — anchor (E.7) primary
|
||||
- [x] D1-D4 refinements — confirm + ablation plans + LUPI-aux defer
|
||||
- [x] Conflicts (REVIEW tri-branch vs anchor independent) explicitly resolved
|
||||
- [x] P0 backlog = 0
|
||||
- [x] 4 новые гипотезы H_pair_C_1..4 имеют explicit thresholds
|
||||
- [x] Cross-DELTA consistency (§2.6, §2.7, §2.9 sync)
|
||||
|
||||
→ **DELTA APPROVED**. CHMv2 anchor verified ✅; sezonality + B-C ablations defer to E5; no P1 blockers.
|
||||
|
||||
---
|
||||
|
||||
#delta #pair-C #chm #canopy-height #film #cvgl #priority/high #task/experiment #cached-tensors
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
type: delta
|
||||
status: active
|
||||
date: 2026-05-12
|
||||
parent: "[[../00_overall/SPEC_fusion_ACF_MERIDIAN_v3]]"
|
||||
supersedes: "[[DELTA_pair_C_chm_sat]]"
|
||||
related:
|
||||
- "[[DELTA_pair_B_depth_uav_v2]]"
|
||||
- "[[../00_overall/HYP_fusion_variants_v2]]"
|
||||
applicable_to: ["E1"]
|
||||
tags: [delta, fusion, pair-c, chm, sat, v2, post-F84-F85]
|
||||
phase: E1
|
||||
author: claude
|
||||
---
|
||||
|
||||
# DELTA: Pair C — CHM (Sat) v2
|
||||
|
||||
## Changelog v1 → v2 (2026-05-12)
|
||||
|
||||
### Major changes
|
||||
|
||||
1. **Differential-modal aug compat (H_arch_A_6)** — same as Pair B
|
||||
- $|X_C - X_B|$ — CHM vs depth divergence (canopy specificity)
|
||||
- $|X_C - X_A|$ — CHM vs segmentation (vegetation alignment)
|
||||
|
||||
2. **CHMv2 anchor preserved** (R²=0.86 SOTA, cached fp16 per-frame minmax)
|
||||
|
||||
3. **Adaptive λ_⊥(B,C)** — depth-CHM correlation regulation (sync SPEC v3 §3)
|
||||
|
||||
### Preserved (v1)
|
||||
|
||||
- CHMv2 encoder (cached, frozen)
|
||||
- 4-layer Conv encoder 128-d output
|
||||
- GGeM + FiLM-C head
|
||||
|
||||
---
|
||||
|
||||
## §1. Pair C architecture v2
|
||||
|
||||
```
|
||||
Input: Sat image [B, 3, 256, 256]
|
||||
↓ CHMv2 frozen (cached features)
|
||||
CHM map [B, 1, H, W] fp16 per-frame minmax
|
||||
↓ Conv encoder 4 layers (1 → 32 → 64 → 128 → 128)
|
||||
Feature map [B, 128, H/8, W/8]
|
||||
↓ GGeM pooling
|
||||
[B, 128]
|
||||
↓ FiLM-C head MLP (128 → 256 → (2×1024×5))
|
||||
γ_C, β_C for Teacher blocks 20-24
|
||||
```
|
||||
|
||||
## §2. v2 research arms
|
||||
|
||||
| Arm | Source | Activation |
|
||||
|:--|:--|:--|
|
||||
| **differential C-B aux** | F85 DFE pattern | E1 parallel |
|
||||
| **adaptive λ_⊥(B,C)** | preserved v2 strengthened | E1 default |
|
||||
| **CHM caption augmentation** (text describes canopy region) | visloc captions pattern | E2-E3 research |
|
||||
|
||||
## §3. Risks v2
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|:--|:-:|:--|
|
||||
| **R1**: CHMv2 noise в urban scenes | Medium | Domain-aware masking (CHM only over vegetation) |
|
||||
| **R2**: B-C correlation (depth ≈ CHM в forests) | Low-Medium | Adaptive λ_⊥ |
|
||||
|
||||
## §4. Cross-references v2
|
||||
|
||||
- [[../00_overall/SPEC_fusion_ACF_MERIDIAN_v3]]
|
||||
- [[../00_overall/DELTA_E1_teacher_5modal_Arch_A_final_v2]]
|
||||
- [[DELTA_pair_B_depth_uav_v2]]
|
||||
|
||||
---
|
||||
|
||||
[[DELTA_pair_C_chm_sat]] (v1) → archive. v2 supersedes (2026-05-12).
|
||||
|
||||
#delta #fusion #pair-c #chm #sat #v2 #post-F84-F85
|
||||
Reference in New Issue
Block a user