Pikaliov/caption-test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Switch to shared DINOv3 WEB encoder (saves ~4-5 GB VRAM)

Browse Source 
- Single DINOv3 WEB for both drone and satellite branches (shared_encoder=True default)
- One set of MONA adapters instead of two: 7M trainable vs 14M
- Total params: 438M (was 748M), trainable: 10.6M (was 17.6M)
- Asymmetric mode still available via shared_encoder=False
- Add gradient accumulation (grad_accum_steps, --grad-accum CLI flag)
- Update model summary in README

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
2026-04-21 21:25:46 +03:00
parent 46b1208891
commit da2d2ea90e
4 changed files with 53 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
README.md
View File

@@ -254,16 +254,18 @@ Mixed precision: AMP fp16 for model forward, fp32 for loss
| Component | Params | Trainable | Notes |
|-----------|--------|-----------|-------|
| DINOv3 ViT-L/16 LVD (drone) | 303M | frozen | backbone weights frozen |
| MONA adapters (drone) | 7.0M | 7.0M | 2 per block × 24 blocks, bottleneck=64 |
| DINOv3 ViT-L/16 SAT (satellite) | 303M | frozen | backbone weights frozen |
| MONA adapters (satellite) | 7.0M | 7.0M | 2 per block × 24 blocks, bottleneck=64 |
| DINOv3 ViT-L/16 WEB (shared) | 303M | frozen | single encoder for drone + satellite |
| MONA adapters (shared) | 7.0M | 7.0M | 2 per block × 24 blocks, bottleneck=64 |
| DGTRS-CLIP ViT-L-14 (text) | 124M | frozen | backbone weights frozen |
| LoRA adapters (text) | 147K | 147K | Q+V, rank=4, 12 blocks |
| TextFusionMLP (shared) | 3.4M | 3.4M | Linear(2304,1024) + GELU + Linear(1024,1024) |
| GatedFusion α_q + α_g | 2 | 2 | separate gate scalars |
| logit_scale | 1 | 1 | learnable temperature |
| **Total** | **748M** | **17.6M (2.35%)** | retrieval dim = 1024 |
| **Total (shared)** | **438M** | **10.6M (2.42%)** | retrieval dim = 1024 |
> **Asymmetric mode** (`--shared-encoder false`): uses separate DINOv3 WEB (drone) + DINOv3 SAT
> (satellite) encoders with independent MONA adapters. Total: 748M params, 17.6M trainable.
> Requires ~4-5 GB more VRAM.
## Experiments