Optimize MONA: fp16, remove conv7x7, bottleneck 64→32

- Remove forced fp32 cast in MONA forward (runs in AMP fp16 now)
- Remove conv7x7 from MonaOp (keep 3x3 + 5x5 only)
- Reduce default bottleneck from 64 to 32
- MONA params: 3.5M (was 7.0M, -50%)
- Total trainable: 7.0M (was 10.5M)
- Peak VRAM at bs=24: 18.6 GB (was 20.3 GB before fp16)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
2026-04-21 21:40:23 +03:00
parent 6b7bcae198
commit d279a6e745
2 changed files with 6 additions and 11 deletions

View File

@@ -298,7 +298,7 @@ class AsymmetricEncoder(nn.Module):
init_gate: float = 0.7,
baseline_mode: bool = False,
shared_encoder: bool = True,
mona_bottleneck: int = 64,
mona_bottleneck: int = 32,
lora_rank: int = 4,
device: str = "cuda",
) -> None: