Optimize MONA: fp16, remove conv7x7, bottleneck 64→32
- Remove forced fp32 cast in MONA forward (runs in AMP fp16 now) - Remove conv7x7 from MonaOp (keep 3x3 + 5x5 only) - Reduce default bottleneck from 64 to 32 - MONA params: 3.5M (was 7.0M, -50%) - Total trainable: 7.0M (was 10.5M) - Peak VRAM at bs=24: 18.6 GB (was 20.3 GB before fp16) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -298,7 +298,7 @@ class AsymmetricEncoder(nn.Module):
|
||||
init_gate: float = 0.7,
|
||||
baseline_mode: bool = False,
|
||||
shared_encoder: bool = True,
|
||||
mona_bottleneck: int = 64,
|
||||
mona_bottleneck: int = 32,
|
||||
lora_rank: int = 4,
|
||||
device: str = "cuda",
|
||||
) -> None:
|
||||
|
||||
Reference in New Issue
Block a user