Add gradient accumulation support
- New config field grad_accum_steps (default=1, no change in behavior) - Loss scaled by 1/accum, optimizer step every N micro-batches - Scheduler counts optimizer steps (not micro-batches) - CLI flag --grad-accum for override - Document gradient accumulation and in-batch negatives in README Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -15,6 +15,7 @@ TrainConfigGTAUAV.learning_rate = 1e-4
|
||||
TrainConfigGTAUAV.text_lr_factor = 0.1
|
||||
TrainConfigGTAUAV.weight_decay = 1e-4
|
||||
TrainConfigGTAUAV.grad_clip = 1.0
|
||||
TrainConfigGTAUAV.grad_accum_steps = 1
|
||||
TrainConfigGTAUAV.use_amp = True
|
||||
TrainConfigGTAUAV.eval_every = 2
|
||||
TrainConfigGTAUAV.warmup_epochs = 2
|
||||
|
||||
Reference in New Issue
Block a user