Add gradient accumulation support

- New config field grad_accum_steps (default=1, no change in behavior)
- Loss scaled by 1/accum, optimizer step every N micro-batches
- Scheduler counts optimizer steps (not micro-batches)
- CLI flag --grad-accum for override
- Document gradient accumulation and in-batch negatives in README

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
2026-04-21 21:21:56 +03:00
parent b5c8015616
commit 46b1208891
3 changed files with 70 additions and 26 deletions

View File

@@ -15,6 +15,7 @@ TrainConfigGTAUAV.learning_rate = 1e-4
TrainConfigGTAUAV.text_lr_factor = 0.1
TrainConfigGTAUAV.weight_decay = 1e-4
TrainConfigGTAUAV.grad_clip = 1.0
TrainConfigGTAUAV.grad_accum_steps = 1
TrainConfigGTAUAV.use_amp = True
TrainConfigGTAUAV.eval_every = 2
TrainConfigGTAUAV.warmup_epochs = 2