Add gradient accumulation support

- New config field grad_accum_steps (default=1, no change in behavior)
- Loss scaled by 1/accum, optimizer step every N micro-batches
- Scheduler counts optimizer steps (not micro-batches)
- CLI flag --grad-accum for override
- Document gradient accumulation and in-batch negatives in README

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
2026-04-21 21:21:56 +03:00
parent b5c8015616
commit 46b1208891
3 changed files with 70 additions and 26 deletions

View File

@@ -186,6 +186,30 @@ Symmetric InfoNCE with learnable temperature (CLIP-style `logit_scale`):
Loss and adapters run in **fp32** (AMP autocast disabled) to prevent gradient overflow.
### Gradient accumulation
With `batch_size=8` on a 24 GB GPU, VRAM is the bottleneck. Gradient accumulation
emulates a larger effective batch without extra memory:
```
effective_batch_size = batch_size × grad_accum_steps
```
| Setting | `batch_size` | `grad_accum_steps` | Effective batch | In-batch negatives |
|---------|:---:|:---:|:---:|:---:|
| Default | 8 | 1 | 8 | 7 |
| Recommended | 8 | 8 | 64 | 7 per micro-batch |
**Note:** gradient accumulation averages gradients across micro-batches, but each
micro-batch still only sees `batch_size` in-batch negatives. To increase the number
of negatives per forward pass, increase `batch_size` directly (requires more VRAM).
```bash
# Example: effective batch of 64 with 8 accumulation steps
python -m src.training.train_gtauav --config conf/gtauav_balanced.gin \
--filter-meta meta/seg_filter.json --grad-accum 8
```
### Metrics
| Metric | Formula | Direction |