Add gradient accumulation support
- New config field grad_accum_steps (default=1, no change in behavior) - Loss scaled by 1/accum, optimizer step every N micro-batches - Scheduler counts optimizer steps (not micro-batches) - CLI flag --grad-accum for override - Document gradient accumulation and in-batch negatives in README Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
24
README.md
24
README.md
@@ -186,6 +186,30 @@ Symmetric InfoNCE with learnable temperature (CLIP-style `logit_scale`):
|
||||
|
||||
Loss and adapters run in **fp32** (AMP autocast disabled) to prevent gradient overflow.
|
||||
|
||||
### Gradient accumulation
|
||||
|
||||
With `batch_size=8` on a 24 GB GPU, VRAM is the bottleneck. Gradient accumulation
|
||||
emulates a larger effective batch without extra memory:
|
||||
|
||||
```
|
||||
effective_batch_size = batch_size × grad_accum_steps
|
||||
```
|
||||
|
||||
| Setting | `batch_size` | `grad_accum_steps` | Effective batch | In-batch negatives |
|
||||
|---------|:---:|:---:|:---:|:---:|
|
||||
| Default | 8 | 1 | 8 | 7 |
|
||||
| Recommended | 8 | 8 | 64 | 7 per micro-batch |
|
||||
|
||||
**Note:** gradient accumulation averages gradients across micro-batches, but each
|
||||
micro-batch still only sees `batch_size` in-batch negatives. To increase the number
|
||||
of negatives per forward pass, increase `batch_size` directly (requires more VRAM).
|
||||
|
||||
```bash
|
||||
# Example: effective batch of 64 with 8 accumulation steps
|
||||
python -m src.training.train_gtauav --config conf/gtauav_balanced.gin \
|
||||
--filter-meta meta/seg_filter.json --grad-accum 8
|
||||
```
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Formula | Direction |
|
||||
|
||||
Reference in New Issue
Block a user