mxnet version: 1.4.0 operation system: linux Dear all: Now I have set all ‘grad_req’ in all Parameter to ‘add’, hoping to accumulate gradient over N batches to solve the problem of memory limitations. This is my code: [47] However, I encounter this Warning when I run the training code: UserWa…

Hey, bro!!! GOOD NEWS!!! I think I figure out the problem!!! The ‘grad_req’ of ‘bn0_moving_mean’ is ‘null’, which means it does not need gradient: [52] However, I write this line in my training code: [42] which compulsively change all ‘grad_req’ to ‘add’, and hence leads to the warning, after …

About stale gradient

Gluon

BluebirdStory April 28, 2019, 2:00pm 9

I make this change to gluon.SymbolBlock.imports, but the warning still exists

Aggregate gradients manually over n batches

Topic		Replies	Views
Implementation of weighted softmax by extending mx.autograd.Function fails	2	655	September 2, 2019
Gradient fetching Discussion	2	594	May 31, 2018
How to implement the addtion of grad in the backback-propagating,how to add extra term (which is the gradient to middle net layer output) to the network	2	593	August 18, 2018
Aggregate gradients manually over n batches Gluon	26	6625	July 2, 2020
WGAN-gp: can't compute gradient penalty with gluon? Gluon	0	412	October 15, 2020

About stale gradient

Related Topics