MXNetError: Out of range value for lr, value='6.018531076210112e-39'

[cbahia]

Hi,

I did run the “Estimator” with 2000 epochs, please see below:

specify the DeepAR model

estimator = DeepAREstimator(freq=‘1min’, prediction_length=5, trainer=Trainer(epochs=2000))

fit the model on the training data

predictor = estimator.train(training_data=train_data, validation_data=val_data)

When get to the epoch=1191/2000, I got the error below:

86%|████████▌ | 43/50 [00:02<00:00, 21.32it/s, epoch=1191/2000, avg_epoch_loss=7.6 92%|█████████▏| 46/50 [00:02<00:00, 21.33it/s, epoch=1191/2000, avg_epoch_loss=7.6 98%|█████████▊| 49/50 [00:02<00:00, 21.33it/s, epoch=1191/2000, avg_epoch_loss=7.6100%|██████████| 50/50 [00:02<00:00, 20.91it/s, epoch=1191/2000, avg_epoch_loss=7.61]
1it [00:00, 63.97it/s, epoch=1191/2000, validation_avg_epoch_loss=8.52]
0%| | 0/50 [00:00<?, ?it/s]
Traceback (most recent call last):
File “G:\Python Kalman\gluonts minuto.py”, line 84, in
predictor = estimator.train(training_data=train_data, validation_data=val_data)
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\gluonts\mx\model\estimator.py”, line 238, in train
return self.train_model(
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\gluonts\mx\model\estimator.py”, line 215, in train_model
self.trainer(
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\gluonts\mx\trainer_base.py”, line 410, in call
epoch_loss = loop(
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\gluonts\mx\trainer_base.py”, line 334, in loop
trainer.step(batch_size)
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\mxnet\gluon\trainer.py”, line 347, in step
self._update(ignore_stale_grad)
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\mxnet\gluon\trainer.py”, line 461, in _update
updater(i, w, g)
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\mxnet\optimizer\optimizer.py”, line 2127, in call
self.optimizer.update_multi_precision(i, w, g, self.states[i])
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\mxnet\optimizer\optimizer.py”, line 306, in update_multi_precision
self.update(index, weight, grad, state)
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\mxnet\optimizer\optimizer.py”, line 1627, in update
adam_update(weight, grad, mean, var, out=weight,
File “”, line 115, in adam_update
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\mxnet_ctypes\ndarray.py”, line 82, in _imperative_invoke
check_call(_LIB.MXImperativeInvokeEx(
File “C:\Users\Dalva\AppData\Local\Programs\Python\Python38\Lib\site-packages\mxnet\base.py”, line 246, in check_call
raise get_last_ffi_error()
mxnet.base.MXNetError: MXNetError: Out of range value for lr, value=‘6.018531076210112e-39’, in operator adam_update(name=“”, lr=“6.018531076210112e-39”, beta1=“0.9”, beta2=“0.999”, epsilon=“1e-08”, wd=“1e-08”, rescale_grad=“0.03125”, clip_gradient=“10.0”, lazy_update=“True”)

I did research over the internet, with no success…
Is there anyone who can give a direction ?

Thank you so much

[EmilPi]

I got this error recently too, after I migrated to MXNet v2.0.0.
Seems obvious, that your learning rate became too small, you are probably using some learning decay factor, or some learning rate scheduler.
Just keep them in sane range. Maybe calculate learning rate decay based on the total number of training epochs.

[cbahia]

@cbahia

estimator = DeepAREstimator(freq=‘1min’, prediction_length=5, trainer=Trainer(epochs=2000))

Thanks for reply @EmilPi…

My mx.version is ‘1.7.0’

I believe that I did not set a learning rate, learning decay factor, or some learning rate scheduler…

My “estimator” is described below:

estimator = DeepAREstimator(freq=‘1min’, prediction_length=5, trainer=Trainer(epochs=2000))

Do you have any clue ?

Thank you so much

[EmilPi]

Maybe you haven’t set, but they are set in the code you are using?
Is it based on some GitHub repo? You can put a link here.

[cbahia]

hi EmilPi,

Thank you so much for your help, I am try to the another way…please se the code below:

Now I cannot have a prediction from the last day (2022-12-27 14:03:00 ) on the test_data, the plot start on “2022-12-24 08:53”, please see the attached file.

training_data → target → data inicial do DF

training_data = ListDataset(
[{“start”: df.index[0], “target”: df.value[:“2022-12-27 10:04:00”]}],
freq = “1min”
)

#n_epochs = 15 # original
n_epochs = 1
estimator = DeepAREstimator(freq=“1min”,
prediction_length=12,

                        trainer=Trainer(epochs=n_epochs
                                        ))

t1 = time.strftime(FMT, time.localtime())

predictor = estimator.train(training_data=training_data)

test_data = ListDataset(
[{“start”: df.index[0], “target”: df.value[:“2022-12-27 14:03:00”]}],
freq = “1min”
)

for test_entry, forecast in zip(test_data, predictor.predict(test_data)):
to_pandas(test_entry)[-60:].plot(linewidth=2)
forecast.plot(color=‘b’, prediction_intervals=[100.0, 100.0])

df
value
timestamp
2022-12-23 12:54:00 111410.0
2022-12-23 12:55:00 111380.0
2022-12-23 12:56:00 111320.0
2022-12-23 12:57:00 111335.0
2022-12-23 12:58:00 111345.0
… …
2022-12-27 13:59:00 110115.0
2022-12-27 14:00:00 110145.0
2022-12-27 14:01:00 110090.0
2022-12-27 14:02:00 110090.0
2022-12-27 14:03:00 110115.0

[1200 rows x 1 columns]

Thank you so much

[cbahia]

Hi EmilPi,

I did attach a print screen of the code, where I could not understand why the last real value plotted is dated of “2022-12-24 05:20:00” and my last real value from data frame I have “2022-12-27 10:31:00” ?
What I am doing wrong ?

Thank you so much for the support

best regards and Happy New Year

[cbahia]

Hi EmilPi

Any clue ?

Thanks