Custom dataset similar to Python generator?

Gluon
1

Hi!

I am coming from Keras/Tensorflow and would like to move to mxnet. So far it looks awesome.

I am however wondering if there is any possibility to set up a custom dataset where the data is generated similar to a Python generator? I am working on time series data and reading it all into memory and storing it in a 2D array is not an option. In Tensorflow this is possible: https://www.tensorflow.org/guide/data#consuming_python_generators

Somewhere I read that using a custom dataset and dataloader this should be no problem? The API will not read all the data into memory at once.

Finally, I found this post https://guertl.me/post/183427090170/custom-dataset-and-dataloader-in-gluon on how to create custom datasets. Could you recommend me any other tutorials?

This is my first post so please let me know if I have done something wrong. Thanks!

2

If you search in the forum you’ll find several discussions about custom datasets. This may also help:

1 Like
3

Thanks! I’ll check it out. :slight_smile:

1 Like