Is dataset transform executed at training or instanciation?

I see that gluon datasets can accomodate a transform function. Is this executed over the data at instanciation of the dataset or executed asynchronously, when needed during training or testing?

The dataloading and pre-processing (transform) is happening asynchronously. That increases performance and ensures that data is loaded into GPU memory while the model is processing the previous batch. Gluon Dataloader allows to use multiple worker processes to perform data pre-processing (transform). This helps to avoid GPU/CPU under-utilization.

see the lazy=True parameter, by default the transforms are executed at run-time but you can have them pre-executed.

1 Like