How to load multi-rec files efficiently?

Performance
1

I made several rec files from different datasets, i 'd like to train the multi-rec files at the same time efficiently. I have tried “for” cycle to read each of the rec of a mini-batch , it works, but slow.

Does anyone has any good idea to load multi-rec files efficiently?

ps: I used imagerecorditer

thanks.

2

Hi @Eva_Hua, couple of questions…

  • Are the RecordIO files from the different datasets of the same schema?
  • Could you provide a quick snippet of what you’re doing at the moment.
  • When you say slow, is it slower than a single RecordIO file? And did you manage to diagnose the issue?