How to train with parquet files?


How do I train mxnet with parquet files?
I have the training data stored in a bunch of parquet files(hundreds) and they cannot fit in memory (2TB+). Until now we have been able to not deal with the issue, because we could handle the training data in memory (we ran on 728GB memory sagemaker instances, but that is no longer sufficient)
We have been looking a long time for solutions, but nothing seems to be working. We are considering switching to PyTorch as that can handle a petastorm reader, which should work with parquet files. However, we feel like there has to be some solution we are not seeing.

Did you solve your problem? I have the same caseā€¦

Online marketers make sure that they create a strong lead generating campaign by seeking help from different channels like traditional marketing methods, social media marketing, search engine optimization, and other web based solutions The lead generator is a tool through which you can generate prospective customers and thus, increase your potential customers. It is one of the key ingredients for any online business. There are various ways through which you can use a lead generator and make sure that you generate high quality and qualified leads for your business.