Does lazy evaluation make batch size less important?

Performance
, ,
1

Hi all,

I’m hoping for some clarification on lazy evaluation: https://mxnet.incubator.apache.org/versions/master/tutorials/basic/ndarray.html#lazy-evaluation-and-automatic-parallelization

This use is for inference, consider I run 50 samples of size (1, 3,256,256) in a for loop, then later I retrieve the results. Lazy evaluation seems to say nothing is executed until I actually need the output. Is this the equivalent of running a single (50, 3, 256, 256) through the net?

#Dummy pseudocode
for sample in samples:
    net_out = my_net(sample)
    mx.nd.concat(output, net_out, dim=0)
#Operations are lazily evaluated here
operation_output = output.asnumpy()

# Format samples in size (50,3,256,256)
net_out = my_net(samples)
# Is this quicker than above?
operation_output = output.asnumpy()

To clarify, my question is: Does lazy evaluation mean MXNet is automatically batching input?

2

Lazy evaluation means that data processing operations are not executed until their results are needed. This allows MXNet’s engine to analyze data dependencies and to schedule computations. It does not mean that input data is automatically batched. So you still need to provide your data in form of batches.