Wall time vs CPU time.
In my experiment, the relative difference between the two times is < 0.01%. I’m wondering is this reasonable?
Variance of experiment results.
I observed that the variance of experiment results is pretty big, and the results vary depending on the specific time that you run the experiments. I guess we’d better run our experiment multiple times during some off-peak hours?
[Feb 2 Update: It becomes much more stable after I disable the mpi barrier right before the second timer, since I found that the cores with relatively large index number drag the speed down. And if I set the pernode flag, the timing on different machines becomes uniform. P.S. I’ve already put lots of IP address in the m list]
In most of the times, wall and cpu time are close for me.
My guess that processes are short and they are not swapped
by the time-sharing system.
The variance is large when you run during peak periods.
So running during off-peak could help.
The pernode option could also help, if you do not run out of machines and wrap around.