Describing Experimental/Simulation Setup

Often, the results of a performance analysis depend on the computers used and specific features of software or libraries used. Hence, it is essential to describe our experimental/simulation set up in details. It enables others to repeat those experiments and check whether the results are rigorous, statistically sound, and unbiased. Unfortunately, “Simulation Setup” is the shortest section in many research papers and thesis where authors try to save space by cutting down as much as details.

Following are some tips on what to include (in addition to describing the experimental/simulation setup) based on my research and supervision experience:
Type of Simulation

• Mention whether the results are obtained using Experimentation, Emulation, or Simulation? If it is a simulation, also mention further details like whether it is Discrete Event, Monte Carlo, Stochastic, or Deterministic simulation
• Also, mention when the results were measured, e.g., Steady-state, Dynamic, Starting/ramp-up, or Terminating state(s)

Number of Experiments/Simulations

• Number of samples taken or experiments conducted
• Sampling methods — is it Random, Systematic/deterministic, Stratified, Snowballing, etc.?
• Confidence intervals or accuracy levels. 95% confidence interval is quite common, which means the probability of the population mean value being between ∓1.96 standard deviations from the sample mean is 95%
• Random number generation, whether a different seed is used for every run, what seeds were used (particularly if you use a widely available simulator/emulator)

Computer(s) Used

• CPU — Clock speed, no of cores, cache sizes, CPU model (specific model no in addition to saying, e.g., Intel i7, Xeon, AMD Opteron), and special instructions used (e.g., MMX, SSE, AVX) or compiler options set
• Memory — capacity, speed
• Accelerators used, e.g., GPUs or Intel Phi (only when they could have an impact)
• Sensors and actuators attached (only when they could have an impact)
• Operating system — type, version, special features, other services that were running, load on the node, performance enhancements/optimizations