When Does Simulated Data Match Real Data? Comparing Model Calibration Functions using Genetic Algorithms

Agent-based models can be manipulated to replicate real- world datasets, but choosing the best set of parameters to achieve this result can be difficult. To validate a model, the real-world dataset is often divided into a training and test set. The training set is used to calibrate the parameters and the test set is used to determine if the calibrated model represents the real-world data. The difference between the real-world data and the simulated data is determined using an error measure. When using an evolutionary computation technique to choose the parameters, this error measure becomes the fitness function, and choosing the appropriate measure becomes even more crucial for a successful calibration process. We survey the effect of five different error measures in the context of a toy problem and a real world problem (simulating on-line news consumption). We use each error measure in turn to calibrate on the training dataset, and then examine the results of all five error measures on both the training and testing datasets. For the toy problem, one measure was the Pareto-dominant choice for calibration, but no error measure dominated all the others for the real-world problem.