Evaluation of excluding deficient models on multi-model analyses using AICc and KIC information criteria

AICc and KIC (Akaike second order and Kashyap information criteria) were compared in experiments where hydraulic conductivity and recharge were optimized for a set of experimental models using different boundary conditions and calibration data sets. Use of experimental models allows us to know “true” conditions, and thus evaluate procedures for improving model predictions. Multi-model analysis was conducted for each full model set and for a model subset which excluded deficient models. Objective criteria were used to identify deficient models. Model-averaged predictions based on AICc and KIC were compared to determine the impact of removal of deficient models. Using full model sets, AICc results were generally more precise than KIC, but less accurate such that some AICc model-averaged predictions did not include the true prediction within the confidence region. This condition persisted even with the removal of deficient models. Use of KIC to model-average predictions results in an extremely wide confidence region for some model sets, but precision was improved for most sets when deficient models were removed. The confidence region based on KIC increased for one experimental set after deficient models were removed. With a reduced model set, KIC results were generally more accurate but less precise than AICc. One exception was a model set where KIC was more precise, but in that case the true predictions were not contained within the KIC confidence region. Some model sets included only two or three models after deficient models were removed and in those cases AICc and KIC results were nearly identical.  In conclusion, for these experiments, the removal of deficient models did not change the quality of AICc model-averaged predictions, but KIC performance was improved. This is likely due to the inclusion of the Fisher Information term in the KIC criterion which will be unreasonably small for deficient models.