Draft:Constrained minimum criterion
- Comment: This appears to be a relatively new mechanism, with the key papers appearing in the past couple of years. There doesn't yet appear to be coverage of this Criterion which is independent and a secondary source, per WP:GNG. Chumpih t 10:31, 24 April 2023 (UTC)
In statistics, the Constrained Minimum Criterion (CMC) is a criterion for selecting regression models founded on the classical theory of likelihood based inference. It is a frequentist alternative to the Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) with certain advantages.
Geometric motivation
For a full regression model with predictor variables and an intercept, the unknown vector of regression parameters is a -vector. Elements of corresponding to active variables are non-zero, and elements corresponding to inactive variables are all zero. The likelihood ratio confidence region for is centred on its maximum likelihood estimator . As the sample size goes to infinity, the confidence region shrinks in size and degenerates into , and converges to . It follows that the whole confidence region converges to , so for sufficiently large , elements of vectors in the confidence region corresponding to active variables are all non-zero. This implies that when is captured by the confidence region, it is a vector in the region having the most zeros in its elements. Because of this, the CMC chooses from the confidence region a vector with the most zeros in its elements as an estimate of , thereby selecting the model defined by variables corresponding to non-zero elements of the chosen vector.
Definition
Let be the collection of subsets of the variables where each represents a subset. Denote by the maximum likelihood estimator for the vector of regression parameters of the reduced model defined by . Augment to be of dimension by adding zeros to its elements to represent variables not in . For a fixed , denote by the likelihood ratio confidence region for which is a region in the -dimensional space centred on . The CMC chooses the model represented by the solution vector of the following constrained minimization problem,
where denotes the norm. The solution vector is called the CMC solution, which is a sparse estimator of . Its corresponding model is called the CMC selection. When there are two or more solution vectors to the minimization problem, the one with the highest likelihood is chosen to be the CMC solution.[1]
Asymptotic properties
Let be the CMC solution and be the corresponding CMC selection. Under regularity conditions for the asymptotic normality of the maximum likelihood estimator , () the CMC solution is consistent in that
as , and () the probability that is the true model has an asymptotic lower bound
where denotes the unknown true model containing only and all active variables.
Tuning parameter
The tuning parameter controls the balance between the false active rate and false inactive rate of the selected model which is also the balance between the fit and the sparsity of the selected model. When the sample size is large, the asymptotic lower bound in () shows that setting to a small value will lead to a high probability that the CMC selection is the true model. When is not large, a small will lead to a high false inactive rate, so a larger value should be used. The recommended default value is . At this default value, the CMC is often more accurate than the AIC and BIC in terms of false active rate and false inactive rate.
The tuning parameter makes it easy to adapt the CMC to special situations such as when is small. The AIC and BIC both require special adjustments to their penalty terms for small situations. The CMC can handle such situations with a simple change of the level. In asymptotic properties () and () above, the level is fixed. Stronger results may be obtained by allowing to vary with . For selecting Gaussian linear models, one may let go to zero at a certain speed depending on as goes to infinity so that the CMC selection is consistent;[2] that is, one may find a sequence of tuning parameter values such that
Computation
For the best subset selection, the AIC and BIC require the computation of the maximum likelihood of all models. The CMC may require far fewer. Denote by the model containing all variables except the th variable . Denote by the maximum likelihood estimator and by the maximum log-likelihood ratio of this model. In some cases, the value of alone is sufficient to determine if will be selected by the CMC. One can first compute and use it to determine if will be selected for . Suppose this has identified variables that will be selected. Then, one only needs to select from the remaining variables. The total number of models that need to be computed by the CMC is thus which could be substantially smaller than required by the AIC and BIC.
Remarks
Comprehensive discussions of model selection philosophies and criteria can be found in the literature. [3] [4] [5] In other model selection strategies such as the AIC and BIC, the sparsity of the selected model comes as a by-product of the model selection process. By directly minimizing the size of the model subject to a lower bound constraint on the likelihood ratio, the CMC is the first model selection method to explicitly pursue the sparsity of the selected model.
See also
- Akaike information criterion
- Bayesian information criterion
- Hannan–Quinn information criterion
- Deviance information criterion
- Minimum message length
References
- ^ Tsao, Min (2023). "Regression model selection via log-likelihood ratio and constrained minimum criterion". Canadian Journal of Statistics. 52: 195–211. arXiv:2107.08529. doi:10.1002/cjs.11756. S2CID 236087375.
- ^ Tsao, Min (2021). "A constrained minimum method for model selection". Stat. 10. doi:10.1002/sta4.387. S2CID 236549659.
- ^ Ding, Jie; Tarokh, Vahid; Yang, Yuhong (2018). "Model Selection Techniques: An Overview". IEEE Signal Processing Magazine. 35 (6): 16–34. arXiv:1810.09583. Bibcode:2018ISPM...35f..16D. doi:10.1109/MSP.2018.2867638. S2CID 53035396.
- ^ Kadane, J.B.; Lazar, N.A. (2004). "Methods and criteria for model selection". Journal of the American Statistical Association. 99 (465): 279–290. doi:10.1198/016214504000000269. S2CID 3138924.
- ^ Miller, Alan (2019). Subset selection in regression (2nd ed.). Chapman & Hall. ISBN 9780367396220.
Further readings
- Burnham, K. P.; Anderson, D. R. (2002), Model Selection and Multimodel Inference: A practical information-theoretic approach (2nd ed.), Springer-Verlag.
- Claeskens, G.; Hjort, N. L. (2008), Model Selection and Model Averaging, Cambridge University Press.
- Hurvich, C. M.; Tsai, C.-L. (1989), "Regression and time series model selection in small samples", Biometrika, 76 (2): 297–307, doi:10.1093/biomet/76.2.297.
- Konishi, S.; Kitagawa, G. (2008), Information Criteria and Statistical Modeling, Springer.
- McQuarrie, A. D. R.; Tsai, C.-L. (1998), Regression and Time Series Model Selection, World Scientific.