AI Glossary
The complete dictionary of Artificial Intelligence
Mean Decrease in Impurity (MDI)
Variable importance evaluation method that measures the average reduction in impurity (Gini or entropy) brought by each feature during the construction of trees in an ensemble model.
Mean Decrease in Accuracy (MDA)
Permutation importance technique that evaluates the impact of a variable by measuring the performance drop of the model when the values of this variable are randomly permuted on the test set.
Gain Ratio Importance
Importance measure based on the information gain ratio, which normalizes the information gain by the entropy of the feature to penalize variables with a large number of distinct values.
Out-of-Bag (OOB) Feature Importance
Importance evaluation technique using the OOB samples of each tree in a random forest, measuring the increase in OOB error when the values of a variable are permuted in these samples.
Conditional Permutation Importance
Variant of permutation importance that respects dependencies between features by permuting values conditionally on other variables, reducing bias for correlated features.
Drop Column Importance
Importance evaluation method that measures the impact of completely removing a feature by retraining the model without this variable and comparing performance with the full model.
Impurity-based Feature Importance
Class of importance evaluation methods based on the reduction of node impurity during tree construction, including MDI and Gini Importance, but which can be biased towards high-cardinality features.