Entropy and Information Gain Calculator
Calculate
e.g. "Sunny: 2, 3" = 2 positive and 3 negative examples reach the Sunny branch. Use the same class order on every line.
Try an example
Information gain
IG = 0.2467 bits
Gain ratio 0.1564 · split information 1.5774 bits · parent Gini 0.4592
Branches
| Branch | Counts | Weight | Entropy (bits) |
|---|---|---|---|
| Parent | 9, 5 | 1 | 0.9403 |
| Sunny | 2, 3 | 5/14 | 0.971 |
| Overcast | 4, 0 | 4/14 | 0 |
| Rain | 3, 2 | 5/14 | 0.971 |
A decision tree (ID3) splits on the attribute with the highest information gain; C4.5 uses the gain ratio to avoid favouring attributes with many values.
Calculate the Shannon entropy of a class distribution, or the information gain, gain ratio and Gini impurity of splitting a data set on an attribute, as a decision tree (ID3 or C4.5) would. Each branch's entropy and weight is shown.
Formula
Worked Example
The parent mix of 9 and 5 has entropy 0.940 bits. Overcast is pure (entropy 0) while Sunny and Rain stay mixed at 0.971 bits each; weighting them 5/14, 4/14 and 5/14 gives 0.694, so splitting on Outlook removes 0.247 bits of uncertainty.
Frequently Asked Questions
What does entropy measure?
The average uncertainty, in bits, about which class a randomly chosen example belongs to. A pure set has entropy 0; two equally likely classes have 1 bit, the most for two classes, and n equally likely classes have log₂ n bits.
What is the difference between information gain and gain ratio?
Information gain favors attributes with many values: splitting on a unique ID gives every branch one example and maximal gain while being useless for prediction. C4.5's gain ratio divides the gain by the split information, the entropy of the branch sizes, to correct for that.
Should I use entropy or Gini impurity?
They rank splits almost identically in practice. Gini (1 − Σp²) is slightly cheaper to compute and is the default in CART and scikit-learn; entropy is used by ID3 and C4.5.