What Is Entropy? From Thermodynamics to Information Theory Explained

Students familiar with the entropy weighting method in evaluation models will recognize its use of entropy information to assign indicator weights, while decision‑tree models in machine learning also rely on entropy‑based criteria such as information gain.

What Is Entropy?

Entropy (entropy, from Greek "entropia" meaning "inner") originates from the second law of thermodynamics and represents a system’s internal degree of disorder. Energy is conserved (the first law), but molecular motion creates entropy, which quantifies the irreversible loss of usable energy.

Entropy Increase and Negative Entropy

Entropy increase (entropy growth) describes a system evolving from order to disorder, while negative entropy (negentropy) describes the opposite trend, such as plants converting sunlight into biomass.

Discovery of the Entropy Law

In 1824 Carnot introduced the principle that the efficiency of heat engines depends only on temperature differences. Clausius later refined this, leading to the formal statement of the second law of thermodynamics in 1850, and finally introduced the term "entropy" in 1865, proving that the total entropy of an isolated system never decreases.

Thus, entropy increase is a natural tendency; reducing entropy requires external work.

Mathematical Description of Entropy

Shannon and Information Entropy

In 1948 Claude Shannon defined information entropy to quantify the amount of uncertainty in a random variable. The greater the uncertainty, the larger the entropy, and the more information is needed to resolve it.

Information Entropy Example

Using a World Cup champion prediction as an example, if all 32 teams have equal chance, five binary questions are sufficient, yielding an entropy of five bits. When probabilities differ, fewer questions are needed, resulting in lower entropy.

Relation Between Information Entropy and Thermal Entropy

Shannon’s entropy shares the same mathematical form as thermal entropy, but thermal entropy has physical units while information entropy is dimensionless. Thermal entropy can be seen as a special case of Shannon entropy applied to molecular phase‑space distributions.

Entropy Law as a Underlying Principle of Life

The book "Entropy Law" applies the concept to personal development, suggesting that individuals can combat entropy by creating dissipative structures and avoiding path dependence. A dissipative structure is an open system where continuous external work maintains order, analogous to regular exercise or continual learning.

Avoiding path dependence—sticking to a single, closed trajectory—encourages exploration of new, open systems, leading to personal and organizational growth.

Conclusion

The article traces entropy from its thermodynamic origins to Shannon’s information theory and shows how the principle can be leveraged for personal and organizational improvement, summarizing key concepts for a deeper grasp of entropy.