INFORMATION ENTROPY OF IEEE 754 COMPUTING STANDARDS: THEORETICAL ANALYSIS AND PRACTICAL IMPLICATIONS

Abstract

This paper proposes an original approach to the study of IEEE 754 floating-point number formats through the lens of Shannon’s information theory. Unlike traditional research that examines IEEE 754 formats exclusively from the perspective of computational accuracy or hardware efficiency, this work systematically investigates the informationentropy characteristics of each functional field – the sign bit, the exponent field, and the mantissa field. An original metric called “specific entropy efficiency” (SEE) is proposed to quantitatively compare the information density of different floating-point formats. A detailed entropy analysis of four standard formats – binary16, binary32, binary64, and binary128 – is conducted, accounting for the specifics of denormalized numbers, dual zero encoding, and the semantic redundancy of NaN values. For the first time, four sources of entropy redundancy in IEEE 754 are systematically classified and an analytical dependence between exponent field width and NaN-redundancy level is established. A binary analogue of Benford’s law in IEEE 754 formats is identified and quantified: non-uniform distribution of the upper mantissa bits creates a compression potential of approximately 2 % per mantissa bit. The results provide a theoretical foundation for designing new numerical formats with optimal information characteristics, particularly for neural network accelerators and large-scale numerical data processing systems.