{"id":86,"date":"2023-06-25T20:07:47","date_gmt":"2023-06-25T20:07:47","guid":{"rendered":"https:\/\/theoreticaltrickery.com\/?p=86"},"modified":"2025-07-03T21:26:47","modified_gmt":"2025-07-03T21:26:47","slug":"information-theory-what-even-is-information-1-n","status":"publish","type":"post","link":"http:\/\/theoreticaltrickery.com\/index.php\/2023\/06\/25\/information-theory-what-even-is-information-1-n\/","title":{"rendered":"Information Theory: What even is information? 1\/n"},"content":{"rendered":"\n

Mass, momentum, and energy are some of the most fundamental properties of matter in our universe. But there is another property that might arguably be on par with these: information.<\/p>\n\n\n\n

But what even is information? Somehow, it feels like a very intuitive concept\u2014more of a \u201cI know it when I see it\u201d<\/em> kind of thing. However, in 1948 (not 1984), Shannon published the seminal paper \u201cA Mathematical Theory of Communication.\u201d<\/em> In it, he showed that not only is it possible to rigorously define what we mean by information, but it is also a measurable quantity!<\/p>\n\n\n\n

So, what is the definition of information?<\/p>\n\n\n\n

Well, information is clearly something different from simply having data. Even if I give you a map, you still might not find your way to my place without me providing an extra piece of information (assuming you haven\u2019t been there before or don\u2019t have an exceptionally good memory).<\/p>\n\n\n\n

A unit of information, then, is in some sense something that allows you to narrow down the possible options within your data. More precisely:<\/p>\n\n\n\n

\n

\u201cOne bit of information is the amount of information required to choose between two equally probable alternatives.\u201d<\/p>\n<\/blockquote>\n\n\n\n

Confusingly, information is measured in units called bits<\/em>. However, one bit of information is not the same as a binary number. A binary number can represent information. For example, if I tell you which left and right turns you need to take to reach my place, you could neatly describe it as a binary sequence, say: 10011010<\/strong> (for R, L, L, R, R, L, R, L). This sequence contains 8 bits of information.<\/p>\n\n\n\n

But here\u2019s the catch: if you already know the way to my place, this binary number doesn\u2019t represent<\/em> information for you\u2014it doesn\u2019t reduce the number of possible routes you might consider. Information, in this sense, is tied to cutting down uncertainty.<\/p>\n\n\n\n

—————————————————————————————–<\/p>\n\n\n\n

To further our understanding of the concept of information, let\u2019s consider the following example. Suppose we are given an image\u2014how much information does it contain?<\/p>\n\n\n\n

For simplicity, let\u2019s consider the image in grayscale.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

This image is 512\u00d7512 pixels with a grayscale of 256 values. In other words, every pixel represents one byte (2\u2078 bits) of information. At this point, you might say to yourself, \u201cWell, this is silly\u2014clearly this image contains exactly 67,108,864 bits of information!\u201d<\/em><\/p>\n\n\n\n

That seems like a reasonable guess. The only problem is that it\u2019s completely wrong.<\/p>\n\n\n\n

But why is it wrong? Remember our definition: 1 bit of information allows us to choose between two equally likely alternatives. Clearly, however, the values of different pixels are not independent of each other (that would make for a rather boring picture). So there must be some redundancy in the description of our image. If we remove all of this redundancy, what we\u2019re left with is the pure information<\/strong> of the image\u2014the minimum amount of bits needed to recreate it without any losses.<\/p>\n\n\n\n

So how do we find this minimal amount of information needed to represent an image? We\u2019ll delve a bit deeper into that in an upcoming post.<\/p>\n","protected":false},"excerpt":{"rendered":"

Mass, momentum, and energy are some of the most fundamental properties of matter in our universe. But there is another property that might arguably be on par with these: information. But what even is information? Somehow, it feels like a … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-86","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/posts\/86","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/comments?post=86"}],"version-history":[{"count":4,"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/posts\/86\/revisions"}],"predecessor-version":[{"id":127,"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/posts\/86\/revisions\/127"}],"wp:attachment":[{"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/media?parent=86"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/categories?post=86"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/theoreticaltrickery.com\/index.php\/wp-json\/wp\/v2\/tags?post=86"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}