Information and Meaning<\/a> if you need more clarification).<\/p>\nIt was this realization that\u00a0led mathematician Claude Shannon to develop an entire theory of information. \u00a0Shannon was working for Bell Laboratories in the mid 20th century on a project to solve an engineering problem about reliable communications on noisy telephone lines. He not only solved that problem, but in the process ended up founding the modern science of information theory on which we have based all our modern communications and computing technology. But Shannon’s theory reaches beyond engineering to go deep into quantifying information as something as fundamental to the universe as energy or matter. \u00a0It would not be wrong to think of Claude Shannon as the “Isaac Newton” of information. \u00a0Where Newton developed a “System of the World” for motion all throughout the universe, Shannon has done the same for information.<\/p>\n
Information Reduces Uncertainty By Answering a Question<\/h3>\n
Shannon’s theory is based on just a few simple concepts.\u00a0First, the function of information is to inform,. \u00a0Shannon says that we are informed to the degree that our\u00a0uncertainty about something is reduced. Uncertainty is not knowing the answer to a question. Information reduces uncertainty by answering a question. \u00a0Think of uncertainty as a hole, and information is that which reduces the size of that hole.<\/p>\n
Uncertainty is Related to the Number of Possible Answers to a Question.<\/h3>\n
Complex questions can be broken down further into simpler sub-questions until you reach the point where the simplest questions have only a certain number of possible answers. \u00a0Let’s call that number N. For example, suppose you wanted to know the exact date that the first shot was fired in the Civil War. \u00a0That question can be broken down to separate questions about the day of the week, the month, day of the month, and the year. \u00a0The number of possible answers for the day-of-week question is seven, so in this case N = 7. \u00a0 Suppose you only have a vague idea for when the war began. \u00a0Let’s say that for the day-of-week question, you are completely clueless. \u00a0Although there are only seven possible answers, they are all equally probable as far as you are concerned. This represents the maximum uncertainty for this question for this case where the number of possible answers is N = 7.<\/p>\n
Although you are maximally uncertain about the week or the month the shot was fired, Shannon would say that the uncertainty about the month is higher than the uncertainty about the week, because\u00a0any particular month has a one in twelve chance of being the answer, whereas any particular week has a only one in seven chance of being the answer. Where all answers are equally probable, the question with the highest N has the highest uncertainty.\u00a0[!fnYou can get a better feel for this by considering the extreme case where one question is was it day or night that the shot was fired (N = 2), vs what was the name of the person who fired the shot (N = some very large number). \u00a0As you might be completely clueless about the answer to either question, you can understand why the uncertainty about the name of the shooter is far higher than the day or night answer.\/fn]<\/p>\n
The Possible Answers Are Represented by a Set of Symbols That Make Up a Code.<\/h3>\n
Shannon says that if each question has a particular number of possible answers (N), the answers to a particular question can be represented by a set of N unique symbols. \u00a0It doesn’t matter what the symbols are as long as there are N symbols that can be distinguished from each other by both the sender and the receiver of information.\u00a0\u00a0The sender needs to be able to select and send a symbol that the receiver can associate with one of the possible answers. \u00a0The set of N symbols that represent the answers to a question is called a Code. [!fnFor example, one code would be English words. \u00a0The list of symbols in the Code for the day and night question could simply be [daytime, \u00a0nighttime]. \u00a0Similarly, days of the week could be the usual English [Mon, Tue, …. , Fri] and so forth. \u00a0 The Code for the month might be [Jan, Feb, Mar, …, Dec].\/fn]<\/p>\n
The Smallest Code With Information has Two Symbols<\/h3>\n
Consider a question that has only one answer, where N = 1.. \u00a0Your uncertainty about that answer is zero, so a message containing the one symbol in the code will not reduce your uncertainty. \u00a0A question with two answers, however, has the possibility of uncertainty, such as the daytime-nighttime question above. \u00a0Any information carrying code has to have at least two symbols.<\/p>\n
Information Can Be Measured in Bits<\/h3>\n
You can represent the symbols in a code by numbering them from 0 to N-1. \u00a0The Code for [Daytime, Nighttime], can be represented by [0, 1], for example. \u00a0If you encoded that in binary bits, you would only need one bit since one bit can be either 0 or 1. The answer to that question can be sent with one binary bit set to either 0\u00a0or 1. \u00a0A question that has more possible answers needs more bits. \u00a0A Code for the day of the week needs seven symbols that could be numbered 0 through 6. \u00a0That range of numbers would require 3 bits [!fn\u00a0Three binary bits can encode the numbers 0 through 7 with the following sequences [000, 001, 010, 011, 100, 101, 110, 111]\/fn]. \u00a0The general rule is that where the number of symbols in the code is N, the number of bits is \u00a0I = ln2(N) [!fn Ln2(N) is the log base 2 of the number \/fn]. This is the average amount of information\u00a0in a symbol in a Code of N symbols.<\/p>\n
The Possible Answers Are Not Always Equally Probable<\/h3>\n
In some cases the receiver might not consider all possible answers as equally probable. Consider a weather application that is telling you the current temperature.\u00a0 Since you are fully aware of the season or can see that it is snowing outside, all possible temperature messages from the weather service to you are not equally probable, as temperatures near freezing in this particular case are far more probable than a temperature of 90%.\u00a0\u00a0A weather application could exploit this by knowing the average daily temperature for each day of the year averaged over the last five years, and send only the differences from that average from the service to your phone.\u00a0 The code could use fewer bits for smaller differences coming closer to an “optimum” code.<\/p>\n
This is much more than an engineering exercise that saves you money on your phone’s data plan.\u00a0 It actually represents the amount of information carried by each symbol in the Code. \u00a0Since your expectation\u00a0increases for values closer to the average daily temperature, receiving one of those values has less influence on your already low uncertainty.\u00a0\u00a0 This is consistent with what was stated above, that the amount of information carried by a symbol in a Code is directly related to how much uncertainty it reduces in the receiver.<\/p>\n
Not All Symbols Carry Information<\/h3>\n
Consider two different messages, with one saying “Last night’s lowest temperature was 10 degrees F”, \u00a0or “Last night’s lowest temperature was a frosty<\/strong> 10 degrees F.” \u00a0 The addition of the word “frosty” adds no additional information to a human receiver. \u00a0If the person already has no uncertainty about what 10 degrees feels like, the word “frosty” does not reduce it further, so it contains no information. \u00a0A Code could be very inefficient that way using far more bits than the minimum required to reduce the uncertainty.<\/p>\nInformation Always Comes in a Message\u00a0Containing Symbols Selected From Codes.<\/h3>\n
Although English words are used for both pieces of information,”daytime,Tuesday” would be a message that answers two questions, with two Codes. The first Code is a set of two possible symbols, followed by a second Code with seven possible symbols. \u00a0Shannon says that information is always carried in a message that contains one or more symbols that are part of one or more Codes answering one or more questions.<\/p>\n
Information is a Material Thing<\/h3>\n
This might come as a surprise to the reader, but the consequences of all of this is that no information moves in the world without involving something to do with energy or matter. \u00a0There is no way you can send a symbol to me without using a pulse of energy or moving some matter somewhere. \u00a0Bits are more than just a mathematical notion. \u00a0You cannot transmit of store even one bit without storing it as energy or as a piece of matter. \u00a0If this were not true, you would never need\u00a0more computer memory or disk space and you would not have to pay your Internet provider\u00a0a usage fee for each megabyte of data you sent or received.<\/p>\n
This became such a powerful fundamental observation that it led to solving such problems as how many extra bits you might need to use to overcome random noise in a communications channel or in imperfect persistence in computer memory. \u00a0It even led to linking Shannon Information to other fundamental properties in the universe such as thermodynamics.<\/p>\n
Information Does Not Require an Intelligent Sender or Receiver<\/h3>\n
Here is the most dramatic outcome from Shannon’s theory. \u00a0Consider that if Shannon Information is not meaning but symbolizes meaning, and that it informs by reducing the receiver’s uncertainty as to which symbol in a Code was selected by the sender, and if it can specified, stored, and transmitted in a certain number of bits, then Shannon Information is not something that only pertains \u00a0to human communications. It turns out that Shannon Information is everywhere in nature. \u00a0We can find it, isolate it, and quantify it just like we can do for a message between two humans. In Shannon’s Information Theory, there is no requirement for an intelligent sender or an intelligent receiver of information. \u00a0In fact, naturally produced information is flowing all throughout the universe.<\/p>\n","protected":false},"excerpt":{"rendered":"
Shannon’s Theory of Information Information is Not Meaning. \u00a0It Symbolizes Meaning. What does it mean to have a theory of information? How can information be studied, categorized or quantified? How can we measure the information in a Shakespeare play or a sad message you receive about the death of a close friend? The answer starts […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"page_like_blog.php","format":"standard","meta":[],"categories":[],"tags":[],"_links":{"self":[{"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/posts\/112"}],"collection":[{"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/comments?post=112"}],"version-history":[{"count":1,"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/posts\/112\/revisions"}],"predecessor-version":[{"id":187,"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/posts\/112\/revisions\/187"}],"wp:attachment":[{"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/media?parent=112"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/categories?post=112"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/theappleandthefinch.com\/wp-json\/wp\/v2\/tags?post=112"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}