A coin toss offers a 50-50 chance of either result. Image by ICMA Photos
This article introduces basic mathematical concepts in probability. Future articles will discuss different aspects, including several paradoxical situations involving probabilities. For those who can’t wait, Solve the Monty Hall Problem using Logic and Mathematics.
Probability, Statistics or Likelihood?
In mathematics, “probability” is the study of how likely it is for some specific outcome to occur as the result of an event. This is expressed either as a percentage from zero to 100%, or as a number between 0.0 and 1.0 inclusive.
“Statistics” is the analysis of the events ruled by probabilities. Often the statistician is concerned with the distribution of probabilities. In the far future, we may delve into statistics.
A related word is “likelihood“, which refers to the odds of the outcomes of past events. By contrast, “probability” refers to future events.
Some examples might help:
- If you plan to flip a coin to make a decision, the probability of the “heads” outcome is 0.5, or 50%. Afterward, when you explain why you watched a “Headhunter” movie rather than playing “Pin the Tail on the Donkey”, you would say the “likelihood” of either activity was 50%. From the statistical point of view, a coin toss is well modeled as a Bournoulli distribution with probability parameter of 0.5.
It seems likely that the first people to study probability were motivated by gambling. This particular article will not prepare anyone to win at a casino. However, the foundation to avoid losing is to realize that the professionals already know the odds.
Discrete and Continuous Probability
The act of tossing a coin for a “head” or “tail” is an example of a discrete probability distribution. Other discrete outcomes may be generated by rolling dice or selecting playing cards or domino tiles.
Although a coin toss has only two outcomes, other discrete distributions might have a countably infinite number of outcomes. (A “countably infinite” set can be placed in a one-to-one relationship with positive integers.) A simple example of a discrete infinite distribution would model the probability of finding “runs” of three heads in a row while repeatedly tossing a coin.
Continuous probability distributions also exist. Examples include predicting how long a light bulb might shine before it wears out, or the amount of rainfall collected at a location over a given period of time.
The first articles will only deal with discrete probabilities.
Defining a Sample Space
Dice present a finite number of results. Image by fyuryu
We need some more definitions before making progress. These are simplified, rather than properly formal.
- Event: One “event” gives only one outcome: an example is a single roll of the dice. Often we are interested in a set of events, such as rolling dice five times to determine which combinations of numbers are seen most frequently.
- Sample Space: The “sample space” of such a set of events is itself a set, containing all the possible outcomes. One says that an event is in a sample space.
The Probability Axioms
Mathematicians like to begin with axioms: statements that seem true and upon which a self-consistent mathematical topic can be constructed. Here, we begin with the Probability Axioms.
Define a discrete sample space, S, with N elements. ‘N‘ could be a finite number or a countably infinity, such as the cardinality of the set of integers.
We then say that the jth event E[j] is in S, and that S is the union of the E[j] for all j from one to N. Each event is considered a set, and the whole sample space is also a set.
There is also a probability, P(E[j]), for each event. As well, there is a probability, P(S), for the whole sample space. A computer programmer would consider ‘P()’ to be a function with a set as the input, that returns the numeric value of the probability.
The Probability Axioms are:
- Zero <= P(E[j]) <= 1 for each j. Each event has a probability in the range from zero to 1, inclusive.
- P(S) = 1. All the possible results are included in S, and it is certain that one of those results will occur.
- P(E[i] U E[j]) = P(E[i]) + P(E[j]) if (E[i]) and P(E[j]) are mutually exclusive. The probability of the union of two exclusive events equals the sum of the probabilities of each of those separate events.
- Extend the third axiom to any group of mutually exclusive events, by saying that the probability of the union of a group of mutually exclusive events equals the sum of probabilities of each of those separate events.
Events are “mutually exclusive” if the intersection set of the events is the empty set.
Consider the sample space “S1” of two coin throws, where we keep track of the sequence. For example, “HT” means that the first toss comes up heads, and the second is tails. The sample space is {HH, HT, TH, TT}.
The probability of each outcome is 0.25, the results are exclusive, and the four probability axioms are satisfied:
- Each event has P(E) = 0.25, and zero < 0.25 < 1.
- After two coin tosses, the outcome is one of the elements in the sample space, so P(S1) = 1.
- Each outcome excludes the others. The probability of the union of two events, say {HH, TT} equals the sum of the probabilities of either alone.
Generally we also have to mention that we are using a “fair coin”, so it is not biased either for heads nor tails, and that we exclude the coin landing on edge or exploding in mid-air.
We could have a different sample space if we did not track the sequence. S2 = {(two heads), (two tails), (one head and one tail in either order)} which could be represented as, say, the number of heads in two tosses : {2, 0, 1}. In sample space S2, the probability of (one head and one tail, in either order) is 0.5. Again, sample space S2 satisfies the Probability Axioms.
The Lottery Axioms, Including the Independence Axiom
Lottery drawings are probability events. Image by Jeffrey Beall
Although the usual “lottery” is a draw rather than a coin toss, it really is just a probability event with the distinction that there is a payoff structure and the observer will prefer some outcomes over others.
In a lottery, the preference for outcome ‘X’ over outcome ‘Y’ is shown as “X > Y”. Indifferent outcomes, such as having equal payouts, are shown as “X ~ Y”.
The Lottery Axioms are similar to the Probability Axioms:
- For all outcomes X and Y, there is a relationship “X > Y” or “Y > X” or “X ~ Y”. This is the “completeness” axiom.
- For all outcomes X, Y and Z, if X > Y and Y > Z, then X > Z. This is the “transitive” axiom. It should probably add that “if X ~ Y and Y ~ Z, then X ~ Z”.
- For all outcomes X, Y and Z, if X > Y and Y > Z, then there is a unique numeric value for ‘p’ such that p*X + (1-p)*Z ~ Y. This “continuity” axiom should, in my opinion, add that 0 < p < 1 in the Real numbers.
- For all outcomes X, Y and Z, if X > Y, then for all ‘p’ where 0 <= p <= 1, p*X + (1-p)*Z > p*Y + (1-p)*Z . This is the “independence” axiom.
In ordinary English, “independent” events are events where one does not “depend on”, or influence, the other. For example, your coin toss is independent of a poker hand dealt by a stranger in another city.
In mathematics, events are independent if the probability of both happening equals the product of their separate probabilities. This is expressed in the equation P(A.B) = P(A)*P(B).
What do these axioms mean for a lottery? Either one outcome is preferable to another, or there is no preference; but the gambler can decide on a preference for each and every outcome.
If preferences were not transitive, there might be no “preferred” outcome out of three. By contrast, the “rock, paper, scissors” game is nottransitive: no one choice is best. Transitive lotteries allow gamblers to make consistent choices.
When playing cards, understanding probability offers an advantage. Image by fdecomite
Suppose the “house” is running three lotteries, and a gambler has the preference X > Y > Z. Then “Continuity” ensures that the “house” could adjust the payoffs for X and Z such that the gambler is indifferent to playing Y or to playing the adjusted mix of X and Z.
Independence means that, if X > Y, then adjusting them equally and adding a third lottery equally to each does not affect the original preference. While this may be sound mathematics, human behaviour can be more complex as will be seen in a future article featuring the Allais Paradox.
An easy lottery example would be drawing from a standard pack of playing cards, with the payout equal to the face value on the card. The independence axiom might add a bonus for the number shown on the roll of two dice.
The Probability of a Sequel
This article is lengthy in its original form, but check this paragraph for links to future articles in this series.
References:
Pegg, Ed Jr. Independence Axiom. MathWorld–A Wolfram Web Resource. (1999-2011). Accessed December 3, 2011.
Weisstein, Eric W. Coin Tossing, Independent Events, Probability, Probability Axioms, Sample Space. MathWorld–A Wolfram Web Resource. Accessed December 3, 2011.
[...] Dec. 2nd, I launched a new math series at Decoded Science. “Introducing Probability Theory without Statistics” starts with some definitions and axioms for both probabilities and [...]
[...] between mathematics and human behaviour is shown in the “Allais Paradox.” Compared to probability theory, in the Allais Paradox, people choose correctly or incorrectly based on irrelevant [...]
[...] first article in this series, Introducing Probability Theory without Statistics, noted that probability distribution might be “discrete” or [...]