text:
The Cat Geek
I use technology to annoy my cats
Python, Udacity Class
Udacity Classes 5: Design of Computer Programs
April 11, 2020 March 7, 2020 thecatgeek
Below are my notes from when I originally took the Udacity course Design of Computer Programs. At that time, I only took notes in Microsoft Word, so there was a real lack of gifs. I'd go through and add some, but this is going to be a HUGE post and WordPress is really struggling to function. Honestly, I'm only adding this post for my own reference and to save my notes, so if I were you I'd go watch some Schitts Creek instead or take the class yourself! 🙂
Design of Computer Programs
The key to progressing from a novice programmer to an expert is mindful practice. In this class you will practice going from a problem description to a solution, using a series of assignments. With each problem you will learn new concepts, patterns, and methods that will expand your ability and help move you along the path from novice towards expertise.
Winning Poker Hands
Steps of the design process; Developing for clarity and generality; Arguments for program correctness; Experimentation and simulation.; Design tradeoffs; Simplicity and Clarity. Decomposition and composability. TRANSCRIPT
Back of envelope calculations; When to use brute force and when to be clever; The Zebra puzzle; Generator expressions; Permutations and combinations. Cryptarithmetic; Recursive and wishful thinking. TRANSCRIPT
Regular Expressions, other languages and interpreters
Defining the language of regular expressions; Interpreting the language; Defining the set of strings matched by a regular expression; Other languages. TRANSCRIPT
Dealing with complexity through search
Search: finding your way with a flashlight or boat; pouring water. Analyzing the efficiency of an algorithm; Recurrence relations; Matching data types with algorithms; Majority algorithm; Longest palindrome substring algorithm. TRANSCRIPT
Dealing with uncertainty through probability
Probability: the game of Pig; Maximizing expected utility to optimize strategy. TRANSCRIPT
Managing complexity; Large sets of words; Appropriate data structures; Word games. TRANSCRIPT
Final exam/project TRANSCRIPT
UNIT 1 TRANSCRIPT
1. Outlining the Problem
Unit1- 2
Writing a poker program is an example of a general process with three steps; understand, specify and design. The process involves starting with a vague understanding that you refine into a formal specification of a problem. You then further specify your understanding into something that is amenable to being coded. Then, after the design process you end up with working code.
Step 1: Understand
Start with a vague understanding that you refine into a problem. In this step you want to take inventory of the concepts you are dealing with. With respect to writing a poker problem, begin with the notion of a "hand," which is five cards. Additionally, note that each card has a "rank" and a "suit."
For example, a five of diamonds card has a rank of five and the suit is diamonds.
Another concept to identify is the "hand rank," which takes a hand and maps to details about the hand.
Step 2: Specify
Specify how this problem can be made amenable to being coded. The main program you are trying to specify is called poker and it takes a list of hands as input and outputs the best hand. The best hands are described here:
http://en.wikipedia.org/wiki/List_of_poker_hands
These rules dictate which hands beat which, that is, how each hand ranks relative to one another.
There are three concepts that make up the hand rank:
Kind means that there are cards of the same rank. "n-kind," where n can be one of a kind, two of a kind or three of a kind.
Straight means that there are five consecutive ranks and the suit does not matter. For example: a five of clubs, a six of spades, a seven of diamonds, an eight of spades and a nine of hearts. This is a straight because the rank of the cards is sequential.
Flush means that all of the cards are the same suit and the ranks do not matter.
For example: a ten of diamonds, a seven of diamonds, five of diamonds, four of diamonds and two of diamonds.
Now you know about the types of data you are dealing with: hands, cards, ranks and suits. You also now know about the functions for them: n-kind, straight, flush. From here you can move on to step 3, the design phase.
Step 3: Design working code
That's what this course is all about!
1.1. Quiz: Representing Hands
Unit1-3
Which of the possible representations for a hand makes the most sense. There may be more than one.
a. ['JS', 'JD', '2S', '2C', '7H']
b. [(11, 'S'), (11, 'D'), (2, 'S'), (2, 'C'), (7, 'H')]
c. set (['JJ', 'JD', '2S', '2C', '7H')])
d. "JS JD 2S 2C 7H"
2. Wild West Poker
Unit1-11
A poker player will call out his hand when he has to reveal it. With this information you will know how to rank and assign the hand.
"Straight flush, Jack high!"
This is all you have to know about the hand; a straight flush and the highest ranking-Jack.
Here is a ranking table of the possible hands:
0- High Card
1- One Pair
2- Two Pair
3- Three of a Kind
4- Straight
5- Flush
6- Full House
7- Four of a Kind
8- Straight Flush
Use the numbers associated with the hand when writing your program.
This hand can be written:
Toggle line numbers
1 (8, 11)
Here the eight stands for straight flush and the number 11 stands for the Jack.
Here are a few examples of what a player might say and how you would write their hand:
Four aces and a queen kicker?
1 (7, 14, 12)
Notice in this case, and when you have one pair, two pair, three of a kind and four of a kind, that there is an extra string in the set that tells you what kind you have.
Full house, eights over kings?
1 (6, 8, 13)
Even though a king is higher than an eight, because there are three eights in this full house, the eight is what matters most so it is included first, followed by 13, which indicates the kings.
Flush, 10-8!
Usually, declaring the two highest cards in the flush is enough to distinguish the hand from all other flushes. However, you might need all of the cards in the hand to break a tie, although it is unlikely that someone else would have the same exact hand.
When you write this, start with the highest card and work your way down.
1 (5, [10, 8, 7, 5, 3])
Straight, Jack high
This is all you need to know, because if Jack is the high card, than the rest of the cards in the hand, since it is a straight, have to be ten, nine, eight and seven.
Three sevens!
Usually, this is enough information to distinguish a hand, but if you really need to break the ties, you can write the complete list of the five cards.
1 (3, 7, [7, 7, 7, 5, 2])
Two pairs, Jacks and threes
While this describes most of the hand, you also need to compare all of the cards, including the two pairs. Write the highest ranking pair first and include a set of the entire hand, just in case there is a tie – if someone else has a hand with two pairs, Jacks and threes.
1 (2, 11, 3, [13, 11, 11, 3, 3])
Pair of twos, Jack high
This partially describes a hand, but your player is not impressed with this hand. Here is how you represent this dismally dealt hand, make sure to include the set of the entire hand as a pair of two does not completely disambiguate this hand:
1 (1, 2, [11, 6, 3, 2, 2])
Got nothing
Sometimes, nobody gets dealt a good hand. How do you decide who wins? Go in order of the ranks of the cards:
1 (0, 7, 5, 4, 3, 2)
2.1. Quiz: Poker Function
Out of the list of hands, you want poker to return the highest -ranking hand. Do you know of a built in function in Python that will allow you to return the highest-ranking item from a list?
1 def poker(hands):
2 "return the best hand: poker([hand,…]) => hand"
3 return ???
2.2. Quiz: Understanding Max
What will the two max calls return?
2 "Return the best hand: poker ([hand, …]) => hand"
3 return max
2.3. Quiz: Using Max
Assume that you have defined a function hand_rank, which takes a hand as input and returns some sort of a rank. Given this, how would you write the definition of the function poker to return the maximum hand according to the highest ranked?
2 "Return the best hand: poker([hand, …]) => hand"
5 def hand_rank(hand):
8 print max([3, 4, 5, 0]), max ([3, 4, -5, 0], key = abs)
2.4. Quiz: Testing
Modify the test() function to include two new test cases:
1) four of a kind (fk) vs. full house (fh) returns fk.
2) full house (fh) vs. full house (fh) returns fh.
3 return max(hands, key=hand_rank)
5 def test():
6 "Test cases for the functions in poker program"
7 sf = "6C 7C 8C 9C TC".split() # => ['6C', '7C', '8C', '9C', 'TC']
8 fk = "9D 9H 9S 9C 7D".split()
9 fh = "TD TC TH 7C 7D".split()
10 assert poker([sf, fk, fh]) == sf
11 assert poker([fk, fh]) == fk
12 assert poker([fh, fh]) == fh
14 # Add 2 new assert statements here. The first
15 # should check that when fk plays fh, fk
16 # is the winner. The second should confirm that
17 # fh playing against fh returns fh.
19 print test()
2.5. Quiz: Extreme Values
2.6. Quiz: Hand Rank Attempt
2.7. Quiz: Representing Rank
3. Back to Hand Rank
3.1. Quiz: Testing Hand Rank
3.2. Quiz: Writing Hand Rank
3.3. Quiz: Testing Card Rank
3.4. Quiz: Fixing Card Rank
3.5. Quiz: Straight and Flush
3.6. Quiz: Kind Function
3.7. Quiz: Two Pair Function
3.8. Quiz: Making Changes
3.9. Quiz: What to Change
3.10. Ace Low Straight
3.11. Quiz: Handling Ties
3.12. Quiz: Allmax
4. Deal
4.1. Quiz: Hand Frequencies
This is the procedure Peter gave us to calculate hand frequencies:
1 def hand_percentages(n=700*1000):
2 counts = [0]*9
3 for i in range(n/10):
4 for hand in deal(10):
5 ranking = hand_rank(hand)[0]
6 counts[ranking] += 1
7 for i in reversed(range(9)):
8 print "%15s: %6.3f %%" % (hand_names[i], 100.*counts[i]/n)
5. Dimensions of Programming
6. Refactoring
The two alternative versions of hand_rank that Peter gave in the refactoring class are:
1 def hand_rank_alt(hand):
2 "Return a value indicating how high the hand ranks."
3 # count is the count of each rank; ranks lists corresponding ranks
4 # E.g. '7 T 7 9 7' => counts = (3, 1, 1) ranks = (7, 10, 9)
5 groups = group(['–23456789TJQKA'.index(r) for r,s in hand])
6 counts, ranks = unzip(groups)
7 if ranks == (14, 5, 4, 3, 2): # Ace low straight
8 ranks = (5, 4, 3, 2, 1)
9 straight = len(ranks) == 5 and max(ranks) – min(ranks) == 4
10 flush = len(set([s for r,s in hand])) == 1
11 return (9 if (5,) == counts else
12 8 if straight and flush else
13 7 if (4, 1) == counts else
15 5 if flush else
16 4 if straight else
17 3 if (3, 1, 1) == counts else
19 1 if (2, 1, 1, 1) == counts else
20 0), ranks
22 def group(items):
23 "Return a list of [(count, x), …], highest count first, then highest x first"
24 groups = [(items.count(x), x) for x in set(items)]
25 return sorted(groups, reverse = True)
27 def unzip(iterable):
28 "Return a tuple of lists from a list of tuples : e.g. [(2, 9), (2, 7)] => ([2, 2], [9, 7])"
29 return zip(*iterable)
The table-based lookup version:
1 count_rankings = {(5,): 10, (4, 1): 7, (3, 2): 6, (3, 1, 1): 3, (2, 2, 1): 2,
2 (2, 1, 1, 1): 1, (1, 1, 1, 1, 1): 0}
4 def hand_rank_table(hand):
10 if ranks == (14, 5, 4, 3, 2): # Ace low straight
11 ranks = (5, 4, 3, 2, 1)
12 straight = len(ranks) == 5 and max(ranks) – min(ranks) == 4
14 return max(count_rankings[counts], 4*straight + 5*flush), ranks
8. Bonus – Shuffling
Bad Shuffle
Shuffle Runtime
Good Shuffle
Is it Random
Testing Shuffles
Comparing Shuffles
Computing or Doing
The shuffling procedures from the bonus videos are:
1 def shuffle1(p):
2 n = len(p)
3 swapped = [False]*n
4 while not all(swapped):
5 i, j = random.randrange(n), random.randrange(n)
6 swap(p, i, j)
7 swapped[i] = swapped[j] = True
10 n = len(p)
11 swapped = [False]*n
12 while not all(swapped):
13 i, j = random.randrange(n), random.randrange(n)
14 swap(p, i, j)
15 swapped[i] = True
17 def shuffle3(p):
19 for i in range(n):
20 swap(p, i, random.randrange(n))
22 def knuth(p):
24 for i in range(n-1):
25 swap(p, i, random.randrange(i, n))
26 def swap(p, i, j):
27 p[i], p[j] = p[j], p[i]
The procedures for testing the different shuffles were:
1 def test_shuffle(shuffler, deck = 'abcd', n = 10000):
2 counts = defaultdict(int)
3 for _ in range(n):
4 input = list(deck)
5 shuffler(input)
6 counts[".join(input)] += 1
7 e = n * 1./factorial(len(deck))
8 ok = all((0.9 <= counts[item]/e <= 1.1) for item in counts)
9 name = shuffler.__name__
10 print '%s(%s) %s' % (name, deck, ('ok' if ok else '*** BAD ***'))
11 print ' ',
12 for item, count in sorted(counts.items()):
13 print "%s:%4.1f" % (item, count * 100. / n),
14 print
16 def test_shufflers(shufflers = [knuth, shuffle1, shuffle2, shuffle3], decks = ['abc', 'ab']):
17 for deck in decks:
19 for f in shufflers:
20 test_shuffle(f, deck)
21 def factorial(n):
22 return 1 if n<= 1 else n * factorial(n-1)
9. Complete Code For Poker Problem
The complete code given by Peter in this unit including some additional test cases:
1 #! /usr/bin/env python
3 import random
6 "Return a list of winning hands: poker([hand,…]) => [hand,…]"
7 return allmax(hands, key=hand_rank)
9 def allmax(iterable, key=None):
10 "Return a list of all items equal to the max of the iterable."
11 iterable.sort(key=key,reverse=True)
12 result = [iterable[0]]
13 maxValue = key(iterable[0]) if key else iterable[0]
14 for value in iterable[1:]:
15 v = key(value) if key else value
16 if v == maxValue: result.append(value)
17 else: break
18 return result
20 def card_ranks(hand):
21 "Return a list of the ranks, sorted with higher first."
22 ranks = ['–23456789TJQKA'.index(r) for r, s in hand]
23 ranks.sort(reverse = True)
24 return [5, 4, 3, 2, 1] if (ranks == [14, 5, 4, 3, 2]) else ranks
26 def flush(hand):
27 "Return True if all the cards have the same suit."
28 suits = [s for r,s in hand]
29 return len(set(suits)) == 1
31 def straight(ranks):
32 "Return True if the ordered ranks form a 5-card straight."
33 return (max(ranks)-min(ranks) == 4) and len(set(ranks)) == 5
35 def kind(n, ranks):
36 """Return the first rank that this hand has exactly n-of-a-kind of.
37 Return None if there is no n-of-a-kind in the hand."""
38 for r in ranks:
39 if ranks.count(r) == n: return r
40 return None
42 def two_pair(ranks):
43 "If there are two pair here, return the two ranks of the two pairs, else None."
44 pair = kind(2, ranks)
45 lowpair = kind(2, list(reversed(ranks)))
46 if pair and lowpair != pair:
47 return (pair, lowpair)
48 else:
52 def hand_rank(hand):
53 "Return a value indicating the ranking of a hand."
54 ranks = card_ranks(hand)
55 if straight(ranks) and flush(hand):
56 return (8, max(ranks))
57 elif kind(4, ranks):
58 return (7, kind(4, ranks), kind(1, ranks))
59 elif kind(3, ranks) and kind(2, ranks):
61 elif flush(hand):
62 return (5, ranks)
63 elif straight(ranks):
66 return (3, kind(3, ranks), ranks)
67 elif two_pair(ranks):
68 return (2, two_pair(ranks), ranks)
74 def hand_rank_alt(hand):
75 "Return a value indicating how high the hand ranks."
76 # count is the count of each rank; ranks lists corresponding ranks
77 # E.g. '7 T 7 9 7' => counts = (3, 1, 1) ranks = (7, 10, 9)
78 groups = group(['–23456789TJQKA'.index(r) for r,s in hand])
79 counts, ranks = unzip(groups)
96 count_rankings = {(5,): 10, (4, 1): 7, (3, 2): 6, (3, 1, 1): 3, (2, 2, 1): 2,
97 (2, 1, 1, 1): 1, (1, 1, 1, 1, 1): 0}
99 def hand_rank_table(hand):
100 "Return a value indicating how high the hand ranks."
101 # count is the count of each rank; ranks lists corresponding ranks
102 # E.g. '7 T 7 9 7' => counts = (3, 1, 1) ranks = (7, 10, 9)
103 groups = group(['–23456789TJQKA'.index(r) for r,s in hand])
104 counts, ranks = unzip(groups)
105 if ranks == (14, 5, 4, 3, 2): # Ace low straight
106 ranks = (5, 4, 3, 2, 1)
107 straight = len(ranks) == 5 and max(ranks) – min(ranks) == 4
108 flush = len(set([s for r,s in hand])) == 1
109 return max(count_rankings[counts], 4*straight + 5*flush), ranks
111 def group(items):
112 "Return a list of [(count, x), …], highest count first, then highest x first"
113 groups = [(items.count(x), x) for x in set(items)]
114 return sorted(groups, reverse = True)
116 def unzip(iterable):
117 "Return a list of tuples from a list of tuples : e.g. [(2, 9), (2, 7)] => [(2, 2), (9, 7)]"
118 return zip(*iterable)
120 mydeck = [r+s for r in '23456789TJQKA' for s in 'SHDC']
122 def deal(numhands, n=5, deck=mydeck):
123 random.shuffle(mydeck)
124 return [mydeck[n*i:n*(i+1)] for i in range(numhands)]
127 hand_names = ["Straight flush", "Four of a kind", "Full house", "Flush", "Straight",
128 "Three of a kind", "Two pair", "One pair", "High card"]
130 def hand_percentages(n=700*1000):
131 counts = [0]*9
132 for i in range(n/10):
133 for hand in deal(10):
134 ranking = hand_rank(hand)[0]
135 counts[ranking] += 1
136 for i in reversed(range(9)):
137 print "%15s: %6.3f %%" % (hand_names[i], 100.*counts[i]/n)
139 def test():
140 "Test cases for the functions in poker program."
141 sf1 = "6C 7C 8C 9C TC".split() # Straight Flush
142 sf2 = "6D 7D 8D 9D TD".split() # Straight Flush
143 fk = "9D 9H 9S 9C 7D".split() # Four of a Kind
144 fh = "TD TC TH 7C 7D".split() # Full House
145 tp = "5D 2C 2H 9H 5C".split() # Two Pair
147 # Testing allmax
148 assert allmax([2,4,7,5,1]) == [7]
149 assert allmax([2,4,7,5,7]) == [7,7]
150 assert allmax([2]) == [2]
151 assert allmax([0,0,0]) == [0,0,0]
153 # Testing card_ranks
154 assert card_ranks(sf1) == [10, 9, 8, 7, 6]
155 assert card_ranks(fk) == [9, 9, 9, 9, 7]
156 assert card_ranks(fh) == [10, 10, 10, 7, 7]
158 # Testing flush
159 assert flush([]) == False
160 assert flush(sf1) == True
161 assert flush(fh) == False
163 # Testing straight
164 assert straight(card_ranks(sf1)) == True
165 assert straight(card_ranks(fk)) == False
167 # Testing kind
168 assert kind(3, card_ranks(sf1)) == None
169 assert kind(4, card_ranks(fk)) == 9
171 # Tesing two pair
172 assert two_pair(card_ranks(sf1)) == None
173 assert two_pair(card_ranks(tp)) == (5,2)
175 # Testing group
176 assert group([2,3,4,6,2,1,9]) == [(2,2),(1,9),(1,6),(1,4),(1,3),(1,1)]
177 assert group([8,8,8,8]) == [(4,8)]
178 assert group([2,6,1]) == [(1,6),(1,2),(1,1)]
180 # Testing unzip
181 assert unzip([(2,2),(1,9),(1,6),(1,4),(1,3),(1,1)]) == [(2,1,1,1,1,1),(2,9,6,4,3,1)]
182 assert unzip([(1,6),(1,2),(1,1)]) == [(1,1,1),(6,2,1)]
183 assert unzip([(2, 9), (2, 7)]) == [(2, 2), (9, 7)]
185 # Testing hand rank
186 assert hand_rank(sf1) == (8,10)
187 assert hand_rank(fk) == (7,9,7)
188 assert hand_rank(fh) == (6,10,7)
190 # Testing hand rank alt
191 assert hand_rank_alt(sf1) == (8, (10,9,8,7,6))
192 assert hand_rank_alt(fk) == (7,(9,7))
193 assert hand_rank_alt(fh) == (6,(10,7))
195 # Testing hand rank table
196 assert hand_rank_table(sf1) == (9, (10,9,8,7,6))
197 assert hand_rank_table(fk) == (7,(9,7))
198 assert hand_rank_table(fh) == (6,(10,7))
200 # Testing poker
201 assert poker([sf1, fk, fh]) == [sf1]
202 assert poker([fk, fh]) == [fk]
203 assert poker([fh, fh]) == [fh, fh]
204 assert poker([fh]) == [fh]
205 assert poker([sf2] + 99*[fh]) == [sf2]
206 assert poker([sf1, sf2, fk, fh]) == [sf1, sf2]
208 return 'tests pass'
10. Complete Code For Homeworks (Warning: Refer this only after submitting homework)
The complete homework solution code given by Peter in this unit.
Homework 1:
1 # CS 212, hw1-1: 7-card stud
3 # —————–
4 # User Instructions
6 # Write a function best_hand(hand) that takes a seven
7 # card hand as input and returns the best possible 5
8 # card hand. The itertools library has some functions
9 # that may help you solve this problem.
11 # —————–
12 # Grading Notes
14 # Muliple correct answers will be accepted in cases
15 # where the best hand is ambiguous (for example, if
16 # you have 4 kings and 3 queens, there are three best
17 # hands: 4 kings along with any of the three queens).
19 import itertools
21 def best_hand(hand):
22 "From a 7-card hand, return the best 5 card hand."
23 return max(itertools.combinations(hand, 5), key=hand_rank)
26 # Provided Functions
28 # You may want to use some of the functions which
29 # you have already defined in the unit to write
30 # your best_hand function.
66 """Return True if the ordered
67 ranks form a 5-card straight."""
71 """Return the first rank that this hand has
72 exactly n-of-a-kind of. Return None if there
73 is no n-of-a-kind in the hand."""
79 """If there are two pair here, return the two
80 ranks of the two pairs, else None."""
88 def test_best_hand():
89 assert (sorted(best_hand("6C 7C 8C 9C TC 5C JS".split()))
90 == ['6C', '7C', '8C', '9C', 'TC'])
91 assert (sorted(best_hand("TD TC TH 7C 7D 8C 8S".split()))
92 == ['8C', '8S', 'TC', 'TD', 'TH'])
93 assert (sorted(best_hand("JD TC TH 7C 7D 7S 7H".split()))
94 == ['7C', '7D', '7H', '7S', 'JD'])
95 return 'test_best_hand passes'
97 print test_best_hand()
1 # CS 212, hw1-2: Jokers Wild
6 # Write a function best_wild_hand(hand) that takes as
7 # input a 7-card hand and returns the best 5 card hand.
8 # In this problem, it is possible for a hand to include
9 # jokers. Jokers will be treated as 'wild cards' which
10 # can take any rank or suit of the same color. The
11 # black joker, '?B', can be used as any spade or club
12 # and the red joker, '?R', can be used as any heart
13 # or diamond.
15 # The itertools library may be helpful. Feel free to
16 # define multiple functions if it helps you solve the
17 # problem.
29 ## Deck adds two cards:
30 ## '?B': black joker; can be used as any black card (S or C)
31 ## '?R': red joker; can be used as any red card (H or D)
33 allranks = '23456789TJQKA'
34 redcards = [r+s for r in allranks for s in 'DH']
35 blackcards = [r+s for r in allranks for s in 'SC']
37 def best_wild_hand(hand):
38 "Try all values for jokers in all 5-card selections."
39 hands = set(best_hand(h)
40 for h in itertools.product(*map(replacements, hand)))
41 return max(hands, key=hand_rank)
43 def replacements(card):
44 """Return a list of the possible replacements for a card.
45 There will be more than 1 only for wild cards."""
46 if card == '?B': return blackcards
47 elif card == '?R': return redcards
48 else: return [card]
54 def test_best_wild_hand():
55 assert (sorted(best_wild_hand("6C 7C 8C 9C TC 5C ?B".split()))
56 == ['7C', '8C', '9C', 'JC', 'TC'])
57 assert (sorted(best_wild_hand("TD TC 5H 5C 7C ?R ?B".split()))
58 == ['7C', 'TC', 'TD', 'TH', 'TS'])
59 assert (sorted(best_wild_hand("JD TC TH 7C 7D 7S 7H".split()))
61 return 'test_best_wild_hand passes'
100 suits = [s for r,s in hand]
101 return len(set(suits)) == 1
103 def straight(ranks):
104 """Return True if the ordered
105 ranks form a 5-card straight."""
106 return (max(ranks)-min(ranks) == 4) and len(set(ranks)) == 5
108 def kind(n, ranks):
109 """Return the first rank that this hand has
110 exactly n-of-a-kind of. Return None if there
111 is no n-of-a-kind in the hand."""
112 for r in ranks:
113 if ranks.count(r) == n: return r
114 return None
116 def two_pair(ranks):
117 """If there are two pair here, return the two
118 ranks of the two pairs, else None."""
119 pair = kind(2, ranks)
120 lowpair = kind(2, list(reversed(ranks)))
121 if pair and lowpair != pair:
122 return (pair, lowpair)
126 print test_best_wild_hand()
1. Zebra-Puzzle
Welcome. Let's talk about a cliche: the back of the envelope. Much easier to write on the back of the envelope than on a napkin, and it's a really valuable skill. It's a valuable skill in real life to be able to do quick and dirty calculations, and it's especially useful for computer programmers. It allows computer programmers to have the important virtue of being lazy. You don't normally think of lazy as being a virtue, but it is.
It allows us to say we're going to come up with the simplest design we can, validate on the back of an envelope that that design is efficient enough for what we're trying to do, and then we can stop. We don't have to try to make it more complex. This whole class is about managing complexity, and one of the most important ways to manage complexity is to leave it out completely, just go for the simple solution. If we can do that, then we're well on our way to better designs.
In this class we'll learn how to do that, we'll learn how to do the calculations of when you're efficient enough, we'll learn when to stop, and we'll learn how to make programs more efficient.
I'm going to start with a well-known puzzle called the Zebra Puzzle.
Here's a description of it.
The Zebra Puzzle
There are five houses.
The Englishman lives in the red house.
The Spaniard owns the dog.
Coffee is drunk in the green house.
The Ukrainian drinks tea.
The green house is immediately to the right of the ivory house.
The Old Gold smoker owns snails.
Kools are smoked in the yellow house.
Milk is drunk in the middle house.
The Norwegian lives in the first house.
The man who smokes Chesterfields lives in the house next to the man with the fox.
Kools are smoked in the house next to the house where the horse is kept.
The Lucky Strike smoker drinks orange juice.
The Japanese smokes Parliaments.
The Norwegian lives next to the blue house.
Who drinks water? Who owns the zebra? Each house is painted a different color, and their inhabitants are of different nationalities, own different pets, drink different beverages and smoke different brands of American cigarettes.
We're going to try to address this puzzle, see if we can come up with a program to solve it, and explore the methodology for how we come up with that solution and the process of deciding what's a good enough solution and whether a brute force solution will work.
You can read the description of the puzzle here.
Now let's start to analyze it. We'll begin with an inventory of the concepts in the puzzle just to make sure that we understand what's going on.
The first concept is houses. We're told there's 5 of them.
And then there's properties of the inhabitants of these houses and of the houses themselves. So there's nationality, colors of the houses, the pets that they own, the drinks that they drink, and the smokes that they smoke.
And then in addition to properties, there's a notion of assignment of properties to houses. And you can think of that either way. You can think of it as assigning house number 2 the color red or think of it as assigning red to house number 2.
Then there's a notion of locations, the locations 1 through 5 that mention the idea of the first house and the middle house and of the next to relation and of the immediately to the right relation.
And I think that covers most of what was in the specification.
Let's go back to it and see if it works.
So I'm seeing lots of concepts that we've already covered:
There's a few words that we haven't covered, things like lives in and owns the.
We covered them in a generic sense of saying it's an assignment of Englishman to a house and assignment of the dog to a house, but the question is, do we need to separate out the different types of assignment?
So the question is, are we missing this idea of a property name with a description attached?
So for example, the property name would be nationality and the description is lived in. Do we need to name them like that, or do we just need the notion of a property group to say that there are these properties of Englishman, Spaniard, Norwegian, Japanese, and Ukranian, and the 5 of them go together but we don't need the name for them, or can we ignore this notion of grouping altogether?
This is somewhat subjective –what's a good design choice?– but tell me which of these 3 do you think are plausible or reasonable design choices and check all that apply.
QUIZ question
1.1. Zebra-Puzzle Answer
Unit2-1 Answer
[Norvig] My answer is defining property names would be a fine design choice. It would probably help us understand what's going on in the definition of the program. Defining groups without names would also be fine.
Ignoring the groups I think would not work, and here's the problem. We have to know that if red is assigned to house number 2, then blue cannot be assigned to house number 2, but orange juice can be assigned to house number 2. So there's this idea that properties within a group are mutually exclusive and properties outside of the group are not. We need to represent that in some way.
2. Where's the Spaniard
[Norvig] The crux of the whole matter is doing the assignments. There are 2 ways to do it. We can deduce what assignment must be true – that it must be the case that the Englishman is in a certain house. Or we can just try different possibilities – try to put the Englishman in house 1, then in house 2, then in house 3 and see what works. You'd have to be very clever to figure out how to do all these deductions, and I don't want to try to be that clever. I want to see if I can get by with just trying all the possibilities, make the computer do the work so I don't have to.
So let's approach that and let's say we're going to try all possibilities. And so first we put the Englishman into house 1, and later we'll try him in houses 2, 3, 4, and 5. What about the Spaniard? If the Englishman is in house 1, what possibilities should we try for the Spaniard?
Give me your best answer:
all the numbers;
only 1;
or 2, 3, 4, 5.
2.1. Where's the Spaniard Answer
And the answer is 2, 3, 4, 5, because Spaniard is in the same property group as Englishman, and so it can't be in 1 but it can be in any of the others.
3. Counting Assignments
Within a class of 5 properties, the class must cover the 5 houses but in any order.
How many ways can we make that assignment? In other words, here's 5 houses, here's 5 properties in a property group, and here's 1 possible assignment.
How many different assignments are there of assigning these properties and matching them up with the houses?
How many ways are there:
3.1. Counting Assignments Answer
And the answer is 5!.
And you can see that because there's 5 different houses that you could assign red to, and then for each of those 5 assignments there would be 4 that you could assign green to of the remaining unpainted houses and then 3 for blue and 2 for yellow and then only 1 left for ivory. That's 5! or 5 factorial.
4. Multiple Properties
But we don't just have 1 property class; we have 5 property classes. We've got the nationalities and the pets and so on. And so if we want to make all the different ways of making assignments of all 5 properties to all 5 houses so each house will have a color, a nationality, a pet and so on in each of the possible ways of making that assignment, how many is that?
5 * 5!
5!2
4.1. Multiple Properties Answer
And the answer is 5!5 because for each of the assignments of 1 property we can have the same number of assignments of the second property, the third, the fourth, and the fifth.
5. Back-of-the-Envelope
And now, how much is 5 factorial to the 5th?
Just back of the envelope, is it approximately:
5.1. Back-of-the-Envelope Answer
And the answer is 20 billion. How do you do that back of the envelope? And we rounded down to get from 120 to 100, so we should round back up, and maybe it's somewhere around 20 billion. It turns out that the actual number is 24.9 billion, approximately, so that's not a bad estimate.
6. Leaving Happy Valley
We're in this range where we might be done but we're not quite sure. Our computers can do about a billion instructions per second or so on a good second – that is, one where they're not wasting part of the second with a page fault or a cache miss. If the answer had turned out to be in the millions, we could say, "Oh, great. We're done." If it had turned out to be in the trillions, we could say, "It's totally infeasible." "We need a better solution." But if it's somewhere in the middle with the billions, then we're not quite sure.
We better try to refine the result a little bit to tell if this brute force approach is going to work.
Think of the space of solutions in terms of execution time as like a contour map.
And here deep down in the valley where there's only millions of computations needed, millions of instructions needed to complete the answer, then we're really happy because we know that's going to go really fast. And outside of the happy valley there are these high peaks where we have trillions of computations needed, and there we're going to be sad. If we're stuck out in these domains, we're going to have to somehow find a path back in because we can't just go ahead and calculate when we're out at the trillions. And in the middle here where we need billions of instructions to complete our computation, then we're not quite sure. So maybe we're happy here and we can stay in this domain. Maybe we want to find our path through the wilderness back into the happy valley. It depends on the problem.
And we're going to try to look at problems in terms of this space, try to find our way back in, but know when to stop when we've got close enough.
Now let's keep thinking about what it means to do an assignment. Try to get just a little bit more concrete about it.
If we want to assign red to house number 1,let's think about the ways in which we could implement that.
house[1].add('red')
Here's 1 possibility where we say we're going to have an array of houses. We take number 1 and we add red to its set of properties. That means that each house is represented as a set.
house[1].color='red'
Here's another possibility where we take house number 1, we set its color property equal to red. Here each house would have to be represented as a user-defined custom class which had properties for color and nationality and so on.
red = 1
Here's a third possibility. We have a variable called red, and we just assign that the number 1. So here we were assigning properties to houses, and in this one we're assigning houses to properties.
I want you to check all the approaches that you think would be reasonable designs for implementing assignment.
6.1. Leaving-Happy-Valley
I know there can be matters of opinion here, but from my point of view they're all reasonable. So they would all work. They'd all be fine.
But for the moment, this one – red = 1 looks just a little bit simpler, so I'm going to go with that until I have some proof that the simple won't work and that we'll have to go to something more complicated.
In this approach we named the properties. We talked about the possibility of whether you have to do that or not. Here we aren't naming them, and so it will be up to me as the programmer to manage the groups of properties.
Here's 1 way I could do it.
I could simultaneously assign the 5 properties that are in the same group of color to the 5 houses. Of course this would only be 1 of the possible assignments. After I tried this one, I'd have to try another assignment – maybe [1, 3, 2, 4, 5] – and I'd have to go through all the possibilities.
Can you think of a good way to do that? What statement or other implementation can you come up with to have the 5 properties go through all the combinations of possible houses?
And if you want, pause the video now and try to think of that on your own.
7. Ordering-Houses
So here's 1 approach.
First we'll define the houses. Say there's 5 houses. That was the number 1 thing stated in the problem.
Then we'll say orderings is some collection of possible orderings of the houses, so [1, 2, 3, 4, 5] would be 1 element of orderings,
[2, 1, 3, 4, 5] would be another element and so on.
And then we just use a for statement to iterate the 5 variables in the property group through all the possible orderings and then put in our code here.
The question is, what is this function F that gives us all the orderings?
the permutation function
the combinations function
the factorial function
7.1. Ordering-Houses
And the answer is permutations. That's sort of the definition of all possible orderings is called the permutations.
8. Length-of-Orderings
It turns out that Python has a permutation function in the itertools module, so you could import that or you could write one yourself.
If you do that, one question is, what would be the length of orderings? Tell me that.
9. Estimating-Runtime Answer
Now, suppose we arrange our program like this. So we set up the houses and we have a list of all 120 possible orderings and then we iterate each of the 5 properties in each of the 5 property groups through all those orderings. And so now, 5 levels nested deep into our code we've got an assignment of all 25 properties. Each of them is assigned to one of the houses, and now we can check the constraints, like the Englishman lives in the red house and the Spaniard owns the dog and so on. Those can all go here. If we find something that checks out, we can just report we've got an answer.
Now, I want you to go back to the back of your envelope and tell me if we wrote the program this way and filled in this code with all the constraints and then ran it, about how long do you think it would take to find a solution?
About a second
an hour
or a day
Click on the best answer.
9.1. Estimating-Runtime Answer
The best estimate is about an hour, and we'll go to the back of the envelope again to figure that out.
IMAGE?
10. Red-Englishman
Unit 2-10
5!5 was 24 billion. And if we assume we have a 2.4 gigahertz computer, which is fairly typical, then if all this could be compiled into 1 computer instruction, then it would take 10 seconds.
But of course that's ridiculous. You can't do all of that in 1 instruction.
If it was 100 instructions, then that would mean it would be 1000 seconds, which is about 16 minutes, but that seems too little.
It's probably going to be more like 1000 instructions to make it all the way through doing all this and then checking the constraints, and so that would be about 160 minutes. And so an hour to 3 hours. Maybe we'll say 2 or 3 hours, somewhere in there. It might be as little as 1 hour. We can't really tell because this is just an estimate. But it's definitely going to be in the hour range and not in the minutes or day range.
In fact, I actually ran this program, and we'll come back later and see how well this estimate of somewhere in there worked out. Let me just say because we've learned that this will take somewhere in the range of an hour, don't try to run it here in the browser by hitting the Run button because we timeout the calculations after just a few seconds, so it's not going to work here.
You could type it into your own computer if you have Python running on your own computer, set it going, and see how it works.
But first we have to figure out how to do all the constraints. So we already did constraint number 1, which was houses = [1, 2, 3, 4, 5]. Now let's do constraint number 2, which is that the Englishman lives in the red house. We want to write some if statement to check if the Englishman lives in the red house and if so, we go on; if not, we give up.
And so how do we check if it's true that the Englishman lives in the red house? So if, and then I want you to fill in some expression here. We'll just say that this is constraint number 2. Tell me what you could put in here.
10.1. Red-Englishman Answer
Unit2-10 Answer
And the answer is all we have to do is check to see if Englishman and red have been assigned the same house number.
And we just do that with saying
1 if (Englishman == red)
And I should note just as an aside if you wanted to be sort of clever and make it look more like English, in this case it would be okay to say
1 if (Englishman is red)
because is is checking for whether 2 objects are identically the same object. Equals checks to see if they have the same value. And it turns out that Python handles small integers like 1, 2, 3, 4, and 5 as a single identical object. So the is check would work as well as the equals check here.
11. Neighbors
Now, before we go on to constraint number 3, I want to go backwards a little bit and say there's 2 concepts we haven't talked about yet: the concepts of being next to and immediately to the right of.
Immediately to the right of–well, because we've used house numbers to assign houses, we can say that house 1 is immediately to the right of house 2 if they differ by 1, if h1 – h2 is 1. So house number 3 is immediately to the right of house number 2.
What I want you to do is fill in the code for nextto to say if 2 houses are next to each other.
11.1. Neighbors Answer
And the simplest way to do that is just ask if the absolute value of the difference rather than the positive or negative value is equal to 1.
But an alternative way to say it if you wanted to break it down in terms of the other primitive, you could say that nextto (h1, h2) is defined as immediate right of h1, h2 or immediate right of h2, h1.
12. Slow Solution
Now let's put the whole thing together. The code below shows the brute force slow solution, followed by a description of how it works.
1 import itertools
3 def imright(h1, h2):
4 "House h1 is immediately right of h2 if h1-h2 == 1."
5 return h1-h2 == 1
7 def nextto(h1, h2):
8 "Two houses are next to each other if they differ by 1."
9 return abs(h2-h1) == 1
11 def zebra_puzzle():
12 "Return a tuple (WATER, ZEBRA) indicating their house numbers"
13 houses = first, _, middle, _, _ = [1,2,3,4,5]
14 orderings = list(itertools.permutations(houses)) #1
15 return next((WATER, ZEBRA)
16 for (red, green, ivory, yellow, blue) in orderings
17 for (Englishman, Spaniard, Ukranian, Japanese, Norwegian) in orderings
18 for (dog, snails, fox, horse, ZEBRA) in orderings
19 for (coffee, tea, milk, oj, WATER) in orderings
20 for (OldGold, Kools, Chesterfields, LuckyStrike, Parliaments) in orderings
21 if Englishman is red #2
22 if Spaniard is dog #3
23 if coffee is green #4
24 if Ukranian is tea #5
25 if imright(green, ivory) #6
26 if OldGold is snails #7
27 if Kooks is yellow #8
28 if milk is middle #9
29 if Norwegian is first #10
30 if nextto(Chesterfields, fox) #11
31 if nextto(Kools, horse) #12
32 if LuckyStrike is oj #13
33 if Japanese is Parliaments #14
34 if nextto(Norwegian, blue) #15
35 )
I've defined zebra_puzzle. It's a function. It doesn't take any arguments because there's only 1 puzzle. There aren't different puzzles that require different arguments.
It's going to return a pair of WATER and ZEBRA, so the 2 house numbers indicating which house drinks water and which house owns a zebra.
I've repeated houses = [1, 2, 3, 4, 5] and I've also defined first and middle. I've repeated the work of figuring out all the orderings. That's constraint #1.
And then I've written the whole function as a generator expression, and I chose to do that rather than sticking with the nested for loops just because the structure is a little bit easier.
What we're doing here is we're asking for the next version. So in other words, the first time through we're asking for the very first solution to this iterator expression where we say iterating through the 5 properties, if each of the tests is true, then return the values of WATER and ZEBRA and then keep on doing that for all possibilities.
But I'm only interested in the very first one, which the next after nothing is the first.
So go ahead and return that. Each of the constraints is very easy to state.
Englishman is red, the Spaniard is dog, coffee is green and so on.
We have some immediate rights and nextto, and that's the whole problem. So in some sense we have a solution; in another sense we know that it's going to take something like an hour or 2 to complete, and maybe we're not happy with that, so we're going to make it faster.
But before I do that, I want to do a little bit of review of these generator expressions, because probably you haven't seen this too much before.
13. List Comprehensions
And before I do generator expressions, we're going to do list comprehensions.
You've seen them before. We saw in Unit 1 we said that we could get the list of suits by saying
1 [s for r, s in cards]
And so the format here is that the first 's' is the individual term of the list comprehension and it is followed by a "for clause".
And in general for list comprehensions, we're going to have 1 for clause at least and then, optionally, we could have more "for" or "if" clauses at the end.
And that's what a list comprehension looks like in general.
Now, what does the list comprehension mean?
This expression is similar to saying we're going to write out a loop where we say the result starts as the empty list, then we take the for part and then we take the term and append that into the result. And then when we're all done, result will hold the value of this expression. This would be written as:
1 result = []
2 for r, s in cards:
3 result.append(s)
Now I'm going to show you a more complex list comprehension.
Say we wanted the list of suits for all our face cards. Then we could say s for r, s in cards if the rank of the cards is, say, in jack, queen, king. This would be written as:
1 [ s for r, s in cards if r in 'JQK' ]
I don't know why you'd want that, but there's a possibility. So I've got the term I'm building up, I've got 1 for loop controlling it, and now I've got an if statement, and that's as if we had written this code:
3 if r in 'JQK':
if we had inserted inside the for loop an if statement and then only done the appending of the term s if the if statement is true.
And in general, we can have any number of for statements, if statements, more for statements, more if statements. We can keep on adding those for clauses and if clauses and intermingling them in any order we want, as long as the for clause comes first. And we can have that 1 big list comprehension, like:
1 [ s for r, s in cards if r in 'JQK'
2 for …
3 if …
5 for …]
That corresponds to putting in more fors and ifs here in this loop, like the following:
6 for ….
Now, going back to looking at the form of the list comprehension again (repeated here again for clarity with the text following it):
The whole thing is read left to right except for the term ('s' in the above example).
So the way to read a list comprehension is to say I'm going to ignore the first 'term' part for now, and then I think of it as saying it's a bunch of nested statements– for statements, if statements, for statements. They all get nested deeper and deeper. Then when I get to the end, the very last clause, now I'm appending together the term, so now read the term.
And that's why it's okay that term at the start of the list comprehension looks like it's referencing a variable s that hasn't been defined yet. That's okay because s has been defined in the following for clause.
And it looks like it's used before it's defined, but that's not the case, because when it's actually used is right down here at the end.
14. Generator Expressions
So that was list comprehensions. Now let's look at generator expressions, which is almost the same idea.
The syntax is the same in that a generator expression consists of a term, a mandatory for clause, and then optional for and ifs clauses – as many of those as you want–0 or more.
There's 2 differences. The generator expression uses parentheses instead of square brackets. Square brackets mean list; parentheses means generator. And then the other difference is that the computation doesn't get done all at once. Instead, a generator expression returns a value, which is a promise to do the computation later.
So if we say g = this and then g is this promise, it hasn't done any calculation yet. It hasn't calculated any of the terms. And then I can ask, give me the next g. Then it starts doing the calculation and it keeps on looping through the for clauses or maybe multiple for clauses until it finds the first term and returns that. And then if I want, I can again ask for the next g and it will give me the second one and so on.
Let's look at an example. Here I've defined the function sq for square of x. It takes in the value x, prints out that it's been called, and returns x * x. Here I've defined a generator from this generator expression that says
g = (sq(x) for x in range(10) if x%2 == 0).
So that's saying if x is an even number. And notice nothing has happened yet. We didn't get any printing of square was called, so square hasn't been called yet.
The generator function is this promise. We can look at it. It says it's a generator object, but no computation has been done yet. We can ask for the next g and now, finally, square gets called with 0 as an argument, and we return 0 as a result. We can do that again. We get 4, 16, 36, 64. And what do you think is going to happen next?
Now we're getting to the end of the loop. Range(10) means 0 through 9, so there are no more. So now when we ask for the next one, Python raises this condition called StopIteration. So it's saying, "I've gotten to the end." "I have to stop the iteration because there's no more I can give you." "I can't give you the next one." This seems a little bit inconvenient because now I've got these errors and my program has to deal with them, but the idea is that you rarely will be calling next directly. Rather, most of the time you'll be doing this within a for loop. So I can say something like this where I say for x2 in this expression do something, and now the protocol for a for loop arranges to call the generator each time, to call the next function, and to deal with the StopIteration exception and catch that. And so everything works fine.
I can also convert the results. Here I've said I've got a generator expression and I'm converting that into a list. It does all the work and then it returns the result as a list. So I never have to deal explicitly with those StopIterations.
Why do you think I chose generator expression to implement the zebra puzzle?
Do you think I wanted to confuse students;
have less indentation so that the code would fit on the page;
stop early as soon as I found the first result;
or make the code easier to edit,
to move around the various pieces of the constraints and so on?
14.1. Generator Expressions Answer
The answer is no, I didn't try to confuse you. In fact, I'm trying to show you a very useful tool in generator expressions.
Yes, I thought the indentation was important. With all those fors and ifs, I would have run out of space across the page.
Yes, I wanted to stop early, and so a list comprehension would have been a bad idea because a list comprehension would have had to do all the work, whereas a generator expression can stop as soon as it's done.
Having statements rather than expressions also would allow me to stop early.
And yes, it's easier to edit.
If I wanted to move around the order of the clauses, instead of having the indented structure that I would have with statements I have a flat structure and it's easy for me to pick out 1 of the constraints and move it somewhere else in the list without having to worry about getting the indentation right. So editing expressions is easier than editing statements in Python.
15. Eliminating Redundancy
I wrote this version of the function, and as soon as I wrote it, as I was recording this video I set it running. And you know what? It's still running. It hasn't completed yet. I've got to admit I've done this before, so I have an idea of how long it's going to take. But we're in a race with it.
Let's see if before it completes – we know it's going to take on the order of an hour or so – can we write a program that's better and faster and maybe even though that program's got a head start, we can catch up with it and finish first?
The problem with this program is it goes through all this work to try all the 5 factorial to the 5th combinations and then they get ruled out really early. And some of the combinations, it seems silly that we're bothering with them. So if the Englishman is not red, we should know that by the time we've got through the second set of assignments here. Here we've assigned red to some house and Englishman to some house. If we didn't assign them to the same house, why are we bothering to go through all the possibilities for the other properties?
And so what we could do is move this constraint up to the earliest time in which both Englishman and red are defined, so there. Now I've moved it up, and now if Englishman is red is false, then we don't even have to bother to go through all 3 of these loops. So we're going to eliminate a lot of work in just that 1 clause.
Now let's consider another of the constraints. Let's look at immediate right of green and ivory. Can we move that up? And where can we move it up to?
Can we move imright(green, ivory) up to here, here, here, or here? IMAGE
15.1. Eliminating Redundancy Answer
And the answer is we can move it all the way up to the top because both green and ivory are bound at this point.
16. Winning the Race
And if we keep on going, moving every constraint up to the highest point it will go, we get this.
9 def zebra_puzzle():
10 houses = [first,_,middle,_,_] = [1,2,3,4,5]
11 orderings = list(itertools.permutations(houses))
12 return next((WATER,ZEBRA)
13 for(red, green, ivory, yellow, blue) in orderings
19 for (cofee, tea, milk, oj, WATER) in orderings
20 if cofee is green #4
24 if Kools is yellow #8
30 if nextto(Chesterfields, fox)
31 if nextto(Kools, horse)
34 print zebra_puzzle()
And now I've checked, and the original version of the puzzle that I set running is still running, and now I can hit Run on this and in less than a second I see my result, (1, 5), meaning water is drunk in house number 1 and the zebra is owned in house number 5.
This is an amazing thing.
It's as if we had a car race and we had the start here and the finish someplace far away and we had this competitor car which was the original zebra puzzle car – we'll call him z for zebra. We said, "Go!" and it started out down the path towards the finish.
We wanted to build a competitor to that, but we just let it go. It's running, and we're thinking and we're analyzing and we're not doing anything, and it's getting farther and farther ahead and closer to the finish line. Maybe we spent half an hour and it's halfway through. It's gotten up to this point. We're still stuck at the Start sign, but what we're doing is we're building a better car. We're building a super fast race car, and we're putting the pieces together using what we know, and eventually this car has gotten a long ways along the way and then we say, "Now we're ready to go."
And when we hit Go, zoom, we're there in less than a second, what took this car who had a half hour head start is not even close.
So it's like the tortoise and the hare, only in reverse.
And by thinking and coming up with a better design, we were able to beat this original design even though we gave him a half hour head start. We won the race. We get the checkered flag. We should be happy at what we've done. And congratulations.
Now, if you're the type who just cares about winning the race and finishing first, we can stop here. But if you're a little bit more analytic and you like to know the exact scores and times, then we've got work to do. Here's 1 thing I can do. There's a module in Python called time. In that module there's a function called clock which gives you the time of day.
And so I can set t0 to be the time before I call the zebra_puzzle, t1 to be the time afterwards, and just return the difference so that will tell me how long the zebra puzzle took. If I hit Run, the answer it came back with was 0.0 seconds. That's because the system I'm running on doesn't have an accurate enough clock. And I know it's accurate down to the thousandth of a second, so all we can say here is that it took less than a thousandth of a second. I've run it on other systems, and it comes back at 0.00028 on that other system. I was able to do this, but this looks like a good opportunity for generalization.
Why do I want a function that only times the zebra puzzle? It would be great if I had a function that could time the execution of any function. So here I've defined a function called timedcall which takes another function as input, sets up the clock, calls that function, calls the clock again to figure out the time, and tells me what the elapsed time is.
So I could call timedcall of zebra_puzzle to get the answer of how long that took, or I can apply this to any function. So I built a useful tool that I can use again and again. I can make it even more useful by doing 2 things. One is saving up the result and returning that as the second value in case we're interested both in the result and in how long it took, and secondly, allowing functions that take arguments.
You may not have seen this notation before, so let's talk about it for a second.
17. Star Args
So the *args notation appears in 2 places:
1 def something(fn, *args): # function definition
3 … fn(*args) # function call
It appears in the definition of a function, and it can appear in a function call.
In the definition of a function, what it means is this function can take any number of arguments and they should all be joined up into a tuple called args. So if the call to the function looked like this:
1 something(f, 1, 2, 3)
then 1, 2, 3 would be bound up into a tuple and assigned to args. So within the body of something, args is equal to the tuple (1, 2, 3).
And then this part here:
means take that tuple and apply it not as a single argument but rather, unpack it and apply it to all those arguments. So fn(*args) is equivalent to writing fn(1, 2, 3).
It's just a way of packing and unpacking arguments with respect to function calls.
One thing I want you to notice here is that we used a function as a variable, as a parameter or something that's passed into another function.
1 def timedcall(fn, *args):
2 "Call function with args; return the time in seconds and result."
3 t0 = time.time()
4 result = fn(*args)
6 return t1-t0, result
We're using the property of Python that functions are first class objects just like integers and lists and everything else. Functions are objects that you can pass around.
That's very powerful because that way we can control both what happens and when it happens.
The idea here is that we want to call this timedcall() function, but we want to delay calling of the function until we've already started the clock. We want it to happen between the time we start the clock and the time we end the clock.
And in order to do that, if we tried to write something like this–
1 timedcall(zebra_puzzle())
–then timedcall wouldn't work.
It would be too late because what we pass to timedcall would be the result of zebra_puzzle.
We don't want to pass the result; we want to pass the function that we're going to call and be able to delay execution of that function until the right time, until we're ready to set the clock.
And so functions allow us to do that–to delay execution until later.
18. Good Science
Now, let's get back to this idea of taking timings.
When we're taking timings we're doing measurements. Now we've changed what we're doing from having computer science being a mathematical enterprise to having it be a experimental enterprise– an experimental science. We're actually doing science.
It's as if we're dealing with the messy world with some chemicals in a beaker, and we're making measurements of those rather than trying to do mathematical proofs.
One thing we know about doing experimental science is that one measurement isn't going to be good enough. That was clear when we got a measurement of 0.0.
We know that wasn't right. We know that there can be errors in measurements.
One way in science that we deal with that is by repeating the measurements multiple times.
I want you to tell me why you think some reasons are that scientists, when they do an experiment, take more than one measurement.
Is it because:
we want to reduce the chance of some external event effecting the result?
we want to reduce the natural random variation?
we want to reduce errors in the measurement process?
18.1. Good Science Answer
The answer is that all three of these are reasons to repeat measurements — take more than one measurement.
Now, just by itself taking repeated measurements won't solve all these problems. If you have a systematic error, then repeating them will just repeat the same systematic error, but they certainly are required, although we may need more.
19. Timed-Calls
Let's build a better tool. Here I built a function called "timedcalls" with the plural s rather than a single timedcall, and it takes a number N, saying repeat this timing N times. Then it takes the function and the arguments to apply to. It builds up a list of the timed calls. It throws away the results and just takes the time. Then it returns three values, the minimum of the times, the average of the times, and the maximum of the times. From those you can do whatever statistical analysis you want to do in order to get a better feeling for what the timing is like.
Now, if the function runs very quickly– say, if the function took 100th of a second then you might want to give an N of 1000 or so. If the function takes about a second to run, maybe you don't want to wait that long, or maybe you want to give a smaller value of N. Part of the problem is if you have a good idea how long the function takes, then you can be precise about what a good value of N is. If you don't, you don't know.
I'm going to propose a different version of timedcalls. This version has the same signature as the three inputs, and returns the min, average, and max, but this time it treats N two different ways. What we're going to do is say if N is an integer, then do the same thing we did before, but if N is a floating point number, then what we want to do is keep on repeatedly call timedcalls for the number of trials it takes until we've added up to that number of seconds.
If N is equal to the integer 10, we repeat 10 timedcalls. If N is equal to the floating point number 10.0, then we repeat it until 10.0 seconds have elapsed.
Here is a hint. We can ask if N is an integer and then do one thing. If N is not an integer, then to the other. See if you can fill in that code.
19.1. Timed-Calls Answer
Here is my answer.
If it's integer, we do what we did before. Otherwise, we go into a loop until the sum of the times is greater than or equal to the floating point number that passed in. Now I think we've got a pretty good handle on timing our functions.
Another interesting question would be how many assignments did we take along the way, not just how long did they take?
Now, if our program had been structured using for statements rather than using the big generator expression, we could do it something like this. We could say we're going to count the number of assignments. We're going to start the count at 0, and then every time we go through an ordering we're going to increment the count by 1.
You might say we want to increment the count by 5, because we've assigned each of the 5 houses one of the colors, but I'm counting that as 1. You can choose which one you want to go on. Then we just have a structure like that, and we can keep track of the counts. That will work fine, and it's not too bad, but it bothers me a little bit that we had to do such violence to this program. We had this program, and we had to go in in so many places, interrupt it, and put in so many statements. I wanted to see can we separate that out a little bit?
20. Cleaning-Up-Functions
Now, when we're designing a program, we're thinking at multiple levels. We're thinking about different aspects of that program.
For example, we always have to think about is the program correct. Does it implement what we want it to implement?
We saw in the zebra puzzle that we also want to worry about is it efficient. Maybe we have a correct program, but it takes an hour, and we'd rather have a program that takes thousandths of a second.
We also have to worry about the debugging process of as we're developing the program, we're building up all sorts of scaffolding to make it run properly that maybe we won't need later.
Traditionally, we write our program. Some of what we're written has to deal with having it be correct. Then some of the code interspersed in there deals with efficiency. Now we've talked about adding further code that deals with debugging it. We end up with this mess that's all interleaved, and wouldn't it be nice if we could break those out so that some of the debugging statements were separate, some of the efficiency statements could live someplace else, and the main sort of correctness program could, say, stay distinct from the other parts.
This idea is called "aspect-oriented programming." It's an ideal that we can strive for. We don't get it completely, but we should always think of can we separate out these different aspects or concerns as much as possible.
What I want to ask is if I've written the function this way as this big generator expression rather than as nested statements, I can't have count equals count plus 1 statements inside here, because there's no place to put something inside a statement, and I'd like to separate the counting as much as I can from the puzzle to do the minimum amount of damage or editing to allow me to insert these bookkeeping nodes that do that counting. Let's think about that. What would be the right way to do that?
I'll tell you what I came up with. This is the second-best answer I came up with. That's to say, well, what am I doing? I'm counting iterations through the orderings, so maybe the place to insert my annotations is right here. I want this to be a debugging tool.
For debugging tools, I'm willing to have shorter names, because they're things that really aren't going to stick around. I'm going to use them for a while, and then I'm going to get rid of them. I'm willing to have a one-character name here, although it bothers me a little bit. I can always rename it as something else.
What I'll do is I'll just insert this call to the function c around each of my uses of orderings. I'm going to insert 1, 2, 3, times 5, 15 characters. There we are. I think that's a lot less intrusive than putting in a lot of count equals count plus 1 statements. I'm pretty happy with that.
What I've done here is defined a function called "instrument function," which will count the calls that it takes to execute the calling of a function with the arguments.
I haven't shown you the definition of c yet, but when I show it to you, you'll see that it keeps track of two counts– the number of starts, the times we started an iteration, started the for loop, that was measured with the c function, and the number of items that we got through.
How many times did we go through that for loop? We initialize those counts to 0, we call the function, and then we say what do we get back. With the zebra puzzle, it only took us 25 iterations over 2,700 items.
Puzzle2–this was the definition for when we took the original definition and then moved only one of the constraints up in the ordering. That gave us this number of iterations and items.
I didn't show the puzzle where none of the constraints are moved up. That would've taken even more. We see even here quite a reduction in the number of iterations in the counts, and this tells you how efficient we are.
21. Yielding Results
To implement the function C, I have to tell you about a new tool called a generator function, which is almost the same as a generator expression, and we'll show you what it looks like. I'm going to define a function to iterate over the integers from some start to some end. This is going to be just like range. Range is a great function, but sometimes it's annoying, because you really don't want to 1 less than the end. You want to go up all the way to the end.
That's what ints is going to do. I'm going to implement it as a generator function. I'm going to start off with an integer I that starts at start number that you told me. Then while i is less than or equal to the end, I'm going to introduce a new type of statement here called yield, and I say yield i, and then I'll say i equals i plus 1. Now, the yield is something new, and it makes this definition a generator function, rather than a regular function. What that means is what's going to happen is it's going to execute. As soon as it sees this yield statement, it's going to generate that value i, but keep it's place and when asked for the next, it will continue incrementing i and then continuing through the loop.
1 def ints(start, end):
2 i = start
3 while i <= end:
4 yield i
5 i = i + 1
When we call this function, if we say L equals the integers from 0 to 1 million, now L won't be a list right away. It'll be a generator. Now L is equal to this generator, some funny object. Then you call next of L to get the next object, and you keep on going through. More commonly, instead of calling next explicitly, you'd say for i in L.
And so generators obey this iteration protocol just like other types of sequences. The great thing is we can yield from anywhere inside this function. We get a very complicated logic here and then yield a partial result, and then continue getting the next result right where we are. This is a convenient feature for writing functions.
Another great thing about generator functions is it allows us to deal with infinite sequences. Let's say we wanted to allow the possibility of an open ended interval of integers. We'll make end be an optional argument, which can be None. The idea is that if we have a call, say ints, starting at zero, but I don't give it a end, then that means all the non-negative integers–0, 1, 2, 3, going on forever and never stopping.
The question for you is how would you modify generator function in order to make it obey this idea of never stopping when end is equal to None. You should be able to modify just this line here to make it obey this idea that it should keep on going forever when end is None.
21.1. Yielding Results Answer
Unit2-21 The answer is we want to continue with this while loop while i is less than end or end is None. In the case where end is None, that will give you an infinite loop.
1 def ints(start, end=None):
3 while i <= end or end is None:
22. All ints
Let's give you practice with one more example. I want you to define for me the function all_ints, which generates the infinite stream of integers in the order of 0 first, then +1, -1, +2, -2, +3, -3, and so on. Put in your definition.
22.1. All ints Answer
Here is my answer. It would start by yielding 0. Then for i in all the positive integers yield first +i and then -i.
1 def all_ints():
2 "Generate integers in the order 0, +1, -1, +2, -2, +3, -3, …"
3 yield 0
4 for i in ints(1):
5 yield +i
6 yield -i
If you didn't want to use ints here, you could do an alternative solution that looks like this. While True means it's an infinite loop, and I'm yielding +i and -i and then incrementing.
4 i = 1
5 while True:
23. Nitty Gritty For Loops
Now, after all that, I think you're finally ready –you're mature enough– to learn the whole truth about how for statements actually work. You've been using them all along, but you may not have known the inner details, the gory truth, about what's inside the for statement.
Now, when I say:
1 for x in items:
2 print x
Assuming items is a list or a tuple or a string, you think of this code probably as something like:
2 while i > len(items):
3 x = items[i]
That's a good model as long as items is one of these sequence types like lists, but items can also be other things, as we've seen. It can be a generator expression, for example.
Overall, Python calls the thing that can be iterated over in a for loop an iterable. Strings and lists are examples of iterables, and so are generators. What actually happens when we implement this piece of code, it's as if we had written this code.
1 it = iter(items)
2 try:
4 x = next(it)
6 except StopIteration:
In the above code, first we take the items and we make an iterator from them by calling the built-in function "iter." I'm going to call that "it." Then we're going to have a while loop that goes forever. "It"s loop control says while True we're going to assign x to be the next item off of the iterator, then do whatever was in the body of the for loop, in this case print x.
We're going to keep on doing this sequence as long as we can get an x value. But when next stops then we'll have an exception, which is a stop iteration exception, and we don't need to do anything more. We're done.
That's what a for loop really means in Python.
We take the items, we create an iterator over them, we call that iterator until we're done, and then we pass through to the next statement. We're finally ready to define this c, this counting function.
1 def c(sequence):
2 """ Generate items in a sequence; keeping counts as we go. c.starts is the
3 number of sequences started; c.items is the number of items generated. """
4 c.starts +=1
5 for item in sequence:
6 c.items += 1
7 yield item
What it does is it takes a sequence, it says this is the first time I've been called. I'm going to initialize my starts to one. Then I'm going to enter into a loop and this means that c is a generator function. The generator function will be returned, and as part of the for protocol, we'll call that generator function each time we want the next item in the sequence and each time we do that, our count of items will be incremented.
When we're done, when the for loop doesn't want any more, we'll stop incrementing.
We don't necessarily go through every item in the sequence. We'll just have just the right number of counts for item. This will give us the right number of starts and the right number of items. We can do that because we control the time of execution, because this is a generator function and not a regular function.
24. Zebra Summary
In summary, what have we learned from our friend the zebra?
Concept inventory
Refine ideas
Simple implementation
Refine code
Well, we learned that we took our approach of making a concept inventory and then refining the ideas and choosing the simplest implementation we could think of and then doing a back of an envelop calculation to say how long is it going to take to run this simple implementation and then refining the code as necessary, making it a little bit faster by swapping around some of the clauses.
We also learned the idea of building tools. We built tools for timing and counts.
In general we learned this idea of separation of aspects from the program to try to keep the design as clean as possible, so that you could tell what was working on getting the problem right and what was working on making it more efficient.
In the end you might say, wow, that was a clever solution. It was great to see how it works out, but we can see if we just follow the steps we can arrive at a solution like that every time.
25. Cryptarithmetic
See Unit2-25
Now we're going to turn our attention to a different type of puzzle.
This is called a cryptarithmetic problem.
"Crypt-" for cryptography–secret writing—and arithmetic for arithmetic– doing addition and other types of problems, and the idea here is that each of these letters of the alphabet stands for some digit from 0 to 9.
The problem is to figure out which digit stands for which such that the equation will be correct.
Some people call these alphametics–is another name for them. They're a puzzle for humans to work out because there are so many possibilities, and humans are limited in the speed in which they can mark them out. But there are some types of inferences that they can make.
For example, in this addition problem here we have an addition problem of of two 3-digit numbers, and the result is a 4-digit number. E is that fourth digit, so what does that tell us about what the letter E must stand for?
What digit it must stand for?
Do you think E should be?
25.1. Cryptarithmetic Answer
The answer is that E has to be 1 because E is the carry digit.
After adding up these two 3-digit numbers we get a little bit more, and the most that the carry digit could be is a 1.
If O stood for 9, even 9 plus 9 is 18, and we might carry over a little bit. But still that's 19. It's not 28. So the E must be a 1.
We could go on from there and figure other things.
26. Odd-or-Even
Is this digit the digit that replaces the N? Do you think that should be odd or even?
26.1. Odd-or-Even Answer
The answer is N must be an even digit because D plus D, no matter what D is, that's equal to 2 times D, so N must be even.
Human problem solvers would continue on like that, making inferences from what they know about the rules of arithmetic to figure out what each letter should be.
In the end we come up with one of two possible solutions.
Either '655 + 655 == 1310', or '855 + 855 == 1710'.
27. Brute-Force-Solution
Here is one possible design for coding up a solver for these types of problems.
That design would be to write down all the rules of arithmetic in terms of carry digits, in terms of odd and even and so on.
Now, that seems like a challenging task.
There's a lot of complexity involved in understanding all the rules about arithmetic.
Even if we figured out everything about addition, there's also subtraction and multiplication and other operators.
So what we really want is a short cut that'll allow us to eliminate this complexity.
Let's go back to the back of the envelope and see if we can figure out a shortcut.
One possibility would be to try all possibilities.
There are 10 digits.
All combinations of digits–all permutations of the digits, rather– would be only 10 factorial, which I happen to know is about 3 million. That's not so many. It seems like it's feasible to try all the possibilities. It's not going to be super quick. We would rather have this be thousands rather than millions because there seems to be a fair amount of work in substituting in all the letters with digits. But we can expect to be able to try all millions, not within a second, but within about a minute or so.
Now we have an approved design, which is we represent our formula as a string, and we'll use official Python notation here with the double equal sign.
Then we fill in with all permutations of the 10 digits for each of the letter, and if there's fewer letters, we have to account for that.
For example, we might substitute a 1 for the Os, and a 2 for the Ds, and a 3 for the Es, and a 4 for the Vs, and a 5 for the Ns.
Otherwise just copy the equation. Then evaluate that and check if that's equal to True. If it is, then we have a solution. If it's not, we'll go back and we'll try another combination– maybe 1, 3, 3, and so on. We'll keep on going through until we find some permutation that works. That's the design.
Now, let's take an inventory of all the concepts we're going to need. First we have equations. There's two types of those–the original and the filled-in. The original has letters. The filled in has digits. Letters and digits are concepts we have to deal with. The assignment of a letter to a digit or set of those is also a concept we have to deal with. I think that's pretty much it.
Maybe one more concept is evaluation or validation that the filled in equation is an accurate one.
Now let's come up with some choices to represent each of these concepts. The original equation can be a string. The filled-in equation can also be a string. The letters will be single character strings like D. The digits will also be single character strings–like 3.
The assignment of letters to digits will be some sort of mapping or a table that consists of this type of mapping that says D would be replaced by 3 and so on. It turns out that there is a facility built into Python that's part of strings called the translate function. We can call the str.translate method in order to enact that type of assignment or substitution.
Then for the evaluation, there is a function called "eval" in Python, which takes a string and evaluates it as an expression.
You may not be that familiar with these last two items, so let's go over them.
Eval is pretty simple. If we asked for eval of the string "2 + 2," then that would give us a value 4. If we asked for eval of the string "2 + 2 == 3," Python would evaluate that using it's normal rules and tell us that that's equal to False.
28. Translation-Tables
Now I'm going to show you how these translation tables work.
I'm going to define a variable called "table" that's using the string.maketrans function, which makes a translation table, and I'm going to tranlate from the characters 'ABC' to '123.' I can give any number of characters here– the characters I want to replace and the ones I want to replace them with. I should say that this is using the string module, so somewhere we have to say import string before we start doing any of this. You only have to do that import once, of course.
Now I'm going to define a formula f to be a simple formula A plus B equals C. Then I'm going to call the translate method of the formula f and pass it this translation table. That will evaluate to the string 1 plus 2 equals 3. It has taken each of the elements in the table, and they correspond A to 1, B to 2, C to 3, substituted those into f and given me back a brand new string. Now if I go ahead and evaluate f.translate of table, which is 1 plus 2 equals 3, then that will give me the result True, because 1 plus 2 is 3, and that's a legal Python expression.
Now what I want you to do is to define for me the function "valid," which takes a filled-in formula like 1 plus 2 equals 3, filled-in formula f, and returns True or False. True if the formula is, in fact, valid. If it represents a true equation like this.
And False if it represents an invalid equation like 1 plus 3 equals 3. Or it should also return False if it represents a error like 1 divided by 0 equals 3. That wouldn't return True or False, that would signal an error, and I want you to handle that within the code for valid.
I'll give you a hint, which is you should consider using a try statement. Try, do something, and then you can say "except ZeroDivisionError" something. What that does is it executes the main body in which you can test if evaluating this expression f is true or not and return appropriately, but if evaluating the expression f causes a zero division error, then this clause will catch it, and then you can do the appropriate thing here.
You should also think about if there's anything else that can go wrong within the execution of valid.
Here's my version of the solution.
1 import string, re
3 def valid(f):
4 "Formula f is valid iff it has no number with leading zero, and evals true"
6 return not re.search(r'\b0[0-9]', f) and eval(f) is True
7 except ArithmeticError:
8 return False
I'm defining valid, takes filled-in formula f, and it's going to return True.
The main part is if we evaluate f and if that's true, then we should return True, but I also had to check for the zero division error and even to be a little bit more sore here, I ended up checking for arithmetic error, which is a super class of zero division error. It covers a few additional things like overflow of really big numbers. You didn't have to do that. I would've been fine to just catch the zero division error. If there is such an error, then you should return False.
But I did one more thing here, and it looks kind of complicated. I'm using a regular expression search, and let's look at exactly what's going on in this confusing part of the clause here.
29. Regular-Expressions
Let's say we have a formula like ODD plus ODD equals EVEN.
Now, one of the rules of the way we form numbers is that we don't want to have a number that starts with 0. EVEN could be, say, 3435. That would be a perfectly valid set of digits to fill in for the word EVEN. It shouldn't be 0405, because we just don't normally write numbers with a leading zero. I excluded that. How did I do it? I did a regular expression search for the regular expression slash b, which matches at a word boundary–b stands for boundary. Then I'm looking for a zero followed by any digit 0-9. I'm looking for that within f, and I want that to be not true.
If I took this string here–something plus something equals 0405, and I did this regular expression search, it's saying find me a word boundary, a boundary between a letter and/or a digit and something that's a punctuation or something else, a word boundary followed by a 0, followed by another digit, it would say, yes, indeed, that search does succeed. It succeeds right here. The 0 and the 4 match. I want that not to succeed, so I would return False.
I would rule out this case where we have a word starting with 0 and 4. I did that, one, just because it's good form that normally you don't allow valid numbers to start with a zero, and also because in Python we could come up with an error because of that. Here is the problem. In a Python program, the string 012 corresponds to an integer, but the integer it corresponds to is not 12. It's 10. Why is that?
Well, it's an old historical accident. Way back in the 1970s, the C programming language defined it that way, where they said any number that starts with a 0 is going to be interpreted as an octal number, which means base 8. So 012 means one 8, zero 8-squared, and 2 more, so 8 + 2 is 10. So that would give us the wrong answer if we allowed octal numbers to sneak in where we were expecting decimal. It would also give us a possible error. If we had the string 09 that gives you an error, because 9 is not an octal digit.
To avoid all those problems, I put in this regular expression that says any time you see a lead zero just rule it out.
30. Solving Cryptarithmetic
Now what I want you to do is write for me the code for the function solve. Solve takes an unfilled formula with letters in it, and if it can find a solution in which you can fill in all the digits, it returns the string with the digits filled in properly. If not, it should return None.
Put your code here.
You can assume that we already have the correct definition of what a valid filled-in formula is. You can assume that we have a function called "fill_in" that I'm not showing yet.
Fill_in takes a formula and generates a collection of all the possible filling ins of the letters. You pass it the string ODD plus ODD equals EVEN, and it passes you back first string filled in with, say, a 1 for the O and a 2 for the Ds and so on. Then the next string filled in with different numbers filling in each of the possible digits, and so on and so on.
30.1. Solving Cryptarithmetic Answer
Here's a code that's pretty simple.
1 def solve(formula):
2 """Given a formula like 'ODD + ODD == EVEN', fill in digits to solve it.
3 Input formula is a string; output is a digit-filled-in string or None."""
4 # Your code here
5 for f in fill_in(formula):
6 if valid(f):
7 return f
All we do is we iterate the variable f over all the possible values of filling in the formula. If we say that filled-in formula f is valid, then we should go ahead and return it. Otherwise, when we get through the end of the loop, we automatically return None.
31. Fill In Function
Now I want to talk about a strategy for defining the function fill_in, which takes an unfilled formula and returns all the possible filled in formulas.
What do we have to do?
Well, let's consider a formula, and I'm going to take one that's simpler than one we've seen before. I'm going to take the formula:
1 'I + I == ME'
What we have to do then is find out what all the letters are in this formula and fill in all possible values for digits for those letters.
It seems like a good thing is to collect all the letters, and in this particular formula the letters should be 'IME'.
What are the possible digits that we want to replace these three letters with?
Well, we can just iterate through all the 3-digit numbers, but make sure that we're not duplicating any of those numbers. So maybe we'll start with 123, then 124, and so on, 120, and then 13.
We wouldn't do 131, because we already have a 1 there, so 132. We just have all these possibilities.
How many possibilities are there?
Well, there's 10 possible first digits. Then we don't want to repeat a digit, so 9 possible second digits, and 8 possible third digits, so that's 720 possibilities. Not very many at all. It should be really fast to go through all these possibilities.
Now, what function gives me all the possible sets of numbers in order (so order matters)?
What function is it that we give it the list of digits?
So we want some function F, we give it the list of digits, and we give it the number that we want, and there are three letters in this particular formula, so we want three of them. What function F takes that and then returns this sequence of all possible sets of numbers?
Maybe it returns it as a tuple or a list or whatever.
The question is what function can do that?
Is it combinations, permutations, factorial, or some other function?
31.1. Fill in Function Answer
The answer is that it's permutations.
That's what the definition of permutations is. We take some collection, and then we pick out some number of them. The order matters, and we have all possibilities.
So there is a built-in permutations function. It happens to be in a library called "itertools."
32. Filling In Fill In
Now with that in mind, I've given you the template of most of the the fill_in function, and I want you to just put in two missing pieces.
What does the fill_in function do?
First, it finds all the letters in the formula, and I want you to fill in that piece. Then it iterates a collection called the digits, a permutation of the digits taken from all the digits n at a time, and you have to fill in the right value of n. Then it builds a table from taking the letters.
That means you want to make sure that the letters up here are represented as a string and not as some other type of collection. It makes a translation table from the letters and a string of all the digits that we got out of the permutations.
The function itertools.permutations returns a tuple of results, and so we want to join together that tuple into one big string, make up the translation table, and then call formula with the translate method with that table to translate all the letters into the appropriate digits and yield that result.
In other words, fill_in is a generator function not a function that returns a list of results.
Why did I do it that way? Well, because you might get lucky. It might be that the very first formula you try or one of the few first formulas you try is the correct one.
If I do for f in fill_in a formula, I ask for the first formula, and if it is valid, then I want to return it right away. I don't want to waste time calculating all the other possible fill_ins. That's why a generator makes more sense here.
See if you can fill in these two missing pieces.
32.1. Filling In Fill In Answer
1 from __future__ import division
2 import string, re, itertools
11 def fill_in(formula):
12 "Generate all possible fillings-in of letters in formula with digits."
13 letters = ".join(set(re.findall('[A-Z]', formula))) #should be a string
14 for digits in itertools.permutations('1234567890', len(letters)):
15 table = string.maketrans(letters, ".join(digits))
16 yield formula.translate(table)
18 def valid(f):
19 """Formula f is valid if and only if it has no
20 numbers with leading zero, and evals true."""
21 try:
22 return not re.search(r'\b0[0-9]', f) and eval(f) is True
23 except ArithmeticError:
The first part here was easy. We want digits to be the same length as the number of letters, because we want to substitute them one for one–123 for ABC or whatever. We want to ask itertools.permutations to take permutations of the letters taken n at a time where n is the number of letters, the length of the letters. That part's easy.
This part's a little bit more complicated–how do we find the letters. Here's what I did. I used the regular expression, the "re" module. I used the findall function so that says find all letters from A to Z, and I didn't specify here capitals before.
My interpretation and the general rules for this type of cryptarithmetic problems is that they should be capitals, but if you allow lowercase as well, that's fine. Find them all within the formula, make a set of those, because if a letter occurs more than once we don't want multiple versions of it. We just want individual ones. Then join them together into a string.
Now, I should say that I snuck in one piece of code here that you haven't seen before that may seem wild at first. That's the very first line. It says from _ _future_ _ import division. That seems pretty amazing. Wow. Python has a built-in time machine.
What does it mean to import division from the future?
33. Future-Imports
In Python 2.x, version 2.6, 2.7, and so on, if you do integer division in Python 2.7, say, 3 divided by 2 evaluates to 1. And 1 divided by 2 evaluates to 0. The reason is in their wisdom the designers of Python said, well, if you're going integer division, you probably want an integer answer, and we'll do the best we can, and we'll have to truncate to give you the answer.
In Python 3, the designers decided, well, that's really confusing. What you really want when you divide 3 over 2 is 1.5, and when you divide 1 over 2 you want 0.5. That's what Python 3 does, so it says let's change the result from an integer to a floating point number if that's the best you can do.
Now, if you want this kind of behavior in Python 2, then you say from the "_ _future_ _" with two underscores on either side of it import division. I want that because in my cryptarithmetic problems I really prefer this type of answer and not that type of answer.
Now I'm done, and I got a pretty concise program. It followed by plan very nicely and easily. I like this design of the three pieces that I've tried to divide it up into.
Look how simple the solve function is. It just says iterate over all possible ways of filling it in, ask if it's valid, and if it is, return. Can't get much more clear than that.
Valid is pretty simple. Valid is almost like asking is eval f True. If it were that simple, I would just put it in line up here, but it's not quite that simple. I need the try and except, and I need to catch the arithmetic error, and then there's this little trick with the leading zero digits.
I like having a separate function for valid to break that out. Then I like having fill_in as a generator. To make the logic of the main function simple but not to slow me down by requiring me to generate all the possible answers at once, the generator comes up with up the answers one at a time. It's separating out the flow of control from the logic.
Now, am I done at this point? Well, no, because what I have to do next is convince myself that I've got it right.
34. Testing
Unit2-34 Here I've defined some test examples, and here I've written a simple test function.
2 import time
4 examples = """TWO + TWO == FOUR
5 A**2 + B**2 == C**2
6 A**2 + BE**2 == BY**2
7 X / X == X
8 A**N + B**N == C**N and N > 1
9 ATOM**0.5 == A + TO + M
10 GLITTERS is not GOLD
11 ONE < TWO and FOUR < FIVE
12 ONE < TWO < THREE
13 RAMN == R**3 + RM**3 == N**3 + RX**3
14 sum(range(AA)) == BB
15 sum(range(POP)) == BOBO
16 ODD + ODD == EVEN
17 PLUTO not in set([PLANETS])""".splitlines()
19 def test():
20 t0 = time.time()
21 for example in examples:
22 print; print 13*' ', example
23 print '%6.4f sec: %s ' % timedcall(solve, example)
24 print '%6.4f tot.' % (time.time()-t0)
26 def timedcall(fn, *args):
27 "Call function with args; return the time in seconds and result."
29 result = fn(*args)
31 return t1-t0, result
What does it do? It iterates through each of the possible examples. It prints out what the example is, the original formula. Then it uses the time call function that we defined in the previous lesson. It calls solve on the example and gets back the time it took and the possible results and prints that out. Then in the end it prints out the total elapsed time and does that by starting a timer at the start and taking a time at the end and just calculating the difference.
It's going to tell me for each example what the answer is, how long it took for that example, and how long it took for all of them together.
Here is the results of the test:
TWO + TWO == FOUR
0.0576 sec: 734 + 734 == 1468
A**2 + B**2 == C**2
0.0030 sec: 3**2 + 4**2 == 5**2
A**2 + BE**2 == BY**2
0.0326 sec: 5**2 + 12**2 == 13**2
X / X == X
0.0000 sec: 1 / 1 == 1
A**N + B**N == C**N and N > 1
0.0247 sec: 3**2 + 4**2 == 5**2 and 2 > 1
ATOM**0.5 == A + TO + M
0.0037 sec: 1296**0.5 == 1 + 29 + 6
GLITTERS is not GOLD
0.0000 sec: 35499187 is not 3652
ONE < TWO and FOUR < FIVE
0.0580 sec: 351 < 893 and 2376 < 2401
ONE < TWO < THREE
0.0000 sec: 341 < 673 < 62511
RAMN == R**3 + RM**3 == N**3 + RX**3
0.4187 sec: 1729 == 1**3 + 12**3 == 9**3 + 10**3
sum(range(AA)) == BB
0.0001 sec: sum(range(11)) == 55
sum(range(POP)) == BOBO
0.0007 sec: sum(range(101)) == 5050
ODD + ODD == EVEN
PLUTO not in set([PLANETS])
0.0000 sec: 63894 not in set([6315297])
0.8982 tot.
For each example, we see the example printed, then the filled in example found, and you can verify that that is, in fact, correct.
It took a tenth of a second or so. Some of them take less. When there's only one variable it's really fast.
Here we find these Pythagorean triplets of 3-squared plus 4-squared equals 5-squared, and 5-squared plus 12-squared equals 13-squared. Notice that we've gone beyond the most simple of cryptarithmetic problems, which were just word plus word equals another word. Here we can allow division and exponentiation. Over here I've been even more complicated where I've come up with A**N plus B**N equals C**N, and N is greater than 1.
The lowercase letters I have not translated, because I said the only letters that count as being translated by digits are going to be the upper case ones, so I can use any of this Python syntax like the and operator.
Now, if you can find a solution for this when N is greater than 2, I want you to let me know.
But here I found one when N is greater than 1.
I've split the atom. I've determined that all that glitters is not gold.
I've determined that 1 is less than 2 and 4 is less than 5.
You can see a bunch of other equations that I've been able to process, including ODD plus ODD equals EVEN. And for these 14 different examples it took 2 seconds. Not too bad.
35. Find-All-Values
Now I want to ask you one quiz question. What if we wanted to return or to print all values of the filled-in formula rather than just the first one? For some of these formulas there are multiple possibilities of digits to fill in that would all work equally well.
If we wanted to see them all, if we wanted to have them all printed out, what should we do?
Should we change the function fill-in to return a list rather than generate results?
Should we change the single word "return" to "print" in the function solve?
Should we change the function valid to do something else to validate each of these possible formulas?
Or do we need to add a new function to do something that we haven't done before?
35.1. Find-All-Values Answer
The answer is all we need is this one word change. So far we're returning the first value we found, but if we print it, then we can see that and the loop will keep on going, and we can see all the possible Fs–the filled-in formulas that are valid solutions.
36. Tracking-Time
Now if you have Python installed on your own machine, and you have some kind of a terminal or shell in which you can get a command prompt, you can type the command Python, and then give it the -m flag, which says module, and then load what's called the cProfile module– lowercase c, uppercase P. Then the name of your file that you want to profile. I called my cryptarithmetic file crypt.py. Execute that, and you'll get a nice table of where all the time goes.
If you don't have Python installed or you can't run a command like this, you can do it from within Python. What you'll have to do is say import the c profile module, and then do cProfile.run and then a string to be evaluated, which is the code that you want to run. Then you'll see output that looks like this.
In the right had column we see the various functions that are being called.
Within my crypt program there are three main functions–solve, fill in, and valid.
Then within the regular expression module, I was calling search and compile, and then there's various other built-in methods of Python, such as the string maketrans function, the eval function, and so on.
These other columns tell us the number of times each of these functions was called, the total time spent, some percentage–we won't worry about that.
That's time per call.
Then the cumulative time, the total number of times spent there.
Mostly I was just called solve. That was about 75 seconds.
Within that 62.7 seconds when to valid, so that's where most of the time is going.
I should say that most of these results are pretty much what I was expecting. I was a little bit surprised that the re search too so much time–12 seconds out of the 75. I was also a little bit surprised that this maketrans and these other methods, the translate methods, took so little time–just about 3 seconds all together.
If we want to make our program faster, it seems obvious that we'd better look where the time is. Out of that 75 seconds, 63 of it is within valid. That's where we have to look. Of valid, 47 seconds is within eval. If we want to make our program faster, it makes sense to concentrate our efforts on the parts where most of the time is. It's not just a good idea. It's the law.
It's called the Law of Diminishing Returns. The way the law works, if we imagine our total execution time as being this bar here, and if we said that goes up to 75 seconds, and 10 seconds was taken up by fill in, and then all the rest by everything else.
We can see if we want to make things faster, we'd better make this bar shorter or maybe this bar; and it won't help much to make these other bars shorter.
For example, if we made fill in and everything else vanishingly small, then if we didn't touch valid we'd still have an execution time of 63 seconds, even if we could improve these infinitely fast and we wouldn't have helped all that much.
37. Increasing-Speed
Here is a quick quiz on diminishing returns.
Say if we have an execution time that's this long and function A takes up 90 seconds, and function B takes up 10 seconds for a total of 100 seconds execution time. Let's saw we're able to make an improvement to our code. We speed up B by a factor of 10. We make B ten times faster.
The question is how much faster do we make the overall execution of our whole program if B is made 10 times faster?
Do we make the whole program 9 times faster,
These are approximate number, not exact numbers.
37.1. Increasing-Speed Answer
The answer is even though we sped up B by a factor of 10, we've only made the whole program 1.1 times faster– that is, 10% faster. The problem is we've left A unchanged, so our new program would have an execution time that looks like this. B would be this tiny little slice here. It would just take 1 second rather than 10 seconds, but the whole program would still take 91 seconds.
38. Rethinking-Eval
If we want to make our program faster effectively, we'd better concentrate on the eval function, because that's taking up about 47 out of our 75 seconds or about 63% of our time.
The problem is that eval is a built-in function. We can't go editing eval to try to make it faster, but if we can't touch eval itself, the only choices we have are we could make fewer calls to eval. We call it fewer times. We'll spend less time in it.
Or we can make easier calls to eval. Pass it an argument that's easier for it to evaluate. Let's concentrate on easier first. Do you see a way to break up the problem of evaluating a formula into smaller pieces in such a way that we could make the resulting program, say, What do I mean by making it easier or breaking it up into pieces? We could do eval of ODD plus ODD and then do eval of EVEN and do those separately rather than do them all together in one equation. Of course, I'd probably substitute in the numbers here rather than eval the letters themselves. That might be one way of breaking it up into smaller pieces. Often this idea of divide and conquer is a good idea for program design.
Do you think that that approach would work here to make the calls to eval easier so that we could cut down on this 47 second execution time.
I guess this is an matter of opinion. You might have some ideas. I might have different ideas, so don't worry if you agree with me too much, but think about it and give me your answer.
38.1. Rethinking-Eval Answer
Well, my answer was no. I couldn't think of a good way to break these up that would make the calls easier and still get the job done. If you've come up with a good way, great. I want to hear about it. Let's talk about it in the forums.
39. Making-Fewer-Calls
But if we can't figure out a way to make the calls easier, then we're going to have to concentrate on having fewer calls.
Let me write down some possibilities for fewer calls and see which ones you think make sense.
One possibility would be for each equation, like ODD plus ODD equals EVEN, rather than evaluating them all N factorial times, maybe we could combine all of those into one big equation. Certainly that would result in fewer calls if we could figure out a way to do that and still get the right answer.
Another way might be to, say, could we figure out a way to fill in one digit at a time? Rather than do all N factorial permutations of digits, if we could do one digit, see if that works, and if it does then do the next digit. That would certainly help us do a smaller number of calls. That's the approach we took with the zebra function where we started out by doing all the permutations, and then we figured out let's go through and if there's a contradiction, let's stop and not do the remaining ones. The question is can we figure out a way to take that general approach for this problem.
Then the third approach would be to eval the formula once but eval it as a function with parameters. To do all the work of figuring out how to understand number plus number equals number, do that just once and then call that function repeatedly with all the permutations. There'd still be lots of calls to the function, but there'd be fewer calls to eval.
Tell me which one of these three do you think is the most promising. Again, it's a matter of opinion. You might have better ideas, but I want you to think about it and tell me what you think.
39.1. Making-Fewer-Calls Answer
My feeling was that this was the most promising.
I couldn't figure out how to fill in one digit at a time and make that work, because if we had something like the word EVEN, and we fill in 1 digit, and we have, say, 8V8N, I didn't figure out how to make sense of that. We have this mixture of some letters that are filled in and some that aren't. That didn't work for me.
This approach, one big formula–yes, we could figure out a way to make one big formula, but the problem is I don't think that would be any faster, because it would take eval a long time because it would be a very long formula.
This approach I think is the most promising, so let's talk about it.
40. Lambda
Let's consider a formula–I'm going to write a new one. Let's say You equals Me-squared. We're treating these formulas as strings. Now when we substitute numbers into this, we get something like 123 equals 45-squared.
What happens when we call eval on this string? What eval has to do is it takes it's input, which is a string, and then it has a process of parsing that string into a parse tree. The parse tree would say something like this is a comparison, and it has an operator, which is a number, which is 123, and another operand, which is an expression which has the number 45, and then the exponentiation operator, and then the number 2. Python builds up a data structure that looks something like that.
There's another operation of code generation in which Python takes this tree and says in order to evaluate this tree I'm going to do something like load the number 123 and then load the number 45 and then do an operation on that and so on and so on and then return the result.
That's a lot of work to build up this tree, generate the code, and then finally, the final operation after we've come up with this, is to execute this code and come up with an answer, which in this case would be false.
Now, this is a lot of work for eval to do, and it seems like there's a lot of duplicate work, because we're going to do this for every permutation of the digits, but each time we go through this part of the work, the parsing, is going to look exactly the same with the exception of the specific numbers down here at the bottom, but the rest of the parse tree is going to look the same. Similarly, this part of the work, the code generation, will also look the same except these numbers will differ. We're going to have to repeat that over and over again.
What we'd like is an approach where we can only do these two parts once and then pass in the specific numbers and then get our final result back.
But one thing I should say is that the eval function doesn't take a statement, like this function definition. It takes an expression. Furthermore, we don't really need this name F. That was kind of arbitrary. We'd be fine with having an expression that represents a function but doesn't have a name associated with it. Then we can pass that directly to eval.
It turns out there is a way to do that in Python. There is a keyword that creates a function as an expression. That keyword is lambda. It's kind of a funny keyword. I would've preferred them to use function or fun or maybe just def and leave the name out, but they chose lambda. It's based on the lambda calculus, the Greek letter λ, which is a branch of formal mathematics defining functions.
The syntax is fairly similar to the syntax of a definition of a function, except we leave the name out. It's an anonymous function with no name. It also turns out for some reason we leave out the parentheses. We just say lambda Y, O, U, M, E, then a colon, and we leave out the return, and then we just put in the rest of the code–100 times blah, blah, blah, blah.
41. Compile-Word
Here's an example of how it all works. This I've actually typed into the Python interpreter. I've defined a function f as a lambda expression. It looks like this.
Then I've asked what F is. All Python prints out is it says that it's an object of type function, which doesn't have a name other than the lambda and the address and memory where it's stored.
Then I'd say, is F true of the sequence 1, 2, 3, 4, 5. That is, Y is equal to 1, M is 2, E is 3, U is 4, and O is 5. The answer is no, it's False. This is not equal.
Then I asked is it true for 2, 1, 7, 9, 8. Yes, that's true. The reason is because it works out to this expression 289 equals 17-squared.
Now I'm thinking of a design where we have some type of a solve function that's going to solve a formula, and we're going to have a compile formula function that's going to take a string formula as input and return a lambda expression function as the result of compiling the formula.
As part of that, I want to have a function that I'm going to call compile_word, It's going to take a word like ME and compile that into something like 10M plus E. You could have some variation on exactly how you want to express that.
It will also take a word like equals and compile that into itself, into equals, and a word like 2 and compile that into 2 itself.
This is the function I want you to write.
Compile_word where compile_word of YOU is something like this– don't worry about the communicativity and associativity. You can write this any way you want as long as it is a code that would compute the right answer. It's important to put parentheses around it. Anything that's not an uppercase word you should leave alone.
41.1. Compile-Word Answer
Here is what I did.
I used the isupper method of the word, which is a string. String.isupper method to check if it is in fact an uppercase word. If it is, then I enumerate all the letters in the word.
I reverse it, so this slice says reverse it.
The missing numbers on either side says take the whole word, and the -1 says go from the back forward, so reverse the word, enumerate it give me pairs of indexes from 0 to N along with the individual digits.
Then I'm going to just say 10 to the ith power and the digit.
That gives us 1 times U plus 10 times O plus 100 times Y.
It works out backwards from the normal YOU order, but that doesn't matter.
I take those results, I put a plus sign in between them, and I wrap parentheses around them.
If it's not upper, I just return the word itself.
Now I explained the whole program.
I'm calling it faster_solve, and I take a formula.
The first thing I do is compile the formula. That gives me back the function that represents it, and while I'm there, I'm also returning the letters that are within the formula. That evaluation or compilation of the formula is done once.
Then I go through all the permutations of the digits taken length of letters at a time. Notice that I'm using actual integers here, not strings, for the individual digits.
If applying the function to the digits is true, then I did the same thing that I did before of making up a translation table and calling the formula.translate to get my final result. Note I only have to do this translation once.
On the specific permutation that I know works, I don't have to do it every time like I did in the previous version of solve. Then if there's an arithmetic error, I catch that and go on to the next permutation.
Now, the function compile_formula takes a formula as input. First I figure out all the letters, so that's going to be the Y, M, E, U, and O. Then I figure out that the list of parameters I want to put into my function is going to be a string consisting of the letters with a comma in between them.
I figure out all the tokens. A token is an individual term like the U or the equal signs or the ME and so on.
42. Speeding-Up
Now let's take a look at what we did here.
So we had re.split and then this regular expression and then out of the formula. So what does this say? This says I want to split up the original formula, and the way I'm splitting it up is I'm taking sequences of the characters A-Z in a row. The plus means one or more, and the parentheses means keep that result.
So when I have YOU = ME squared, this part and this part will match, and they'll be returned as individual elements in this split, and so will the other parts. Re.split will return the list consisting of the parts that it found in the split plus everything in between. And it will be this type of list, and then I'm going to map compile word– it didn't quite fit on one line– to re.split, and that will give me– these tokens will remain unchanged, and these will be converted into the form that multiplies out the digits. Then I'm going to assign that to the variable tokens. Now I've got my tokens.
The body of the function we just formed by concatenating all the tokens together. If I wanted to I could put spaces between them; it doesn't matter. And the function is lambda created with the parameters in the body, and then I just return the evaluation of the function, so that complies the function, does it only once rather than once per permutation.
And I return the letters that I found.
As a convenience, because when you're debugging this function you may want to say, "Well, what did this function look like?" "What did I come up with? What did I come up with?" I have an optional parameter here that says if you pass in verbose equals true, then it just prints out what it found. It just makes it a little bit easier to debug this function.
And now, when I run profiling again, look what I get. The time has sped up quite a bit on the simple list of 14 examples that we saw before. This version of the code is 25 times faster. On a more complex list, it's only 13 times faster. But it's at least an order of magnitude faster.
And we did that because we eliminated calls to eval. So we found the bottleneck. We found that eval was the function that was taking up the most amount of time in the previous profiling. We figured out a way to call eval less often by precompiling, and now we've done away with that difficulty.
43. Recap
Let's do a recap of what we learned in this unit.
First, we talked about some Python features that were maybe new to you.
We used complex list comprehensions. That's something like x-squared for x in blah, blah, blah, if something.
We showed generator expressions, and that's similar but with parentheses.
We talked about generator functions or just generators, and we recognize those with the yield statement.
We talked about the idea of handling different types. This has the fancy name of polymorphism, meaning different forms.
We saw an example of that in timed calls where we said that the input n– and then there are other inputs there–n could be either an integer, in which case we would do one thing, or a float, in which case we would do something else. We checked which is which by using "isinstance."
We talked about the eval function and how we can use that to map from a string to a Python object, which is the result of evaluating the string.
In particular, the case of evaluating to a function. Eval is a way of making the computation be done once and getting all that work over with so that we can then use that work repeatedly.
We also talked about instrumentation, and we did timing with the time.clock method that's built in, and then we built up timedcall and timedcalls routine.
And we talked about counting number of invocations of functions or assignment statements or whatever. There we came up with our own routine that we called c.
I guess I should say a little bit about variable naming conventions. Why did I use a short name like c here, whereas other places I had long, more expressive names? I guess the reason is c was used only for debugging purposes. It wasn't intended to be part of the final part of the program. I felt justified in having that be short, because I was going to be typing it and deleting it frequently. Things that are going to persist for longer have longer names
For example, it's fine to say "for i in range something" where there we know that i is an index integer, and it only persists over this short loop. It's okay to have a short name. If something lasts longer, we probably want it to have a longer, more descriptive name.
44. Zebra Puzzle Code with Additional Output
1 def find_water_zebra():
4 houses = [first, _, middle, _, _] = [1,2,3,4,5]
6 orderings = list(itertools.permutations(houses)) #1
8 def imright(h1,h2):
9 "House h1 is immediately right of h2 if h1=h2 = 1"
10 return h1-h2 == 1
12 def nextto(h1,h2):
13 "Two houses are next to each other if they differ by 1"
14 return abs(h1-h2) == 1
16 return [result for result in (
17 ( ('Drinks', {'coffee':coffee,'tea':tea,'milk':milk,'WATER':WATER,'oj':oj}),
18 ('Nations', {'Englishman':Englishman, 'Spaniard':Spaniard,
19 'Ukranian':Ukranian, 'Japanese':Japanese, 'Norwegian':Norwegian}),
20 ('Colours', {'red':red, 'green':green, 'ivory':ivory, 'yellow':yellow, 'blue':blue}),
21 ('Pets', {'dog':dog, 'snails':snails, 'fox':fox, 'horse':horse, 'ZEBRA':ZEBRA}),
22 ('Smokes', {'OldGold':OldGold, 'Kools':Kools, 'Chesterfields':Chesterfields,
23 'LuckyStrike':LuckyStrike, 'Parliaments':Parliaments}),
46 ]
49 hps = find_water_zebra()
50 print 'Result count:',len(hps)
51 if len(hps) < 20:
52 for hp in hps:
53 print 'House ',".join('%15s'%(num if num else 'House') for num in range(1,6))
54 for item, props in hp:
55 propKeys = sorted(props.keys(),key=lambda k:props[k]) # sort on props[k]: house
56 print '%-7s'%item, ".join(['%15s'%propKey for propKey in propKeys])
The result of running this code is:
Result count: 1
House 1 2 3 4 5
Drinks WATER tea milk oj coffee
Nations Norwegian Ukranian Englishman Spaniard Japanese
Colours yellow blue red ivory green
Pets fox horse snails dog ZEBRA
Smokes Kools Chesterfields OldGold LuckyStrike Parliaments
1. 1 Introduction
Hi. Welcome back.
As you can see from this spread, this lesson is going to be about tools. Now, we homo sapiens fancy ourselves as tool makers. We think that's what distinguishes us from the other animals. In the physical world, we've invented lots of tools. These general ones like screw drivers and pliers, you're probably very familiar with, and some very specific tools — look at this one. You may not know what this does, but look how nicely it fits into my hand. Or look at this one. What do you think this one does? If somebody can tell me on the forum what this one does, I'll send them a swell prize. Or look at this one — surgeons forceps. Now, these have been refined over the years and now we can refine them where they serve as a third hand. It's got this trick where I can attach it on, and now I don't have to put my hand on it anymore, because it's got a little locking mechanism.
So our tools get better and better over time.
Now, there's a saying that a poor craftsman blames his tools. I always thought that all that meant was you shouldn't complain, you shouldn't whine, but I realized it means more than that. It means that the tools become a part of you, and if you're a good craftsman then the tools are a part of you. You know what to do with them, and so you're not going to end up blaming your tools. That's what we're going to learn in this lesson about software tools.
We're going to learn about some general software tools, about some very specific software tools, and specifically, we're going to talk about two.
We're going to talk about, first, language. Now, language is perhaps homo sapien's greatest invention–our greatest tool. In computer programming, well, a computer program is written in a language, but a computer program can also employ language as a tool to do its job.
The other tool we're going to talk about is functions. Of course, you've been using functions all along, but we're going to talk about new techniques for using functions and learn why they're more general and malleable than other tools. So see you in the lesson.
2. 2 Language
You already know one language we've been using–the Python language obviously. That has a syntax for statements and expressions, but there are also some subparts of Python that seems like quite a different language. One example would be the language for formatting strings. Here's an example from the Python reference language.
1 print '%(language)s has %(#)03d quote types.' % {
2 'language': "Python", "#": 2}
You see that this rather bizarre string here with %(#)03d and so on. There's a language of format strings itself to produce this type of output.
There are other parts of the Python language that allow you to customize it to your use.
You can create your own type of classes, and they can have their own type of expression, including operator overloading. We're used to expressions like x + y. Say if x is 2 and y is 3 then that would be 5.
But you can also say x + y where x and y are your own types of data types. They might be matrices. They might be components of some kind of building that you're building, and this can say to put them together, it may be two shapes that you can concatenate on the screen. You can define your own language over the types of objects you want to deal with.
You can go one step farther than that and define your own domain specific language.
This is a language for describing an optimization problem having to do with the price on octane and various types of fuels, and you can describe the parameters and then build the language processor to take this as input and compute the output.
Of course, you could do that with regular Python statements just as well, but here what we've done is design the language specifically for this problem and then written the problem description in that language.
In this unit we'll cover
what is a language,
what is a grammar,
the difference between a compiler and an interpreter, and
how to use languages as a design tool.
3. 3 Regular Expressions
We'll start with a language that many of you may be familiar with — the language of regular expressions.
You've seen them if you've taken CS101. Maybe you've seen them elsewhere. In any event, we'll give an overview of them.
There's a language for regular expressions. They can be expressed as strings.
For example, the string a*b*c* describes a language, and that language consists of any number of a's followed by any number of b's followed by any number of c's. Elements of this language include the strings abc, aaa, bcc.
Stars can be any number, so it could be zero of them. Say, just b would be an example. The empty string all by itself would be an example. ccccc would be an example. An so on.
Now, there's a whole language of symbols like + and ? and so on for regular expressions. To make sense of them, we have to be able to describe what are the possible grammars and then what are the possible languages that those grammars correspond to.
A grammar is a description of a language, and a language is a set of strings.
Now, this form of description of the grammar as a long sequence of characters is convenient when you're quickly typing something in, but it can be difficult to work with. Grammar expressions get long. So we're going to describe the possible grammars in a format that's more compositional.
In other words, what I'm going to describe is an API, which stands for application programming interface. This is the interface that a programmer uses rather than the UI or user interface that a user uses when you click with your mouse.
We'll describe a series of function calls that can be used to describe the grammar of a regular expression. We'll say that a regular expression can be built up by these types of calls.
First, a literal of some string 's'. For example, if we say lit('a') then that describes the language consisting of just the character string "a" and nothing else.
We have the API call seq(x, y). We could say seq(lit('a'), lit('b')), and that would consist of just the string "ab." So far not very interesting.
Then we could say alt(x, y). Similarly, alt(lit('a'), lit('b')), and that would consist of two possibilities — either the string "a" or the string "b." We'll use the standard notation for the name of our API call here.
star(x) stands for any number of repetitions — zero or more. star(lit('a')) would be the empty string or "a" or "aa" and so on.
We can say oneof(c) and then string of possible characters. That's that same as the alternative of all the individual characters. oneof('abc') matches "a" or "b" or "c." It's a constrained version of the alt function.
We'll use the symbol "eol," standing for "end of line" to match only the end of a character string and nowhere else. What matches is the empty string, but it matches only at the end. The only example we can give is "eol" itself, and we can give an example of seq(lit('a'), eol), and that matches exactly the "a" and nothing else at the end.
Then we'll add dot, which matches any possible character — a, b, c, any other character in the alphabet.
In the chart below, s is a string (representing something that must be matched exactly), c is also a string (but represents a set of characters; we match any one of them), and x and y are patterns (that is, regular expressions) formed by any of the elements in the chart.
Grammar Example Language
lit(s) lit('a') {a}
seq(x,y) seq(lit('a'), lit('b')) {ab}
alt(x,y) alt(lit('a'),lit('b')) {a,b}
star(x) star(lit('a')) {",a,aa,aaa,…}
oneof(c) oneof('abc') {a,b,c}
eol eol {"}
seq(lit('a'), eol) {a}
dot dot {a,b,c,….}
4. 4 Specifications
Now, of course Python has a regular expression module called the "re module". If you really want to use regular expressions, you import re and get started.
The point here is not so much to use regular expressions as to show you what it's like to build a language processer. So to get started doing that, I started by writing a test function that defines some regular expressions using this API.
1 def test_search():
2 a, b, c = lit('a'), lit('b'), lit('c')
3 abcstars = seq(star(a), seq(star(b), star(c)))
4 dotstar = star(dot)
5 assert search(lit('def'), 'abcdefg') == 'def'
6 assert search(seq(lit('def'), eol), 'abcdef') == 'def'
7 assert search(seq(lit('def'), eol), 'abcdefg') == None
8 assert search(a, 'not the start') == 'a'
9 assert match(a, 'not the start') == None
10 assert match(abcstars, 'aaabbbccccccccdef') == 'aaabbbcccccccc'
11 assert match(abcstars, 'junk') == "
12 assert all(match(seq(abcstars, eol), s) == s
13 for s in 'abc aaabbccc aaaabcccc'.split())
14 assert all(match(seq(abcstars, eol), s) == None
15 for s in 'cab aaabbcccd aaaa-b-cccc'.split())
16 r = seq(lit('ab'), seq(dotstar, seq(lit('aca'), seq(dotstar, seq(a, eol)))))
17 assert all(search(r, s) is not None
18 for s in 'abracadabra abacaa about-acacia-flora'.split())
19 assert all(match(seq(c, seq(dotstar, b)), s) is not None
20 for s in 'cab cob carob cb carbuncle'.split())
21 assert not any(match(seq(c, seq(dot, b)), s)
22 for s in 'crab cb across scab'.split())
23 return 'test_search passes'
First I defined a, b, and c as being these literals. Then I combine some more complicated examples. Then I showed some example of assertions, of what we want to be true.
We've defined two functions that match the functions in the re module. The first function is called search. It takes a pattern and a text, and in the regular expression module this function returns something called a "match object."
What we'll have it do is return a string which will be the earliest match in the text for the pattern, and if there is more than one match at the same location, it'll be the longest of those.
For example, search('def', 'abcdef') would return 'def' because it's found there.
The next function is called match. It also takes a pattern and a text and returns the string that matches. But for match, the match must occur at the very start of the string.
match('def', 'abcdef') would return None, indicating that there is no match. But match('def', 'def fn(x):') in this string that has 'def' at the beginning would return that match.
Here are some examples of the types of things that I want to be able to do. That gives me the start of a specification for how I want to write my search and match functions.
5. 5 Concept Inventory
I always like to start out with an inventory of the concepts that are going to be used.
So far we have the notion of a pattern or a regular expression, of a text to match against, the result, which will also be a string of some kind. It doesn't seem like there's all that many more concepts. One thing that it looks like we'll need is some notion of a partial result, and some notion of control over the iteration.
text — result
partial result
control over iteration
What do I mean by that?
Well, some of the matches are easy. If we search for a literal 'def' within 'abcdef,' it's easy to imagine just going down the line of this string and saying does it match here? No. Here? No. Here? No. Here? Yes. We'll return that result.
But what if we're matching with the pattern — let's say we have the expression 'a*b+' — any number of a's followed by one or more b's?
In our API notation, we would write that as seq(star(lit('a')), plus(lit('b'))).
Let's say we're matching that against the string 'aaab'.
Now if we had a control structure that says sequence, look to match the first, and then look at the second, and if the first — star(lit('a')) — only had one possible result, then it would say, yes, it matches here right at the start, now look for something after that.
Does it match plus(lit('b'))?
I've got to have iteration over the possible results.
I have to say that star(lit('a')) can match in more than one location.
It can match with zero instances of a, with 1, with 2, with 3, and it's only after 3 thatthen we can match the second part, find the b, and then find that the whole expression matches.
That's going to be the tricky part — worrying about this control when one part of a sequence doesn't match or similarly when we have an alternative between two possibilities — a or b or alt(a, b).
This trickiness seems like it's going to be difficult, but it all resolves itself after wemake one good choice.
It takes some experience to see what that choice can be, but if we decide to represent these partial results as a set of remainders of the text, then everything falls into place.
What do I mean by remainder?
I mean everything else after the match.
If we match a literal a the remainder after we match zero characters of this string would be aaab — three a's followed by a b.
The remainder after we match one a would be two a's followed by a b and so on.
What I'm going to do is define an auxiliary function called match_set, and it takes a pattern and a text, and it returns this set of remainders.
When given just this pattern here as the input, star(lit('a')), and this text as the text then the remainder would be the set consisting of three a's and a b, two a's and a b, one a and a b, or just b.
In other words, star(lit('a')) could have consumed 0, 1, 2, or 3 a's, and that's the remainder that's left over.
So the result will be this set: {aaab, aab, ab, b}.
6. 6 Matchset
Here's the code:
1 def matchset(pattern, text):
2 "Match pattern at start of text; return a set of remainders of text."
3 op, x, y = components(pattern)
4 if 'lit' == op:
5 return set([text[len(x):]]) if text.startswith(x) else null
6 elif 'seq' == op:
7 return set(t2 for t1 in matchset(x, text) for t2 in matchset(y, t1))
8 elif 'alt' == op:
9 return matchset(x, text) | matchset(y, text)
10 elif 'dot' == op:
11 return # Enter code here
12 elif 'oneof' == op:
14 elif 'eol' == op:
15 return set(["]) if text == " else null
16 elif 'star' == op:
17 return (set([text]) |
18 set(t2 for t1 in matchset(x, text)
19 for t2 in matchset(pattern, t1) if t1 != text))
21 raise ValueError('unknown pattern: %s' % pattern)
It's just 20 lines or so.
I've left out a couple pieces to give you something to do, but let me first try to explain how it works.
We take a pattern and a text. We're breaking down the pattern into its components. The components are an operator and then and x and a y part, depending on the definition.
For example the literal will only have an x component. Sequence and alternative will have both and x and a y. We'll see how that component is written later, but for now just assume it's going to pick out the right pieces. Then we decide which operator we have, literal, sequence, alt, and so on, and return a proper set accordingly.
For example, if the operator is asking for literal string x, we ask does the text start with x? If it does, then the remainder will be a singleton set, a set of just one element, which is the remainder of the text after we've broken off the length of text characters.
If we matched a three-character sequence for x, we return the text without the first three characters.
If x was 1 character, we return the text without the first character. Otherwise, we return the null set. There's no match.
The alternative is particularly nice. It says if we have an alternative between x and y we just return the union of the those two match sets. This character, the or bar, means "union" in Python, the union of two sets.
Sequence is a complicated one. It says if we're asked for a sequence of x and y, we first find a matching set for x. That's going to be a set of possible remainders, and we go through all of those. Then take the remainder of that text, which would be t1. Then match y against that remainder. For each of those alternatives, that's what we're going to return in this set.
For example, if this is our pattern — seq(star(lit('a')) plus(lit('b'))) — it looks like that, and we're matching against "aaab" then the x is the star(lit('a')), and y is the plus(lit('b')), and the matchset for x is this set here ({aaab, aab, ab, b}).
We try to match y, the plus(lit('b')), against all of these match sets, and it'll fail to match against each of these three (aaab, aab, ab). It will match against this one (b), and so now we have a match that consumes the entire string. The result from the match of this sequence of x and y will be the set consisting of just the empty string, because we've matched off all the a's and one b, and there's no remainder left over.
Note that there's a big difference between the outcome of saying here's a result consisting of one string, the empty string, versus the set that is the empty set. The empty set is a failed match, and the set consisting of the empty string is the successful match.
Now let's see if you can fill in your code for these two missing cases here.
Remember, you're going to be returning a set of possible remainders if the match is successful.
7. 6 Matchset (Answer)
Here's my answer. A dot matches any character in the text. If the text has at least one character, then there's going to be a match, and the match is going to have a remainder, which is all the rest of the text.
That remainder is a set, and it's the set of one element, and the element is the text from the first character on. In other words, dropping off the 0th character. We're going to return that if there is any text at all. That is, if the text is not empty. Otherwise, if the text is empty then we're going to return the null set. We defined the null set down here.
How about for oneof? Oneof takes a string of possible characters and what it should return is similar to dot. It should return the remaining characters if the first character is equal to one of the characters in x. We're going to return exactly the same thing, a set consisting of a signal answer which is the original text with the first character dropped off, and we're going to return that. What I'd like to be able to say if the text starts with any of the characters in x. It turns out that I can actually say exactly that — text.startswith(x), if I arrange to have x be a *tuple* of characters rather than a character string.
Here I have the documentation from the Python manual for the string that starts with method, and it says it's true if the string starts with a prefix, so we can ask does some string start with a particular string — yes or no? But prefix can also be a *tuple* of prefixes to look for. All we have to do is arrange for that x to be a tuple, and then it automatically searches for all of them.
What I'm saying is I want the API function oneof('abc'). That should return some object, but we're not yet specifying what that object is, such that the operator of that object is oneof, and the x for that object should be the tuple ('a', 'b', 'c'). It's sort of a little optimization here that I've defined the API to construct an object of just the right form so that I can use it efficiently here. Just go ahead and check does the text start with any one of the possible x's. Otherwise, return no. If you didn't know about this form of text.startswith, you could have just checked to see if any of the character c for c in x. We'd say return the rest of the string if any of the characters in x if the text starts with any one of those characters. Otherwise, return "no."
8. 7 Filling out the API
This looks like a good time to define the rest of the APIs for the constructors for patterns. Here I've defined literal and sequence to say they're going to return tuples where the first element is the operator, the second — if it exists — is x, and then the final — if it exists — is the y. Those are for the components that take an operand or two, but dot and end of line don't, so I just defined dot to be a one element tuple. See if you can go ahead and fill in the remainder of this API.
9. 7 Filling out the API (Answer)
Here are the definitions I came up with. Note that I decided here to define plus and opt in terms of things that we had already defined — sequence and alt and star and lit. You could've just defined them to be similar to the other ones if you prefer that representation. It's just a question of whether you want more work to go on here in the constructors (plus and opt) for the patterns or here in the definition of match_set.
10. 8 Search and Match
Let's just review what we've defined in terms of our API. We have a function match and a function search, and they both take a pattern and a text, and they both return a string representing the earliest longest match.
match(p, t) -> '…'
search(p, t) -> '…'
But for match the string would only return if it's at the start of the string. For search, it'll be anywhere within the string. If they don't match, then they return None.
We've also defined an API of functions to create patterns — a literal string, an alternative between two patterns x and y, a sequence of two patterns x and y, and so on. That's sort of the API that we expect the programmer to program to. You create a pattern on this side and then you use a pattern over here against a text to get some result. Then below the line of the API — sort of an internal function — we've defined matchset, which is not really designed for the programmer to call it. It was designed for the programmer to go through this interface (match and search), but this function is there. It also takes a pattern and a text. Instead of returning a single string, which is a match, it returns a set of strings, which are remainders. For any remainder we have the constraint that the match plus the remainder equals the original text. Here I've written versions of search and match. We already wrote matchset. The one part that we missed out was this component that pulls out the x, y and op components out of a tuple. I've left out two pieces of code here that I want you to fill in.
11. 8 Search and Match (Answer)
Let's do match first. Match interfaces with the matchset, finds the set of remainders. If there is a set of remainders, then it finds the shortest one.
The shortest remainder should be the longest text, and then we want to return that.
We want to return the text, and it's a match not a search, so the match has to be at the beginning of the text, and it would go up to match. So we match from the beginning of the text. How far do we want to go? Well, everything except the remainder.
How much is that? Well, we can just subtract from the length of the text the length of the shortest, and that gives us the initial piece of the longest possible match.
Here search calls into match. What we do is we start iterating. We start at position number zero. If there is a match there for the text starting at position zero, then we want to return it. If not, we increment i to 1, and we say does the text from position 1 on — is there a match there and so on. We just keep on going through until we find one.
Here what we want to say is if the match is not None, then return m. Notice that it would be a bad idea to say if m return m. Normally it's idiomatic in Python to say that if we're looking for a true value. But the problem here is that the empty string we want to count as a true value. A pattern might match the empty string. That counts as a match, not as a failure. We can't just say if m because the empty string counts as false. We have to say if m is not None.
12. 9 Compiling
Let's quickly summarize how a language interpreter works.
For regular expressions we have patterns like (a|b)+, which define languages. A language is a set of strings like {a, b, ab, ba, …} and so on, defined by that pattern.
Then we have interpreters like matchset, which in this case takes a pattern and a text and returns a list of strings or a set of strings.
So we saw that matchset is an interpreter because it takes a description of the language, namely a pattern as a data structure, and operates over that pattern. Here's the definition of matchset.
You see it looks at the pattern, breaks out its components, and then the first thing it does is this big case statement to figure out what type of operator we have and to do the appropriate thing.
There's an inherent inefficiency here in that the pattern is defined once, and it's always the same pattern over a long string of text and maybe over many possible texts.
We want to apply the same pattern to many texts. Yet every time we get to that pattern, we're doing this same process of trying to figure out what operator we have when, in fact, we should already know that, because the pattern static, is constant.
So this is kind of repeated work. We're doing this over and over again for no good reason.
There's another kind of interpreter called a "compiler" which does that work all at once. The very first time when the pattern is defined we do the work of figuring out which parts of the pattern are which so we don't have to repeat that every time we apply the pattern to a text.
Where an interpreter takes a pattern and a text and operates on those, a compiler has two steps. In the first step, there is a compilation function, which takes just the pattern and returns a compiled object, which we'll call c. Then there's the execution of that compiled object where we take c and we apply that to the text to get the result.
Here work can be done repeatedly every time we have a text.
Here the work is split up. Some of it is done in the compilation stage to get this compiled object. Then the rest of it is done every time we get a new text. Let's see how that works.
Here is the definition of the interpreter. Let's focus just on this line here:
This says if the op is a literal, then we return this result.
The way I'm going to change this interpreter into a compiler is I'm going to take the individual statements like this that were in the interpreter, and I'm going to throw them into various parts of a compiler, and each of those parts is going to live in the constructor for the individual type of pattern.
We have a constructor — literal takes a string as input and let's return a tuple that just represents what we have, and then the interpreter deals with that.
Now, we're going to have literal act as a compiler. What it's going to do is return a function that's going to do the work.
1 def lit(s): return lambda text: set([text[len(s):]]) if text.startswith(s) else null
What is this saying?
We have the exact same expression here as we had before, but what we're saying is that as soon as we construct a literal rather than having that return a tuple, what it's returning is a function from the text to the result that matchset would have given us.
13. 10 Lower Level Compilers
We can define a pattern — let's say pattern is lit('a').
1 >>> pat = lit('a')
Now what is a pattern? Well, it's a function. It's no longer a tuple.
1 >>> pat
2 <function <lambda> at 0x101b7bd70>
We can apply that pattern to a string and we get back the set of the remainders.
1 >>> pat('a string')
2 set([' string'])
It says, yes, we were able to successfully parse a off of a string, and the remainder is a string.
We can define another pattern. Let's say pattern 2 equals plus(pat).
1 >>> pat2 = plus(pat)
2 >>> pat2
3 <function <lambda> at 0x101b7bcf8>
Pattern 2 is also a function, and we can call pattern 2 of let's say the string of five a's followed by a b.
1 >>> pat2('aaaaab')
2 set(['b', 'ab', 'aaab', 'aaaab', 'aab'])
Now we get back this set that says we can break off any number of a's because we're asking for a and the plus of that and the closure of a. These are the possible remainders if we break off all of the a's or all but one or all but three, and so on. Essentially we're doing the same computation that in the previous incarnation with an interpreter we would have done with:
1 >>> matchset(pat2, 'aaaab')
Now we don't have to do that. Now we're calling the pattern directly. So we don't have matchset, which has to look at the pattern and figure out, yes, the top-level pattern is a plus and the embedded pattern is a lit. Instead the pattern is now a composition of functions, and each function does directly what it wants to do. It doesn't have to look up what it should do.
In interpreter we have a way of writing patterns that describes the language that the patterns below to. In a compiler there are two sets of descriptions to deal with. There's a description for what the patterns look like, and then there's a description for what the compiled code looks like.
Now, in our case — the compiler we just built — the compile code consists of Python functions. They're good target representations because they're so flexible. You can combine them in lots of different ways. You can call each other and so on. That's the best unit that we have in Python for building up compiled code.
There are other possibilities. Compilers for languages like C generate code that's the actual machine instructions for the computer that you're running on, but that's a pretty complicated process to describe a compiler that can go all the way down to machine instructions. It's much easier to target Python functions.
Now there's an intermediate level where we target a virtual machine, which has its own set of instructions, which are portable across different computers. Java uses that, and in fact Python also uses the virtual machine approach, although it's a little bit more complicated to deal with. But it is a possibility, and we won't cover it in this class, but I want you to be aware of the possibility.
Here is what the so-called byte code from the Python virtual machine looks like. I've loaded the module dis for disassemble and dis.dis takes a function as input and tells me what all the instructions are in that function.
Here's a function that takes the square root of x-squared plus y-squared.
1 >>> import dis
2 >>> import math
3 >>> sqrt = math.sqrt
4 >>> dis.dis(lambda x, y: sqrt(X ** 2 + y ** 2))
5 1 0 LOAD_GLOBAL 0 (sqrt)
6 3 LOAD_GLOBAL 0 (x)
7 6 LOAD_CONST 1 (2)
8 9 BINARY_POWER
9 10 LOAD_FAST 1 (y)
10 13 LOAD_CONST 1 (2)
11 16 BINARY_POWER
12 17 BINARY_ADD
13 18 CALL_FUNCTION 1
14 21 RETURN_VALUE
This is how Python executes that. It loads the square root function. It loads the x and the 2, and then does a binary power, loads the y and the 2, does a binary power, adds the first two things off the top of the stack, and then calls the function, which is the square root function with that value, and then returns it.
This is a possible target language, but much more complicated to deal with this type of code than to deal with composition of functions.
14. 11 Alt
Let's get back to our compiler.
1 def matchset(pattern, text)
Again, in matchset I pulled out one more clause. This is a clause for sequence, and this is what we return. If I want to write the compiler for that sequence clause, I would say let's define seq(x, y).
1 def seq(x, y): return lambda text: set().union(*map(y, x(text))
It's a compiler so it's going to return a function that operates on x and y, take as input a text and then returns as result. We could take exactly that result. While I'm moving everything to this more functional notation, I decided let's just show you a different way to do this. This way to do it would be fine, but I could have the function return that. But instead, let's have it say what we're really trying to do is form a union of sets. What are the sets? The sets that we're going to apply union to. First we apply x to the text, and that's going to give us a set of remainders. For each of the remainders, we want to apply y to it. What we're saying is we're going to map y to each set of remainders. Then we want to union all those together. Now, union, it turns out, doesn't take a collection. It takes arguments with union a, b, c. So we want to turn this collection into a list of arguments to union. We do that using this apply notation of saying let's just put a star in there. Now, we've got out compiler for sequence. It's the function from text to the set that results from finding all the remainders for x and then finding all the remainders from each of those after we apply y. Unioning all those together in union will eliminate duplicates.
Now it's your turn to do one. This was the definition of alt in the interpreter matchset. Now I want you to write the definition of the compiler for alt, take a pattern for (x, y), and return the function that implements that.
15. 11 Alt (Answer)
Here is the answer.
11 def lit(s): return lambda t: set([t[len(s):]]) if t.startswith(s) else null
12 def seq(x, y): return lambda t: set().union(*map(y, x(t)))
13 def alt(x, y): return lambda t: x(t) | y(t)
The structure is exactly the same. It's the union of these two sets. The difference is that with a compiler the calling convention is pattern gets called with the text as argument. In the interpreter the calling convention is matchset calls with the pattern and the text.
16. 12 Simple Compilers
Here's the whole program.
1 def match(pattern, text):
2 "Match pattern against start of text; return longest match found or None."
3 remainders = pattern(text)
4 if remainders:
5 shortest = min(remainders, key = len)
6 return text[:len(text)-len(shortest)]
8 def lit(s): return lambda t: set([t[len(s):]]) if t.startswith(s) else null
9 def seq(x, y): return lambda t: set().union(*map(y, x(t)))
11 def oneof(chars): return lambda t: set([t[1:]]) if (t and t[0] in chars) else null
12 dot = lambda t: set([t[1:]]) if t else null
13 eol = lambda t: set(["]) if t == " else null
14 def star(x): return lambda t: (set([t]) |
15 set(t2 for t1 in x(t) if t1 != t
16 for t2 in star(x)(t1)))
Now, compilers have a reputation as being difficult and more complicated than interpreters, but notice here that the compilers is actually in many ways simpler than the interpreter.
It's fewer lines of code over all. One reason is because we didn't have to duplicate effort here of saying first we need constructors to build up a literal and then within matchset have an interpreter for that literal. Rather we did it just once. Just once! We said the constructor for literal returns a function which is going to be the implementation of the compiler for that type of pattern. It's very concise. Most of these are one-liners. Maybe I cheated a little bit and I replaced the word "text" with the word "t" to make it a little bit shorter and fit on one line.
There's only one that's complicated. That's the star of x, because it's recursive. The ones I haven't listed here is because they're all the same as before. Before we get into star(x) let me note that.
I didn't have to put down search here, because search is exactly the same as before.
I didn't have to put down plus, because plus is exactly the same as before. It's defined in terms of star.
What is the definition of star? One thing we could return is the remainder could be the text itself. Star of something — you could choose not to take any of it and return the entire text as the remainder. That's one possibility. The other possibility is we could apply the pattern x. From star(x) apply the pattern x to the text and take those sets as remainders. For every remainder that's not the text itself — because we already took care of that. We don't need to take care of it again. For all the remainders that are different from the whole text then we go through and we apply star(x) to that remainder. We get a new remainder and that's the result. That's all we need for the compiler result.
Oh, one piece that was missing is how do interface the match function, which takes a pattern and a text, with this compiler where a pattern is applied to the text. That's one line, which is slightly different. Here before we called matchset. In the previous implementation we had
2 remainders = matchset(pattern, text)
Your job then is to replace that with the proper code for the implementation that calls the compiler.
17. 12 Simple Compilers (Answer)
The answer is that the interface with the compiler is we just call a pattern with text as the input. That's all we need to do.
18. 13 Recognizers and Generators
So far what we've done is call the recognizer task. We have a function match which takes a pattern and a text, and that returns back a substring of text if it matches or None.
It's called a recognizer, because we're recognizing whether the prefix of text is in the language defined by the pattern.
There's a whole other task called the generator in which we generate from a pattern a complete language defined by that pattern.
For example, the pattern a or b sequenced with a or b — (a|b)(a|b). That defines a language of four different strings — {aa, ab, ba, bb}, and we could define a function that takes a pattern and generates out that language. That all seems fine.
One problem, though. If we have a language like a* then the answer of that should be the empty string or a or aa or aaa and so on — {", a, aa, aaa, …}. It's an infinite set. That's a problem. How are we going to represent this infinite set?
Now, it's possible, we could have a generator function that generates the items one at a time. That's a pretty good interface, but instead I'm going to have one where we limit the sizes of the strings we want. If we say we want all strings up to n characters in length, then that's always going to be a finite set.
I'm going to take the compiler approach. Rather than write a function "generate," I'm going to have the generator be compiled into the patterns. What we're going to write is a pattern, which is a compiled function, and we're going to apply that to a set of integers representing the possible range of lengths that we want to retrieve. That's going to return a set of strings.
pat({int}) –> {str}
So for example, if we define pattern to be a* — we did that appropriately — and then we asked for pattern, and we gave it the set {1, 2, 3},
pat = a*
pat({1,2,3}) –> {a, aa, aaa}
then that should return all strings which are derived from the pattern that have a length 1, 2, or 3. So that should be the set {a, aa, aaa}. Now let's go ahead and implement this.
19. 14 Oneof and Alt
Okay. Here's the code for the compiler.
1 def lit(s): return lambda Ns: set([s]) if len(s) in Ns else null
2 def alt(x, y): return lambda Ns: # your code here
3 def star(x): return lambda Ns: opt(plus(x))(Ns)
4 def plus(x): return lambda Ns: genseq(x, star(x), Ns, startx=1) #Tricky
5 def oneof(chars): return lambda Ns: # your code here
6 def seq(x, y): return lambda Ns: genseq(x, y, Ns)
7 def opt(x): return alt(epsilon, x)
8 dot = oneof('?') # You could expand the alphabet to more chars.
9 epsilon = lit(") # The pattern that matches the empty string.
11 null = frozenset([])
15 f = lit('hello')
16 assert f(set([1, 2, 3, 4, 5])) == set(['hello'])
17 assert f(set([1, 2, 3, 4])) == null
19 g = alt(lit('hi'), lit('bye'))
20 assert g(set([1, 2, 3, 4, 5, 6])) == set(['bye', 'hi'])
21 assert g(set([1, 3, 5])) == set(['bye'])
23 h = oneof('theseletters')
24 assert h(set([1, 2, 3])) == set(['t', 'h', 'e', 's', 'l', 'r'])
25 assert h(set([2, 3, 4])) == null
27 return 'tests pass'
Now, remember the way the compiler works is the constructor for each of the patterns takes some arguments — a string, and x and y pattern, or whatever — and it's going to return a function that matches the protocol that we've defined for the compiler.
The protocol is that each pattern function will take a set of numbers where the set of numbers is a list of possible lengths that we're looking for. Then it will return a set of strings.
What have I done for lit(s)? I've said we return the function which takes a set of numbers as input, and if the length of the string is in that set of number — if the literal string was "hello" and if hello has five letters and if 5 is one of the numbers we're trying to look for — then return the set consisting of a single element — the string itself. Otherwise, return the null set.
star I can define in terms of other things.
plus I've defined in terms of a function sequence that we'll get to in a minute. It's a little bit complicated. It's really the only complicated one here. We can reduce all the other complications down to calling plus, which calls genseq(). seq does that too.
I've introduced epsilon, which is the standard name in language theory for the empty string. So it's the empty string. It's the same as just the literal of the empty string, which matches just itself if we're looking for strings of length 0.
For dot — dot matches any character. I've decided to just return a question mark to indicate that. You could return all 256 characters or whatever you want. Your results would start to get bigger and bigger. You can change that if you want to.
I left space for you to do some work.
Give me the definitions for oneof(chars).
If we ask for oneof('abc') , what should that match?
What it should match is if 1 is an element of Ns then it should be abc. Otherwise, it shouldn't be anything.
Similarly for alt. Give me the code for that.
20. 14 Oneof and Alt (Answer)
2 def alt(x, y): return lambda Ns: x(Ns) | y(Ns)
4 def plus(x): return lambda Ns: genseq(x, star(x), Ns, startx = 1) #Tricky
5 def oneof(chars): return lambda Ns: set(chars) if 1 in Ns else null
8 dot = oneof('?^') # You could expand the alphabet to more chars.
The answer is if we want an alternative of the patterns x and y, then we use a protocol. We say let's apply x to the set of numbers. That gives us a set of strings that matches. We'll just union that with the set that comes back from applying y to the set of numbers. Now, for one of char — this will be a list of possible characters that we're trying to match one of. If 1 is in our list of numbers, then we should return all of those, and we have to return them as a set. We'll say the set of character if 1 is in there. Otherwise, there are no matches at all, so return null.
21. 15 Avoiding Repetition
That's the whole compiler. I want to show you just a little bit of the possibility of doing some compiler optimizations. Notice this sort of barrier here where we introduce lambda, where we introduce a function. Remember I said that there's two parts to a compiler. There's the part where we're first defining a language. When we call lit and give it a string, then we're doing some work to build up this function that's going to do the work every time we call it again. Anything that's on the right of the lambda is stuff that gets done every time. Anything that's to the left is stuff that gets done only once.
Notice that there is a part here building up this set of s that I'm doing every time, but that's wasteful because s doesn't depend on the input. s is always going to be the same.
I can pull this out and do it at compile time rather than do it every time we call the resulting function.
I'll make this set of s and I'll give that a name — set_s. Over here I'll do set_s equals that value. It looks like I'd better break this up into multiple lines.
Now I pulled out that precomputation so it only gets done once rather than gets done every time. You could look around for other places to do that.
I could pull out the computation of this set of characters and do that only once as well.
That's a lifting operation that stops us from repeating over and over again what we only need to do once. That's one of the advantages of having a compiler in the loop. There is a place to do something once rather than to have to repeat it every time.
22. 16 Genseq
Now there's only one bit left — this generate sequence. Let's talk about that. Now sequence in this formulation is a function that takes x and y, two patterns, and what it returns is a function, and that function takes a list of numbers and returns a set of text that match. So sequence is delaying the calculation. It's computing a function which can do the calculation later on. Genseq does the calculation immediately. It takes x and y and a set of numbers, and it immediately calculates the set of possible text. Now the question is what do we know about genseq in terms of the patterns x and y and the set of possible numbers. We know at some point we're going to have to call the pattern x with some set of numbers. We're not yet quite sure what. That's going to return a list of possible text. Then we're going to have to call y with some other set of numbers. Then we're going to have to concatenate them together and see if they make sense, if the concatenation of some x and some y, if that length is within this allowable set. Now, what do we know about what these Ns should be in terms of this set of possible numbers here regardless of what this set is. This could be a dense set, so we could have Ns equals 0, 1, 2, all the way up to 10 or something. Or it could be a sparse set. It could be, say, only the number 10. But either way, the restriction on x and y is such that they have to add up to no more than 10. But x could be anything. If the list of possible numbers that we want to add up to is only 10, that doesn't constrain x at all other than to be less than 10. This N should be everything up to the maximum of N sub s. Then what should y be? Well, we have two choices. One, we could for each x that comes back we could generate the y's. Or we could generate the y's all at once and then try to combine them together and see if they match up. I think that's actually easier. So for the y's also, they can be any size up to the maximum. Then we take the two together, add up the x match and the y match and see if that length is within N. In this example, if Ns is equal to 10, here we want to have the Ns be everything from 0 up to 10 inclusive in both cases, and we get back some results like, say, a, abb, acde, and so on, and some other result over here — ab, bcd. Then for each of them we add them up, and if we say abb plus ab and check to see if that's in Ns. If it is, we keep it. If it's not, we don't keep it. Here is candidate solution for genseq. We take x, y, and a set of numbers, and then we define Ns as being everything up to the largest number, including the largest number. We have to add 1 to the maximum number in order to get a range going from 0 up to and including the largest number. Now that we know the possible values of the numbers that we're looking for for the sizes of the two components-the x and the y components — then we can say m1 is all the possible matches for x, m2 is all the possible matches for y. If the length of m1 plus m2 is in the original set of numbers that we're looking for, then return m1 plus m2. This seems reasonable. It looks like it's doing about what we're looking for to generate all sequences of x and y concatenated together. But I want you to think about it and say, have we really gotten this right? The choices are is this function correct for all inputs? Or is in incorrect for some? Does it return incorrect results? Or is it correct when it returns, but doesn't doesn't always return? Think about that. Think about is there any result that looks like it's incorrect that's being formed. Think about does it infinite loop or not. Think about base cases on recursion and saying is there any case where it looks like it might not return. This is a tricky question, so I want you to try it, but it may be difficult to get this one right.
23. 16 Genseq (Answer)
The answer is that it is correct when it returns.
All the values it builds up are correct, but unfortunately it doesn't always return. Let's try to figure out why.
In thinking about this, we want to think about recursive patterns.
Let's look at the pattern x+. We've defined x+ as being the sequence of x followed by x*. And now for most instances of x that's not a problem.
If we had plus(lit('a')), it not going to be a problem. That's going to generate a, aa, aaa, and so on.
But consider this — let's define a equals lit('a'), pat equals plus(opt('a')).
Now, this should be the same. This should also generate a, aa, aaa.
The way we can see that is we have a plus so that generates any number of these. If we pick a once, we get this. It we pick a twice we get this. If we pick a three times we get this. But the problem is there's all these other choices in between.
opt(a) means we we can either be picking a or the empty string. As we go through the loop for plus, we could pick empty string, empty string, empty string. We could pick empty string an infinite number of times. Even though our N is finite — at some point we're going to ask for pattern of some N — let's say the set {1, 2, 3} — we won't have a problem with having an infinite number of a's, but we will have a problem of choosing from the opt(a) the empty part. If an infinite number of times we choose the empty string rather than choosing a, then we're never going to get past three as the highest value. We're going to keep going forever. That's the problem. We've got to somehow say I don't want to keep choosing the empty string. I want to make progress and choose something each time through. So how can we make sure that happens?
24. 17 Induction
What I have to do is I have to check all my cases where I have recursion and eliminate any possibility for infinite recursion. Now, there are two possibilities in the star function and the plus function. Those are the two cases where regular expressions have recursion. But now star I defined in terms of plus, so all that's left is to fix plus to not have an infinite recursion. Here's how I define plus. Basically, I said that x+ is defined as x followed by x*, and the x* is in turn defined in terms of x+. The problem was that I was going through and saying, okay, for x when I'm doing plus(opt(a)), for my x I want to choose opt(a). Okay, I think I'll choose the empty string. So I chose that, and now I'm left in a recursion where I have an x*, which is defined in terms of x+, and I haven't made any progress. I have a recursive call that's defined in terms of itself. We know in order to make sure that a recursion terminates, we have to have some induction where we're reducing something. It makes sense here that what we're going to reduce is our set Ns. One way to guarantee to do that is to say when I'm generating the x followed by the x* let's make sure that the x generates at least 1 character. If we can guarantee that x generates a character, then when we go to do the x* we've reduced our input. So we have this inductive property saying that now our set of Ns will be smaller. It's smaller by 1 each time, and if it started out finite and we reduce by 1 every time, then eventually we're going to terminate. Let's see how we can implement that idea of saying every time we have an x+ we have a x and an x*, we have to choose at least 1 character for x. Note that that's not limiting us in any way. That hasn't stopped us from generating all possible expressions, because if we were going to generate something, it would have to come from somewhere — either from this x or this x — and we might as well make sure it comes from here rather than adding an infinite number of nothings before we generate something.
25. 18 Testing genseq
Here's what gensequence looks like. We have a recursive base case that says, if there are no numbers that we're looking for, we can't generate anything of those lengths, and so return the empty set. Then we say the xmatches we get by applying x to any number up to the maximum of Ns, including the maximum of Ns, but then we got to do some computation to figure out what can be the allowable sizes for y, and we do that by saying, let's take all the possible values that came back from the xmatches and then for each of those values and for each of the original values for the lengths that we're looking for, subtract those off and say, total is going to be one of the things we got from x and one of the things we got from y, that better add up to one of the things in Ns. Then we call y with that set of possible ends for y and then we do the same thing that we were going to do before. We go through those matches, but this is going to be with a reduced set of possibilities and count those up, and now, the thing that makes it all work is this optional argument here, saying the number that we're going to start at for the possible sizes, for x in the default case, that's 0, and so we start the range at 0. But in the case where we're calling from +, we're going to set that to 1. Let's see what that looks like. Here's the constructors, the compilers for sequence and plus. For a regular sequence, there is no constraint on this start for x. X can be any size up to the maximum of the N's. But for plus, we're going to always ask that the x part have a length of at least 1, and then the y part will be whatever is left over. That's how we break the recursion, and we make sure that genseq will always terminate. Now this language generation program is a little bit complex. So I wanted to make sure that I wrote a test suite for it to test the generation. So here I've just defined some helper functions and then wrote a whole bunch of statements here. If we check one of 'ab' and limit that to size 2, that should be equal to this set. It's gone off the page. Let's put it back where it belongs. One element of size 0, 2 elements of size 1, and 4 elements of size 2, just what you would expect. Here are sequences of a star, b star, c star of size exactly 4. Here they are and so on and so on. We've made all these tests. I should probably make more than these, but this will give you some confidence that the program is doing the right thing if it passes at least this minimal test suite.
26. 19 Theory and Practice
This is a good time to pause and summarize what we've learned so far. We've learned some theory and some practice. In theory, we've learned about patterns, which are grammars which describe languages, where a language is a set of strings. We've learned about interpreters over those languages, and about compilers, which can do the same thing only faster. In terms of practice, we've learned that regular expressions are useful for all sorts of things, and they're a concise language for getting work done. We've learned that interpreters, including compilers, can be valuable tools, and that they can be more expressive and more natural to describe a problem in terms of a native language that makes sense for the problem rather than in terms of Python code that doesn't necessarily make sense. We learned functions are more composable than other things in Python. For example, in Python we have expressions, and we have statements. They can only be composed by the Python programmer whereas functions can be composed dynamically. We can take 2 functions and put them together. We can take f and call g with that and then apply that to some x. We can do that for any value of f and g. We can pass those into a function and manipulate them and have different ones applying to x. We can't do that with expressions and statements. We can do it with the values of expressions, but we can't do it with expressions themselves. Functions provide a composability that we don't get elsewhere. Functions also provide control over time, so we can divide up the work that we want to do into do some now and do some later. A function allows us to do that. Expressions and statements don't do that because they just get done at 1 time when they're executed. Functions allow us to package up computation that we want to do later.
27. 20 Changing seq
Now one thing I noticed as I was writing all those test patterns is that functions like seq and alt are binary, which means if I want a sequence of 4 patterns, I have to have a sequence of (a, followed by the sequence of (b, followed by sequence of (c,d), and then I have to count the number of parens and get them right. It seems like it'd be much easier if I could just write sequence of (a, b, c, d). And we talked before about this idea of refactoring, that is changing your code to come up with a better interface that makes the program easier to use, and this looks like a good example. This would be a really convenient thing to do. Why did I write seq this way? Well, it was really convenient to be able to define sequence of (x,y) and only have to worry about exactly 2 cases. If I had done it like this, and I had to define sequence of an arbitrary number of arguments, then the definition of sequence would have been more complex. So it's understandable that I did this. I want to make a change, so let's draw a picture. Imagine this is my whole program and then somewhere here is the sequence part of my program. Now, of course, this has connections to other parts of the program. Sequence is called by and calls other components, and if we make a change to sequence, then we have to consider the effects of those changes everywhere else in which it's used. When we consider these changes, there are 2 factors we would like to break out. One is, is the change backward compatible? That is, if I make some change to sequence, am I guaranteed that however it was used before, that those uses are still good, and they don't have to be changed? If so, then my change will be local to sequence, and I won't have to be able to go all over the program changing it everywhere else. So that's a good property to have. So for example, in this case, if I change sequence so that it still accepted than that would be a backwards compatible change as long as I didn't break anything else. And then the second factor is whether the change is internal or external. So am I changing something on the inside of sequence that doesn't effect all the callers, than that's okay. In general, that's going to be backwards compatible. Or am I changing something on the outside — changing the interface to the rest of the world? In this case, going from the binary version to this n_ary version, I can make it backwards compatible if I'm careful. It's definitely going to be both an internal and external change. So I'm going to have to do something to the internal part of sequence. And then I'm also changing the signature of the function, so I'm effecting the outside as well. I can make that effect in a backwards compatible way. Thinking about those 2 factors, what would be the better way to implement this call? Let's say we're dealing with the match-set version where we're returning a tuple, would it be better to return the tuple sequence (a, b, c, d) or the tuple sequence of (a, sequence of (b, sequence of (c, d)? Tell me which of these do you prefer from these criteria.
28. 20 Changing seq (Answer)
The answer is this approach is much better because now from the external part everybody else sees exactly the same thing. But internally, I can write the calls to the function in a convenient form and they still get returned in a way that the rest of the program can deal with, and I don't have to change the rest of the program.
29. 21 Changing Functions
Let's go about implementing that. Let's say I had my old definition of sequence of (x,y), and we say return the tuple consisting of a sequence x and y. Now I want to change that. How would I change that? Well, instead of x and y, I think I'm going to insist that we have at least 1 argument, so I can say x and then the rest of the args, and I can say, if the length of the rest of the args equals 1, then I've got this binary case and then I can take sequence of x. The second arg is now no longer called y. It's called args at 0. Else: now I've got a recursive case with more than 2 args, and I can do something there. So it's not that hard, but I had to do a lot of violence to this definition of sequence, and come to think of it, I may be repeating myself because I had to do this for sequence, and I'm also going to have to do it for alt, and if I expand my program, let's say I start wanting to take on arithmetic as well as regular expressions, then I may have functions for addition and multiplication and others, and I'm going to have to make exactly the same changes to all these binary functions. So that seems to violate the "Don't Repeat Yourself" principle. I'm making the same changes over and over again. It's more work for me. There's a possibility of introducing bugs. Is there a better way? So let's back up and say, what are we doing in general? Well, we're taking a binary function f of (x,y), and we want to transform that somehow into an n_ary function — f prime, which takes x and any number of arguments. The question is, can we come up with a way to do that to automatically — change one function or modify or generate a new function from the definition of that original function.
30. 22 Function Mapping
What's the best way to do that? How can I map from function f to a function f prime? One possibility would be to edit the bytecode of f. Another possibility would be to edit the source string of f and concatenate some strings together. Another possibility would be to use an assigment statement to say f = some function g of f to give us a new version of f. Which of these would be the best solution?
I think it's pretty clear that this one is the best because we know how to do this quite easily. We know how to compose functions together, and that's simple, but editing the bytecode or the source code, that's going to be much trickier and not quite as general, so let's go for the solution.
32. 23 n_ary Function
What I want to do is define a function, and let's call it n_ary, and it takes (f), which should be a binary function, that is a function that takes exactly 2 arguments, and n_ary should return a new function that can take any number of arguments. We'll call this one f2, so that f2 of (a, b, c) is = f(a, f(b, c)), and that will be true for any number of arguments — 2 or more. It doesn't have to just be a, b, c. So let's see if you can write this function n_ary. Here's a description of what it should do. It takes a binary function (f) as input, and it should return this n_ary function, that when given more than 2 arguments returns this composition of arguments. When given 2 arguments, it should return exactly what (f) returns. We should also allow it to take a single argument and return just that argument. That makes sense for a lot of functions (f), say for sequence. The sequence of 1 item is the item. For alt, the alternative of 1 item is the item. I mentioned addition and multiplication makes sense to say the addition of a number by itself is that number or same with multiplication. So that's a nice extension for n_ary. See if you can put your code here. So what we're doing is, we're passed in a function. We're defining this new n_ary function, putting the code in there, and then we're returning that n_ary function as the value of that call.
33. 23 n_ary Function (Answer)
Here's the answer. It's pretty straight forward. If there's only 1 argument, you return it. Otherwise, you call the original f that was passed in with the first argument as the first argument, and the result of the n-ary composition as the other argument.
34. 24 Update Wrapper
Now how do we use this? Well, we take a function we define, say seq of x, y, and then we can say sequence is redefined as being an n_ary function of sequence. Oops — I guess I got to fix this typo here. From now on, I can call sequence and pass in any number of numbers, and it will return the result that looks like that. So that looks good. In fact, this pattern is so common in Python that there's a special notation for it. The notation is called the decorator notation. It looks like this. All we have to do is say, @ sign, then the name of a function, and then the definition. This is the same as saying sequence = n_ary of sequence. It's just an easier way to write it. But there is one problem with the way we specified this particular decorator, which is if I'm in an interactive session, and I ask for help on sequence, I would like to see the argument list and if there is a doc string, I want to see the documentation here. I didn't happen to put in any documentation for sequence. But when I ask for help, what I get is this. I'm told that sequence is called n_ary function. Well, why is that? Because this is what we returned when we define sequence = n_ary of sequence. We return this thing that has the name n_ary function. So we would like to fix n_ary up so that when the object that it returns has the same function name and the same function documentation — if there is any documentation — and have that copied over into the n_ary f function. Now it turns out that there is a function to do exacty that, and so I'm going to go get it. I'm going to say from the functools — the functional tools package. I want to import the function called update_wrapper. Update_wrapper takes 2 functions, and it copies over the function name and the documentation and several other stuff from the old function to the new function, and I can change n_ary to do that, so once I've defined the n_ary function, then I can go ahead and update the wrapper of the n_ary function — the thing I'm going to be returning from the old function. So this will be the old sequence, which has a sequence name, a list of arguments, maybe some documentation string, and this will be the function that we were returning, and we're copying over everything from f into n_ary f. Now when I ask for help — when I define n_ary sequence, and I ask for help on sequence, what I'll see is the correct name for sequence, and if there was any documentation string for sequence, that would appear here as well. So update_wrappers is a helpful tool. It helps us when we're debugging. It doesn't really help us in the execution of the program, but in doing debugging, it's really helpful to know what the correct names of your functions are. Notice that we may be violating the Don't Repeat Yourself principle here. So this n_ary function is a decorator that I'm using in this form to update the definition of sequence. I had to — within my definition of n_ary — I had to write down that I want to update the wrapper. But it seems like I'm going to want to update the wrapper for every decorator, not just for n_ary, and I don't want to repeat myself on every decorator that I'm going to define.
35. 25 Decorated Wrappers
So here's an idea. Let's get rid of this line, and instead, let's declare that n_ary is a decorator. We'll write a definition of what it means to be a decorator in terms of updating wrappers. Then we'll be done, and we've done it once and for all. We can apply it to n_ary, and we can apply it to any other decorator that we define. This is starting to get a little bit confusing because here we're trying to define decorator, and decorator is a decorator. Have we gone too far into recursion? Is that going to bottom out? Let's draw some pictures and try to make sense of it. So we've defined n_ary, and we've declared that as being a decorator, and that's the same as saying n_ary = decorator of n_ary. Then we've used n_ary as a decorator. We've defined sequence to be an n_ary function. That's the same as saying sequence = n_ary of sequence. Now we wanted to make sure that there's an update so that the documentation and the name of sequence gets copied over. We want to take it from this function, pass it over to this function because that's the one we're going to keep. While we're at it, we might as well do it for n_ary as well. We want to have the name of n_ary be n_ary and not something arbitrary that came out of decorator. So we've got 2 updates that we want to do for the function that we decorated and for the decorator itself. Now let's see if we can write decorator so that it does those 2 updates. So let's define decorator. It takes an argument (d), which is a function. Then we'll call the function we're going to return _d, and that takes a function as input. So it returns the update wrapper from applying the decorator to the function and copying over onto that decorated function, the contents of the original function's documentation and name, and then we also want to update the wrapper for the decorator itself. So from (d) the decorated function, we want to copy that over into _d and then return _d. Now which update is which? Well, this one here is the update of _d with d, and this one is the update of the decorated function from the function. So here we're saying the new n_ary that we're defining gets the name from the old n_ary, the name in the documentation string, and here we're saying the new sequence, the new n_ary sequence, gets its name from the old sequence. Here's what it all looks like. If you didn't quite follow that the first time, don't worry about it. This is probably the most confusing thing in the entire class because we've got functions pointing to other functions, pointing to other functions. Try to follow the pictures. If you can't follow the pictures, that's okay. Just type it into the interpreter. Put these definitions in. Decorate some functions. Decorate some n_ary functions. Take a look at them and see how it works.
36. 26 Decorated Decorators
Okay, now for a quick quiz. We have this definition of decorator, and we've seen how that works. Here's an alternative that was proposed by Darius Bacon, which is this1 line — return the function that updates wrapper for the decorator applied to the function from the original function, and then 1 more line, which says, decorator = decorator of decorator. Can you get any more recursive than that? The question is, does this work? I want you to give me the best answer. The possible answers are yes, it does; no, it's an error; no, it correctly updates decorator such as n_ary, but not the decorated function such as (seq); or no, my brain hurts.
37. 26 Decorated Decorators (Answer)
Now if you answered, no, my brain hurts — well, who am I to argue with that? But I think the best answer is yes, it does in fact work. Even though there's only 1 update wrapper call, both of them happen, and the reason is because decorator becomes a decorator. So this version of decorator updates (seq), and then this version that gets created when we make it a decorator updates the decorator.
38. 27 Cache Management
If you took CS 101, you saw the idea of memoization. If you haven't seen it, the idea is that sometimes particularly with the recursive function, you will be making the same function calls over and over again. If the result of a function call is always the same and the computation took a long time, it's better to store the results of each value of N with its result in a cache, a table data structure, and then look it up each time rather than try to recompute it.
We can make this function be cached very simply with a couple extra lines of code. We ask if the argument is already in the cache, then we just go ahead and return it. Otherwise, we compute it, store it, and then return it. So this part with the dot, dot, dot, is the body of the function. All the rest is just the boiler plate that you have to do to implement this idea of a cache. We've done this once, and that's fine, but I'm worrying about the principle of Don't Repeat Yourself. There's probably going to be lots of functions in which I want to store intermediate results in a cache, and I don't want to have to repeat this code all of the time. So this is a great idea for a decorator. We can define a decorator called memo, which will go ahead and do this cache management, and we can apply it to any function. The great thing about this pattern of using memoization is that it will speed up any function f that you pass to it because doing a table look-up is going to be faster than a computation as long as the computation is nontrivial, is more than just a look-up.
Now the hockey player, Wayne Gretzsky, once said that you miss 100% of the shots you don't take. This is kind of the converse. This is saying you speed up 100% of the computations that you don't make. So here's the memo decorator.
1 @decorator
2 def memo(f):
3 """Decorator that caches the return value for each call to f(args).
4 Then when called again with same args, we can just look it up."""
5 cache = {}
6 def _f(*args):
8 return cache[args]
9 except KeyError:
10 cache[args] = result = f(*args)
12 except TypeError:
13 # some element of args can't be a dict key
14 return f(args)
15 return _f
The guts of it is the same as what I sketched out previously.
If we haven't computed the result already, we compute the result by applying the function f to the arguments. It gives us the result. We cache that result away, then we return it for this time. It's ready for next time. Next time we come through, we try to look up the arguments in the cache to see if they're there. If they are, we return the result.
And now I've decided to structure this one as a try-except statement rather than an if-then statement. In Python, you always have 2 choices. You can first ask for permission to say are the args in the cache, and if so, return cache or args, or you can use the try-except pattern to ask for forgiveness afterwards to say, I'm first going to try to say, if the args are in the cache, go ahead and return it. If I get a keyError, then I have to fill in the cache by doing the computation and then returning the result.
The reason I use the try structure here rather than the if structure is because I knew I was going to need it anyways for this third case. Either the args are in the cache, or they aren't, but then there's this third case which says that the args are not even hashable.
Start out with a dictionary d being empty, and then I'm going to have a variable x, and let's say x is a number. If I now ask, is x in d? That's going to tell me false. It's not in the dictionary yet. But now, let's say I have another variable, which is y, which is the list [1, 2, 3] and now if I ask is y in d? You'd think that would tell me false, but in fact, it doesn't. Instead, it gives me an error, and what it's going to tell me is type error: unhashable type: list.
That means we were trying to look up y in the dictionary, and a dictionary is a hash table — implemented as a hash table. In order to do that, we have to compute the hash code for y and then look in that slot in the dictionary. But this error is telling us that there is no hash code for a list.
Why do you think that is?
Are lists unhashable because:
lists can be arbitrarily long?
lists can hold any type of data as the elements, not just integers?
lists are mutable?
Now I recognize this might be a hard problem if you're not up on hash tables. This might not be a question you can answer. But give it a shot and give me your one best response.
39. 27 Cache Management (Answer)
The answer is because lists are mutable. That makes them unlikely candidates to put into hash tables. Here's why. Let's suppose we did allow lists to be hashable. Now we're trying to compute the hash function for y, and let's say we have a very simple hash function — not a very good one — that just says add up the values of the elements. Let's also say that the hash of an integer is itself, so the hash code for this list would be equal to 6, the sum of the elements. But now the problem is, because these lists are mutable, I could go in, and I could say, y[0] = 10. Y would be the list [10, 2, 3]. Now when we check and say, is y in d? We're going to compute the hash value 10 + 2 + 3 = 15. That's a different hash value than we had before. So if we stored y into the dictionary when it had the value 6, and now we're trying to fetch it when it has the value 15, you can see how Python is going to be confused. Now, there's 2 ways you could handle that. One — the way that Python does take is to disallow putting the list into the hash table in the first place because it potentially could lead to errors if it was modified. The other way is Python could allow you to put it in, but then recognize that it's the programmers fault, and if you go ahead and modify it, then things are not going to work anymore, and Python does not take that approach, although some other languages do.
40. 28 Save Time Now
Now to show off how insanely great memo is, we'll want to have before and after pictures, showing the amazing weight loss of a model that was fat before and was thin after applying memo. Oh! Wait a minute. It's not weight loss. It's time loss that we're going to try to measure. We want to show that before, we have a function f, and that's going to run very slowly, making us sad. And after, we have a function memo of f, and that's going to run very quickly, making us happy. Now we could do that with a timer and say it took 20 seconds to do this and .001 seconds after, but instead of doing it with timing, I think it's a little bit more dramatic just to show the number of function calls, and I could go into my function and modify it to count the number of function calls, but you could probably guess a better way to do that. I'm going to define a decorator to count the calls to a function because I'm probably going to want to count the calls to more than 1 function as I'm trying to improve the speed of my programs. So it's nice to have that decorator around. So here's the decorator countcalls, you pass it a function, and this is the function that it returns. It's going to be a function that just increments entry for this function in a dictionary callcounts. Increment that by 1 and then go ahead and apply the function to the arguments and return that result. We have to initialize the number of calls to each funciton to be 0, and that's all there is to it. So here I've defined the Fibonacci function. Through cursive function it calls itself twice for each call, except for the base case. I can count the calls both with and without the memoized version. So I'm going to do before and after pictures — before and after memoizing. So here's before. I have the values of n and a value computed for Fibonacci number of n, the number of calls created by countcalls, and then I have the ratio of the number of calls to the previous number. We see the number of calls goes up by the time we get up to n = 20. We got 10,000 calls. We can scroll down, and by the time we're up to n = 30, we have 2.6 million calls. And here's the after. Now we've applied the memo decorator. Now the number of calls is much more modest. Now at 20, we're only at 39 calls, and at 30, we're at 59 calls rather than 2.6 million. So that's pretty amazing weight loss to go from 2.6 million down to 59, just by writing 1 little decorator and applying it to the function. Now just as an aside here, and for you math fans in the audience, I'm back to the before part. This is without the memoization. This number here in this column is the ratio of the number of calls for n = 30 to the number of calls for n = 29. You can see that it's converging to this number 1.6180. Math fans out there, I want you to tell me if you recognize that number. Do you think it's converging to 1 + square root of 5 / 2, or the square root of e?
41. 28 Save Time Now (Answer)
The answer is 1 + square root of 5 over 2, otherwise known as the Golden Ratio. The Golden Ratio I knew was actually the ratio of success of elements of the Fibonacci sequence — the sequence 1,1, 2, 3, 5, 8, and so on — converges to that ratio. But I didn't know that the number of calls in the implementation also converges to that ratio. So that's something new.
42. 29 Trace Tool
I want to make the point there are different types of tools that you can use in your tool box. We just saw the count calls. I think I would classify that as a debugging tool, and we saw a memo, and I'll classify that as a performance tool. Earlier, we saw n_ary, another decorator, which I can classify as an expressivenes tool. It gives you more power to say more about your language. This gives you no more power but makes it faster. This isn't going to end up in your final program, but helps you develop the program faster. I want to add another tool here in debugging called trace, which can help you see the execution of your program. So I'm going to define a decorator trace, which when we apply to fib, gives us this kind of output. When I ask here for what's the 6th Fibonacci number? It says for each recursive call, we have an indented call with an arrow going to the right saying we're making a call, and for each return, we have an arrow going to the left. When you ask for fib of 6, you keep on going down the list until we get near the end. When we ask for fib of 2, then that's defined in terms of 1 and 0, and they both return 1, so that means fib of 2 returns 2 and so on. We can see the shape of the trace here as we go. It's coming to the right and then returning back and coming to the right some more and returning back. The pattern takes a long time to reveal itself and would take even longer for larger arguments other than 6. But it gives you some idea for the flow of control of the program. So that's a useful tool have, and here's an implementation. It follows the same pattern as usual. Decorator takes a function as input. We're going to create another function, and this is what it's going to look like. We're going to figure out what it is that we're going to print. We're going to keep a variable, which we keep as an attribute of the trace function itself called the level. We'll increment that as we come in, print out some results here. We initialize the trace level to 0 — the indentation level — and then finally, we return the function that we just built up. I've left out some bits here, and I want you to fill them in to make this function work properly to show the trace that I just showed.
43. 29 Trace Tool (Answer)
So the code you had to write was pretty straight forward. Like most decorators, we compute the result here by applying the function to the args, then we return the result down here. Maybe a little bit tricky is what's in this finally clause, which is we're decreasing the trace level. So the indentation level goes up by 1 every time we enter a new function and down by 1 every time we go back. The issue here is, we want to make sure that we do decrement this. If we increment this once, and then calling the function results in an error, and we get thrown out of that error, we want to make sure we put it back where it belongs. We don't want to mess with something and then not restore it. So that's why we put this in a try finally.
44. 30 Disable Decorator
We're coming to the end of what I want to say about decorators. I wanted to add one more debug tool. That's one I'm going to call disabled. It's very simple. Disabled is another name for the identity function — that is the function that returns its argument without doing any computation on it whatsoever. Why do I want it and why do I call it "disabled?" Well, the idea is that if I'm using some of these debugging tools like trace or countcalls, I might have scattered throughout my program trace define f and some other traced functions. Then I might decide I think I'm okay now. I think I've got it debugged. I don't want to trace any more. Then what I can do is I can just say "trace = disabled" and reload my program, and now the decorator trace will be applied to the function, but what it will do is return the function itself. Notice we don't have to say that disabled is a decorator, even though we're using it as if it were one, because it doesn't create a new function. It just returns the original function. That way we won't have the trace output cluttering up our output, and the function will be efficient. There won't even be a test to see if we are tracing or not. It'll just use the exact function that passed in.
45. 31 Back to Languages
Now we're done with decorators, and I want to go back to languages. The first thing I want to say is that our culture is full of stories of wishful thinking. We have the story of Pinocchio, who wishes to be a real boy, and the of Dorothy, who wishes to return from Oz to her home in Kansas by clicking her shoes together. They find that they have the power within them to fulfill their wishes. Programming is often like that. I'm righting the body of the definition of a function, and I wish I had the function "fib" defined already. If I just assume that I did, then eventually my wish will come true. In this case, it was good while writing the right-hand side to just assume I wish I had the function I want and proceed as if you did, and sometimes it's a good idea to say I wish I had the language I want and proceed as if you did. Here's an example. Suppose you had to deal with algebraic expressions and not just compute them the way we can type this expression into Python and see its value if x is defined but manipulate them. Have the user type them in, modify them, and treat them as objects rather than as something to be evaluated. Now, my question for you is it is possible to write a regular expression which could recognize expressions like this that are valid. Is the answer, yes, it is possible to write that? No, it's not possible because our language we're trying to define has the plus and asterisk symbols and those are special within regular expressions? Or no, we can't parse this language because this language includes parentheses, and we can't do balanced parentheses for the regular expressions. If you're not familiar with language theory, this may be a hard question for you, but go ahead and give it a shot anyways.
46. 31 Back to Languages (Answer)
The answer is that the problem is the balanced parentheses. Regular expressions can handle a set number of nesting parentheses — one or two or three — but they can't handle an arbitrary number of nestings. It makes sure that all the left parentheses balance with the right parentheses. We're going to need something else. The thing we traditionally look at is called context-free languages, which are more general than the regular languages. We'll see how to handle that.
1. 01 l Water Pouring Problem
I'm going to begin this unit with an old problem known as the "water-pouring problem." Here's what we're given: two glasses of water and we have a faucet in a sink, which can be the source of as much water as we want. Now, these glass are of different sizes. I haven't drawn them that much different, but this one is 4 oz, and this one is 9 oz. For those of you in the rest of the world besides the U.S., an ounce is about 30 mL. Our goal is to measure out a specific amount of water. What we want to have is 6 oz of water measured out. Six ounces won't fit in this glass. The idea is at the end want to have this glass filled with exactly 6 oz of water. There's no gradated markers. It's not like a function.graduated cylinder or measuring cup where we have the measurements on the glass. It wouldn't be accurate enough to just eyeball it. What we've got to do is we've got to figure out how to do that by measuring out a precise amounts into the cups and pouring them off. For example, if the goal had been 5 oz, then that would have been easy. We'd just fill the 9 oz all the way up to the top, and then pour the 9 oz into the 4 oz until the 4 oz is all the way full, and then what would be remaining here because there's 9 altogether would be 5 in this glass. Five ounces is easy. Six ounces is not as obvious how to get there. The puzzle is to find a sequence of pouring actions, and the pouring can be from one glass to another. It can go in the other direction. It can go from the faucet into each of the glasses. And it can go from the glasses down the drain. Six different actions we can take, and we want to find a sequence of actions that arrives at this goal of 6 oz. Of course, we can generalize the problem and put any number rather than 9 and 4 and 6. As usual, let's make our inventory of concepts that we're going to be dealing with. We have the glass, and the glass has a capacity and a current level. This glass would have capacity 9, current level 5. We're also going to need collection of glasses probably–a pair of glasses. I guess we can say that the pair of glasses and they're current levels represents a complete state of the world. We'll think of that as a state of the world. Everything we need to know about where we are in the problem. Then we have a goal that we're trying to reach. We have the pouring actions–1, 2, 3, 4, 5, 6. That breaks down into emptying, filling, and transferring. The transferring, I think, is a little bit tricky, because there are two ways to do it. When we were transferring from the 9 oz into the 4 oz– so we transfer from x to y–we can do that until y is full. That's what happened here. The 4 oz was full. Or we could do it until x is empty. If we were starting to pour back 4 oz from here into an empty one, we could do it until it was empty. Anything else in the inventory? Oh, well, we certainly need a notion of a solution. A solution is going to be a sequence of steps– to pour from here to here, then from here to the drain, then fill up, then pour again, and so on. What this unit is really all about is techniques for finding these solutions, which are sequences of steps. Again, we're always talking about managing complexity in this class. The complexity we're trying to manage here is a complexity
2. 03 q Combinatorial Complexity
There's a complexity that comes from combinatorial problems. We've seen that before. In the cryptarithmetic problems ODD + ODD = EVEN. We had these up to 10! different permutations of digits to assign, and it was complex because we had to consider them all. In the zebra puzzle we had 5!^5 combinations to consider. It was complex because it took a long time to consider them all. We came up with an optimization to consider a few of them by going one at a time. For our pouring problem, we know there are 6 actions, 2 empties, 2 fills, and 2 pours. The glasses are of size 4 and 9. The goal is 6 oz. I guess my question for you is how many combinations do we need? For cryptarithmetic it was 10!. For zebra it was 5!^5. For pouring is it 64, 6(9-4), 66, or 69, or can't tell–none of the above?
3. 03 s Combinatorial Complexity
The answer is that you can't tell. This is a different type of combinatorial problem than the previous ones. In the previous ones we had a fixed number of variables, and we knew how many combinations we had for each variable. In the zebra problem, there were 25 variables, and that's all there was. We could enumerate all the combinations. For the pouring problem we're trying to not fill static variables but rather put together a sequence of actions to go from one state to the next. We don't know how long that sequence is, and of course, at each point we have 6 different options of different ways to go. From each of those 6 more. We know it's going to be roughly 6 to the something, because we branch 6 at each point, but we don't know what that x is, because we don't know how long the sequence is. So that makes the problem slightly different. If we want to be foraml, we call it a combinatorial optimization problem, but usually we just called it a "search" problem.
4. 04 q Exploring the Space
Now it's called search traditionally, but I think "exploration" is a better name for it. We start out at home, and in this case our home is where we have two glasses. Zero and zero are the values for how full the glasses are. Then we start to explore. One way we could explore is to fill one of the glasses Then we're at this state–say we're at 0 and 4– but we know that there are other actions in which we could explore in other directions. Now we could take one of the other states and explore from there in other directions. We have lots of choices going forward of this huge space that we're exploring. Now, somewhere out in this space–and we don't know which direction it is– is this goal state, which has 6 and then actually any amount in the other glass. We're trying to reach that, and we're distinguishing this part of the state space as a goal. So I drew this as one, but really it's a collection of states in that every state that has 6 on one side and anything on the other should be considered part of this collection of goals. We're trying to search forwards towards that. One reason I like to call it an exploration problem is because we can think of going forward, exploring a new land, and part of that exploration is that we've got a frontier. Here's all the states that are the farthest out that we've gone. If we want to make progress towards the goal, then we're probably going to have to step from one of the frontier nodes farther out. We've separated the set of all possible states into the goal state, the frontier states, and the previously explored states. Then you can see that the way to make progress is to say let's take one of the frontier states and expand that, and we have the advantage here of being a computer that an individual explorer doesn't have. An individual explorer has to take one path, and if they decide they've gone in the wrong direction, they have to go all the way back. A computer can store lots of states in memory. Computer exploration is more like a collection of explorers all collectively expanding the frontier. Our next move can be to say we'll take one of these explorers, say the one in this state here, and say now tell me what's next. You've got 6 actions from there. Where do they go to? Maybe some of them explore the world and generate new states that we haven't seen before. Maybe some of them go to a state that we already know is on the frontier. Maybe some of them regress backwards into previously explored territory. But we can keep on going, expanding out our frontier until eventually the frontier keeps on expanding. When it overlaps the goal, then we've got a solution. Now, in exploration problems like this, there are two problems that we have to worry about. One problem is that there is no solution at all, that the goals are not connected to the to start state. So there's no path from here to there. Then what we want to do is do the exploration we need and report back that it's impossible. We want to find out that it's impossible. Then the other problem is if there is some path that eventually makes it to the goal, We want to make sure that we find that in a reasonable amount of time. That means we want to be efficient about the way we explore the space. It also means that we don't want to get stuck in an infinite loop. Now, if there is a finite number of states and they are connected, then we should be able to find the path. But if we aren't clever, we may miss the solution even though it's possible to find it. For example, if we had a strategy that says first I'm going to explore in this direction– say this is pouring from cup x into cup y– and then I go in this direction, pouring from cup y back in to cup x, and then I pour the water back again–so I'm continually just taking water and pouring it between two different cups back and forth, those are all legal steps to take, but I'm ending up with an infinitely long path and I'm not making any progress. We'd like to come up with a strategy for exploration, and the strategy corresponds to deciding which path to expand next. Strategy is always there's some path–let's say this one– and we say that's the one we're going to explore from next. To avoid this type of infinite loop, here's some possibilities. One possibility would be don't reverse an action. If you come from state A to state B, don't allow the action that goes immediately back to state A. Another strategy would be to say always take the shortest path first. Out of all the paths that you've built so far, when we go to choose which one we're going to expand next, always choose one of the shortest ones. That way we might start to build up an infinitely long path, but at least we won't continue it. First we'll do another one before we do that one. Then another strategy would be don't re-explore. That is, if we're on the frontier–let's say we're here on the frontier– and we have a move that moves us back out of the frontier into the previously explored zone, then we should not allow that path. My question is check all the strategies that would eventually lead us to the goal. Don't worry about the efficiency of getting to the goal, but which one will eventually get us there and won't get stuck in an infinite loop.
5. 04 Exploring the Space
The answer is shortest first would work. If there is a path, it'll eventually find it. It will waste some time repeating itself, and may not be the most efficient. But we'll get there. Don't re-explore seems more efficient, because it stops off some of these paths. Don't reverse isn't quite good enough, because if we said, okay, we're going to eliminate the steps that go from A to B and then back to A, but that doesn't stop us from going from A to B to C to D and then back to A and having that longer loop and having that be infinite.
6. 05 Pouring Solution
Now let's get to solving the problem and coding it up.
But before I do that, I want to introduce one more piece of jargon, which is if I'm at a particular state, and I decide that that's the endpoint of the path that I want to expand, and I come up with the states you can get to from there by expanding the path and the steps that it takes to get to those states. I call that the successors to this state.
The successors are a collection of states that you can reach and the steps that it took to get there.
Here is my solution. It's a little bit complicated. Let's go through it step-by-step.
1 def pour_problem(X, Y, goal, start = (0, 0)):
2 """X and Y are the capacity of glasses; (x,y) is current fill levels and
3 represent a state. The goal is a level that can be in either glass. Start at
4 start state and follow successors until we reach the goal. Keep track of
5 frontier and previously explored; fail when no frontier."""
6 if goal in start:
7 return [start]
8 explored = set() # set the states we have visited
9 frontier = [ [start] ] # ordered list of paths we have blazed
10 while frontier:
11 path = frontier.pop(0)
12 (x, y) = path[-1] # Last state in the first path of the frontier
13 for (state, action) in successors(x, y, X, Y).items():
14 if state not in explored:
15 explored.add(state)
16 path2 = path + [action, state]
17 if goal in state:
18 return path2
20 frontier.append(path2)
21 return Fail
22 Fail = []
I'm saying the input to this pour problem function are X and Y, which are the capacity of the glass for that. Then the goal, which is going to be an integer, like 6, to say that's how much I'm trying to get to. That can be in either one of the glasses. Then the start state, which I'm defaulting to 0 and 0, saying both glasses have current level 0, but if you wanted you could generalize the problem and pass in something else as what we're starting with. I'm using lowercase x and lowercase y to indicate the current capacity of the glasses.
Here I check and see are we done before we even get going?
Did you give me a start state and say the goal is the have a glass with zero in it? Then we're done before doing any actions. Go ahead and return that. What I'm going to return is called a "path."
The path is a alteration of states and an arrow, which will give a name to each action, and then the other states that it goes to, and we alternate out with the states action states and so on.
Here, if there's nothing to do, it's just a state with no actions. We're going to keep track of the states that we've already explored and that's going to be a set.
We're going to keep track of the frontier. Conceptually, that's a set too, but we're going to pull the items off of the frontier one at a time, so I've made it an ordered list rather than a set.
I know which element of the frontier I want to explore first. So the explored is a set of states, and a frontier is an ordered list of paths. The only path we have so far is the trivial path that says we're starting at the start, and we haven't gone anywhere else yet. That's what we start our frontier with.
While the frontier is left, while there is still frontier states that we haven't explored from yet, we pop off the first one. Pop(0) says take the 0th element of the list, so we're going to pull elements off of the front of the list and push them onto the end of the list. Then say the current state is the last element of the path, so the path goes from one state to the next, and the last element of the path is the current state. Let's take x and y from there.
Then I've defined a successor function that gives me all the successor states and the actions we used to get from there. There should be six of those.
Then if we say if that new state is not explored then it's something new. If it was explored, there is nothing left to do. We're already explored from there. If it hasn't been explored yet, then add it to the explored set, make up a new path, which consists of the old path plus we follow an action to get to the new state.
If the goal number is somewhere in that state, so the goal is 6 and the state is the two levels of the glasses, say 6 and 3, yes, 6 is in 6 and 3. Then we're done. Return that path as the winner, the path that reached the goal. Otherwise, just add this path onto the frontier, and we'll pull something off the frontier later.
If we go all the way through and we run out of frontiers to explore from, then we can't reach the goal and we return fail. You could have Fail be None. I decided to make it the empty list, because all the other things we're returning were lists. Either way, None or Fail, both are equivalent to False in Python if statements. So probably either one would do fine.
Here's my successor function.
1 def successors(x, y, X, Y):
2 """Return a dict of {state:action} pairs describing what can be reached from
3 the (x, y) state and how."""
4 assert x <= X and y <= Y ## (x, y) is glass levels; X and Y are glass sizes
5 return {((0, y+x) if y+x <= Y else (x-(Y-y), y+(Y-y))): 'X->Y',
6 ((x+y, 0) if x+y <= X else (x+(X-x), y-(X-x))): 'X<-Y',
7 (X, y): 'fill X',
8 (x, Y): 'fill Y',
9 (0, y): 'empty X',
10 (x, 0): 'empty Y'
It takes the current levels of the glasses and the maximum capacity of the glasses. What it's going to return is a dictionary of state-action pairs. The state is just an x-y pair of what the levels of the glasses are going to be, and the action is how you got there. We're just going to use strings to represent those actions, so it's just something that we can print out that is otherwise unimportant in the operation of the program.
First I wanted to check that this is a legal state that the fill level of x is less than its capacity and the same for y. Then I said here are the six possibilities. The pouring is complicated. Let's do the filling first.
The filling says:
You can fill X up to its capacity–capital X.
You can fill Y up to its capacity–capital Y.
You can empty X. That'll become 0.
You can empty Y. It will become 0.
Then the fill–there are two cases.
If the total amount of water is less than y, then you can take all the water in the first glass, which is x, and add it into y, so you get y plus x. Same thing in the other direction.
But if the total amount of water is more than the destination that you're trying to pour it into, then you could only pour as much as will fill up the other glass.
We can see that there is conservation of water here. The total amount is x + y minus this difference plus this difference.
I got the definition of my program pretty much just by following out the implications of this diagram.
We're going to keep track of an explored set, never try to return there, expand the frontier, pop off one element of the frontier, add in the new elements, and check when we get to the goal. Then that was all kind of generic for any exploration problem.
Then for the specific water problem, the successor function and the way that was laid out was specific to what we're doing with the glasses.
7. 06 Doctest
Now that was a lot of code again, so I'm really going to need some tests to makes sure I got this right.
Rather than write the types of tests that we had before with the search statements, I'm going to introduce a new type of test. This comes from the standard Python module called "doctest." It stands for documentation test.
The idea is that you can write comments– the sort of comments that go with your class items and with your function items and then automatically have them run its tests. The tests look just like something that you would type into the Python interpreter.
The way doctest knows that you've got a test is you have three-arrow prompt, and an expression is input and the following lines are the output that comes back from that expression. It tests to see if what comes back when you run the test is what was expected.
Here I've typed in what I've done at an interactive session, what the results should be, and then when I make a change to my program I can run it again and make sure I haven't messed anything up.
1 import doctest
3 class Test:
5 >>> successors(0, 0, 4, 9)
6 {(0, 9): 'fill Y', (0, 0): 'empty Y', (4, 0): 'fill X'}
9 {(4, 5): 'fill X', (4, 4): 'X<-Y', (3, 0): 'empty Y', (3, 9): 'fill Y', (0, 5): 'empty X', (0, 8): 'X->Y'}
11 >>> successors(3, 7, 4, 9)
12 {(4, 7): 'fill X', (4, 6): 'X<-Y', (3, 0): 'empty Y', (0, 7): 'empty X', (3, 9): 'fill Y', (1, 9): 'X->Y'}
14 >>> pour_problem(4, 9, 6)
15 [(0, 0), 'fill Y', (0, 9), 'X<-Y', (4, 5), 'empty X', (0, 5), 'X<-Y', (4, 1), 'empty X', (0, 1), 'X<-Y', (1, 0), 'fill Y', (1, 9), 'X<-Y', (4, 6)]
17 ## What problem, with X, Y, and goal < 10 has the longest solution?
18 ## Answer: pour_problem(7, 9, 8) with 14 steps.
20 >>> def num_actions(triplet): X, Y, goal = triplet; return len(pour_problem(X, Y, goal)) / 2
22 >>> def hardness(triplet): X, Y, goal = triplet; return num_actions((X, Y, goal)) – max(X, Y)
23 >>> max([(X, Y, goal) for X in range(1, 10) for Y in range(1, 10)
24 … for goal in range(1, max(X, Y))], key = num_actions)
25 (7, 9, 8)
28 … for goal in range(1, max(X, Y))], key = hardness)
32 [(0, 0), 'fill Y', (0, 9), 'X<-Y', (7, 2), 'empty X', (0, 2), 'X<-Y', (2, 0), 'fill Y', (2, 9), 'X<-Y', (7, 4), 'empty X', (0, 4), 'X<-Y', (4, 0), 'fill Y', (4, 9), 'X<-Y', (7, 6), 'empty X', (0, 6), 'X<-Y', (6, 0), 'fill Y', (6, 9), 'X<-Y', (7, 8)]
35 print(doctest.testmod())
36 # TestResults(failed=0, attempted=9)
For example, at the start here I just want to test out what are the successors of the start state with both glasses empty and when one glass has capacity 4 and the other has capacity 9. In general there are six actions but here a lot of them end up being the same, because if you pour zero into zero either way or if you empty out either of them, it all comes out the same.
We only end up with three states, and they happen to have these labels– (0, 9) filling Y, (0, 0)–we called that emptying Y, but of course emptying 0 gives you 0. It could have been the no opt, but that's just the way the successor function works out. Then (4, 0) is filling X.
More interestingly, if you have 3 and 5 and you fill– so this is testing when we aren't exceeding the capacity, and this test is when we do exceed the capacity. We can see they work out to the right numbers.
Then we solve a problem and come up with a solution and so on.
Doctest is a nice capacity to allow you to write tests this way. You can sprinkle them throughout your program, and then you can run the test. Just say:
1 print doctest.testmod()
which stands for test module. If you give it no arguments, it tests the current module.
When I run this I get the comforting message that there's a test result that is none of the tests failed, and there were 9 that were attempted.
Let's go back and look at the solution.
I'm asking given glasses of levels 4 and 9 trying to find the goal 6. This is the shortest solution possible–fill Y, pour from Y into X, empty X, do the same, empty X again, fill Y into X again, fill Y, and pour from Y into X, and then we end up with a 6 in Y.
We can solve problems more generally.
Here I've defined a function num_actions, which says given an X and Y capacity and a goal how long does it take to solve the goal–the total number of steps it's going to take. Then I asked here for all values of X and Y less than 10–for all capacities less than 10– and for all goals smaller than the capacity, what's the longest? What's the hardest? Which combinations of those takes the most actions? The answer was if you're given glasses of size 7 and 9 and asked to pour out 8, that's the hardest problem within that range.
8. 07 l Bridge Problem
Now let's introduce another problem. We have a cavern here with a rickety bridge connecting it. On this side, which we'll call "here," we have a collection of 4 people who want to get to the other side, which we'll call "there." Part of the problem is this is nighttime, and it's dark. Fortunately, our team has a flashlight or a torch. The setup is such that the bridge is so rickety that only 2 people at a time can cross, so either one or two people can cross. It's so dark that they need the flash light with them. For everybody to get across, two people are going to have to go across. One is going to have to come back with the flashlight. They'll shuttle each back and forth like that. Now, each of the people has different physical abilities and fear levels, so they each take different times to cross the bridge. This person is speedy, takes 1 minute, 2 minutes, 5 minutes, and 10 minutes. The question is what combinations of actions will get everybody across the bridge the fastest.
9. 08 q Representing State
Let's take our usual approach– start making an inventory of concepts and figure how to represent them. We want to represent a person, a collection of people, and probably it looks like we want to have two collections of people. One, the collection of people on the here side, and one, the collection of people on the there side. We also need to represent the light or the torch. From there it seems like that's about it, and the other concepts we need are the concepts we already had of states and paths. Now, how about the representation choices. For person, well, I hate to reduce people to numbers, but in this case that seems like the perfect thing to do. This person, regardless of all his wonderful individual qualities, we can just represent by the number 5. How about a collection of people? We could represent a collection as a tuple–1, 2, 5, 10– as a list, as a set. There's also this data type in Python called a frozen set. What I want you to tell me is of these four, which do you think would be okay for representations just in terms of being able to to manipulate them and calculate the successors. Which of these are hashable? Hashable is important, because if we're going to use the same type of technique we used before for our search we had our explored set, which was a set of states, and members of a set have to be hashable. That's a property that we might want to worry about. Now, I should say one more thing in that the description of the problem it was explicitly stated that each of the people has different speeds. That bothered me a little bit, because I could certainly imagine two people having the same speed. But let's just solve what we were asked to solve where every person has a distinct speed.
10. 08 s Representing State
The answer that all four of these representations would be fine. We can generate successors by appending or adding elements to set lists, tuples, or frozen sets. None of those is too hard to do. It's a little bit easier with sets than with the other ones. In terms of hashing, the immutable data types–frozen sets and tuple– are hashable, and the mutable types–list and set–are not hashable.
11. 09 p Bridge Successors
Now, out of those many choices, I made a choice to say I'm going to represent as a tuple of (here, there, t), where "here" represents everything that's on this side, "there" represents everything that's on that side, and "t" is the total elapsed time since the start. I'm going to represent here and there with frozen sets, because those are hashable. So this collection here would be the frozen set consisting of {1, 2, 5, 10}, and I'm going to just use the string "light" to represent the flashlight. There would be the empty frozen set. Now, consider this state here representing the start state. What are the successors of that state? Well, any one of the people could go across. They've got to bring the light with them. In the successor state, the light will definitely be there, and it will not be here. It can only be in one place. At least one of the people will be over there and possibly two of the people, so all combinations of sending either one person or two people to the other side, those will each be distinct successor states. Let's see–we've got 4 x 3 is 12, but order doesn't matter, so there's 6 of those. Then 4 more, so it looks like there should be 10 successor states. What I want you to do is write for me the successor function. We're calling it bsuccessors, because we already had a and we're on to b. Or b could stand for "bridge." Remember that a result of the successor function is the dictionary of state action pairs. A state is this (here, there, t) tuple. Here and there have to be frozen sets. The frozen sets contained people–1, 2, 5, and 10– and/or this light, indicated by the string "light." Show me the function that will generate all the successors. Here I've given you a hint of here's a way to break up the state into those three variables. Then put your code here. Oh, one more thing I forgot is what are the actions. Well, let's say that an action will be represented by the character string arrow going to the right if we're moving from here to there and an arrow going to the left if we're moving from there to here.
12. 09 s Bridge Successors
Here's my solution. I've got to say that my solution came out a little bit more complicated than I expected it to. I think maybe I made a bad choice for the representation. I threw in the flashlight along with the set of people, because I figured you want one set to represent everything that's on one side. But I'm think now after this came out the way that it did that maybe I should have had the flashlight be a separate part of the state. In other words, have the state be a 4-tuple, not of things that are here or there but of people that are here or there, then the time, and then a fourth element being the flashlight saying where is the flashlight. That could either be true or false, saying it is it here, or it could be a character string, saying it's there or here, or it could be a integer–0 or 1. I think it might've been easier if I'd chosen one of those representations. But it didn't bother me enough to go back and make a change. If you want to, you could spend time refractoring and change that. I'm going to just push ahead. Here's what I did. I said if the light is here, then let's look at all the people in here. We'll look at all the pairs of people–A and B. To make sure that they're people, I have to say that they're not the light. For all pairs of people A and B, we can generate a successor state, which is the set of people that were here minus the two people and the light, because the light is going to move from here to there. The second part of the successor state is everything that was already over on the other side on there unioned with the things that are coming over, which are people A and B and the light. Then the time is the time plus the maximum time that it took for A and B to get over. Then I know it says in the specification here that the action is represented just by an arrow. If I want to get the problem right I would do that, but then I decided later on that maybe the action should be more than just the arrow. Maybe the action should also tell who went across. I have the option of doing thing. If I want to just solve the problem the way it was specified then I would return just the arrow to represent the action, and I would do the same thing over here. One subtlety of this that worked out well in my favor– it's a little bit messy dealing with frozen sets. I don't like the idea of that the name is so long, but I didn't have to consider separately the idea of one person going across and two persons going across. Because we were dealing with sets, the set of people a, b when a is equal to b is equal to 1 person. I get the 1 person crossing for free. That's one nice thing about my representation. But notice that everything is in flux here. I'm trying to choose a good representation. I'm changing my mind as I go along. Should the actions be represented by a single arrow or should they be represented by an arrow along with the names of the people that are going? That's all up in flux. I should say that that type of flux is okay as long as it remains contained. If you have uncertainties that are going to cross barriers between lots of different functions, then probably you want to nail them down. If you think that they're contained, then it's okay to have some uncertainty and be able to explore the exact options later.
13. 10 q Bridge Solution
Now I'm going to show you the solution to the search problem rather than try to make you do it yourself, because there are still a few tricks here that are different from the previous search problem. I'm going to define problem, which takes a sequence of elements here. If you want, you can pass in a frozen set of {1, 2, 5, 10} or whatever, but if you didn't I'm going to go ahead and do that kind of version for you. I'm going to make it into a frozen set, and I'm going to add in the light in case you forgot to specify that. You can just ask bridge_problem of the list 1, 2, 5, 10. I'll take care of it all for you. Like before, the explored set starts off being the empty set. The frontier starts off being the one initial state, which is the frozen set we just made up for everything that's on the here side, and empty set for everything that's on the there side, and 0 for the elapsed time. The idea is to get everybody away from here onto the other side. If we were given a trival problem where there was already nobody here, then we're done and we return that initial state. Otherwise, just like before, we start popping things off the frontier. Just like before we're looking at our successors, and the only difference is down here. Whereas before we put a path on the end, and we were expanding our our frontier and taking off the shortest path first from our frontier, because in the previous problem, in the water-pouring problem, the best solution was to find as the solution that was shortest, with the smallest number of steps. In this problem, the best solution is defined as the one with the smallest elapsed time where the elapsed time of a path is the second element. That's the t element here of the final element of the path. That would be the total elapsed time of a path. So we sort the frontier by the total elapsed time. Now it is a little bit wasteful here that we're going through this loop, we only added in one new element, and we sorted the whole thing. Python's actually pretty good at that type of sort. There are other ways to make that more efficient, but just conceptually that's what we're doing. We always want to have the frontier sorted, so that we're taking the fastest time first. I typed that program in, and I ran it for the very first time. Bridge_problem([1, 2, 5, 10]). I got an answer back. Remember, the answer is a path, which is an alternation of states and actions. We can pick out just the actions, like this, by asking for the path and then taking a slice of that path, starting at element number 1, going to the end, and giving us every other element. That'll be just the actions. Those are these three actions. That's my proposed solution that my program came up with. My question is is that correct? Yes or no?
14. 10 s Bridge Solution
The answer is no, that's not correct at all. I've been cheating a little along the way in that I've been showing you solutions that I got the second or third time once I'd debugged them and got them right. This time I wanted to show you a little bit of the debugging process. I got something wrong here. I don't always get them right the first time. This is so wrong looks what's happening. I said the first move is at the 5 and the 2 go across together. It seems like a perfectly reasonable move. They're going from here to there. The second move was that the 1, by his or herself, comes back from there to here. But 1 isn't even over there. How could 1 come back? I must have messed up the successor function. Let's take a look.
15. 11 q Debugging
Here's the problem. I was careful about doing the here case. I made up this nice expression, but then I did a copy and paste, and I edited the expression, and I swapped around the here and the there in this part. When I created the new state, I did that correctly. But down here I'm iterating over the people that were here. I'm trying to have candidates move from there to here, and I'm iterating over people that are here. That doesn't make any sense at all. I've got to fix that. Now the question is is it going to run this time. I found a bug. I fixed it. Is the program correct now? Yes, no, or not enough information, you can't tell yet?
16. 11 s Debugging
I think the right answer is that you just can't tell. I'm hopeful that it's going to work, but I know I fixed one bug. I don't know whether there are other bugs lurking in there.
17. 12 q Did it work
Now I run it again. This is the path I get. These are the actions in the path. Let's see if it makes sense. Now 1 and 2, the two fastest people, go over first, That looks like a pretty good solution. It came up with a total time of 19. The question is is the program correct now? Yes, it is. No, this example is wrong–there might be a faster example than this and it didn't find it? Or, no, this example is okay. It is the fastest, but the other examples are wrong. Or you still can't tell.
18. 12 s Did it work
The answer to that is that this example is actually wrong. It does get everybody across, and it gets them across in 19, but there's another solution that's faster than that. So let's look at our program and see what we did wrong and why we missed the fastest solution.
19. 13 q Improving the Solution
Unfortunately, we got the wrong answer. Yes, we got a path that leads to the goal, but we didn't get the fastest path. Let's see what went wrong. We had our start state, and then we started expanding that and moving out. That defined our frontier. Then we were very careful about sorting the elements on the frontier, and then we pulled off the very best, the one with the least cost. Then expanded out from there. Let's say the cost of getting to the end of this path with 14, this one 15, this one 16. This is the lowest cost path, we expand that first. Let's say one of the steps cost 5, so that gets us to this state with a cost of 19. Let's say that is in fact a goal state. Now we just stopped there. We said we took off the least cost path. We expanded it. We found a goal. We're done. When we were looking for the shortest path in terms of the least number of steps, that was the right approach, but when we're looking for the least cost path, that's not the right approach. Because even though we pulled off the cheapest path here–the one with the lowerst cost– here's another path that has a higher cost, but if we expand that there might be a step that only costs 2. We get to this state with cost 17 and that's also a goal. So we made a mistake. We stopped here when we got this result that was 19 when we really wanted this result that was 17. I think the problem was we were prematurely acting. We said just because this was the fastest solution here, we went ahead and took one step away from the fastest and accepted that when that might not be the best answer overall. How can we fix this? One possibility would be to exhaust the frontier. That is, we've got a frontier here. Even though we find a solution from the first element of the frontier, we keep going until we visit everybody on the frontier and give everybody a change to find the better solution. Another possibility is to give everybody one more chance. Once we've found the first solution, now we say, okay, everybody on the frontier gets one more step to see if they can find a solution. The third possibility would be to test later. That is, when we generate this solution, we don't check right here to see if it is a solution. Rather, we just go ahead and throw this onto the frontier and only check to see if it's a solution when we pull the next element off of the frontier. Rather than when we generate a new node and we're about to add them, do the checks later once we've pulled them off the frontier. Now tell me which, if any, of these will work to give us this fastest solution.
20. 13 s Improving the Solution
The answer is exhausting the frontier won't work, because the frontier might be infinite. In this particular problem, there's only a finite number of states, but in some problems there might be an infinite number. If we kept on generating new elements onto the frontier we may never get to the end. Doing one step won't do it either. In this case, if once we found the solution from this 14, we then gave all the other guys one step, it would work in this case. But it might be that it took two steps. Maybe from the 15 there'd be one step that costs 1 and another step that cost 2. I might not just be one step, so that's not going to work. The test later part will work. The reason it works is because now we've guaranteed that everybody on the frontier is sorted, and we're pulling off the shortest one first. If we put it back onto the frontier rather than recognizing immediately that it's a goal, then since we're pulling them off in order of increasing cost, then we know that the first one we pull off the frontier that is a goal that must be the cheapest path to the goal.
21. 14 p Modify Code
What I want you to do is take– this is the same version of the bridge problem solver that we saw before, and I want you to modify this so that it tests for the goal later after pulling a state off the frontier, not when we're about to put it on the frontier.
22. 14 s Modify Code
Here's the solution. Two changes are here and here. We pull up the test to this point where we check for solution when we pulled the best path off, and we check for our goal only there, and we don't check for the goal when we're putting something on the frontier.
23. 15 p Adding Tests
It looks like this is a tricky problem. There are lots of cases that we have to take care of. It seems like a good idea to right some more tests. I've done that here. I've written a few tests. I really should write a lot more. What I want you to do is write at least 3 more tests and run them. I don't have a way of knowing for sure whether you've come up with good ones or not, but go ahead and add at least three more tests to this class of test
24. 16 p Refactoring Paths
Now, mostly we're looking for correct code. If you wrote some more tests, you may start to have some more confidence in the code that we have. We're also considering efficiency to some degree. It seems like there's a big problem with the efficiency of the program we have so far. Let me show you one of the issues. Now we represented states as a (here, there, t) triplet. The problem with this is there can be two states that have identical here and there's but differ in the t, and they're going to be considered different states. Why is that a problem? Consider this problem. We have two people–one who takes 1 unit to cross the bridge, and one who takes 1000. It seems pretty clear there is an easy solution. The two of them go across together. It takes 1000, but look how we're going to explore this space. We're going to start out in the initial state that took time 0, and then we're going to start adding things to the frontier. Out of all the ways we could cross, the one that adds the least is for the 1 to go across by himself. Now he's on the other side with the 1 on the other side and the 1000 on the original side. That only took 1 step. Now what's the fastest thing we can do after that? We could take 1 more step and go back to the original state. Here we had 1 and we'll call K for the 1000 on the left-hand side. Here K was left behind and 1 went over to the right. Here we took one more time unit, and we had 1, K on this side. If we continue taking the fastest step we can, we'll get to another distinct state where K is on this side and 1 is on the other side. The flashlight is always going with the 1. We keep on going on like that. We'll go out 1000 different steps. Each of these will be a distinct state, because this will be the state with time t equals 0. Here time t equals 1, t equals 2, t equals 3. But really, although it looks like we're getting different states, in another way of looking at it, we're always getting the same state. We're just going back and forth from here to there and back to here and back and back. We're going around in circles. In order to recognize that these are in fact the same states, we're going to have to take t out of our state, and we're going to have to deal with the t someplace else. We want our representation of a state to be just (here, there). We've got to figure out someplace else to put the t. I'm not sure what the right way to do it is, but why don't we do it this way? We have a path, which is state, action, state dot, dot, dot– keeps on alternating between states and actions. Let's change that so that the path is a state followed by a tuple of the action and the total time it took after applying that action, then the next state, then the next action and the total time after applying that, and so on. That'll be our new representation. States are going to look like that, and paths are going to look like that. Now, I want you to write the new successor function for the bridge problem. We'll call it bsuccessors2–the "2" just to keep it distinct from the first version. Again it returns a dict of state-action pairs. A state now is just a two-tuple of (here, there), and the here and there are still frozen sets. It's pretty much the same except we dropped out the time t. Go ahead and implement that for me.
25. 16 s Refactoring Paths
Here it is–pretty straightforward. I just dropped out the time, and I'm just building up these two components.
26. 17 p Calculating Costs
Now, we got rid of the times in the successor function, so we've got to put them back in someplace. I'm going to generalize a little bit, and instead of talking about times, I'm going to talk about costs for a path. I'm just thinking of maybe we might want to do some other problems that also have paths in them and that aren't dealing with optimizing time but are dealing with optimizing some type of cost. What I want you to for me is to define this function path_cost, which takes a path as input and returns the total cost of that path. That's already stored away. We don't have to compute anything new. Because we decided that our convention for paths was it was going to be stored there. That is, we said that a path is equal to a state followed by an action and a total cost followed by another state, etc. Here I've just said, well, if we don't have any actions there or if it's the empty path, then do one thing. Otherwise do something else. Then I also want you to find the bridge cost–bcost is the abbreviation I'll use. That's the cost of an individual action. An action in this domain is something like 2, 5, arrow to the right. I want you to figure out what's the cost of that action.
27. 17 s Calculating Costs
Pretty straightforward. If we don't have at least 3 elements in the path, that means we don't have an action there. It's just an individual state. The cost of that should be 0. Otherwise, we look at the second element from the end. There's a final state and then there's a final action. That should be the final action and total cost–this tuple–we just return the total cost. For the bridge cost of an action, it's just the maximum of the two times.
28. 18 l Putting it Together
Now we've got our new successor function. We know how to deal with costs. Now it's time to put it all together. It's a little bit tricky, so I'm not going to ask you to do this as a quiz. If you want to you can pause the video now and do it on your own. You're certainly welcome to give it a try. I'm going to go ahead and show it to you. Okay, here it is. The tricky part is just keeping track of the costs and putting them in the right location. Just like before we're popping paths off the frontier. We're checking to see if we hit a goal. We're keeping track of states that we've already explored. But now we're doing something new. We're computing the cost of the path that we just popped off, and that's just pulling the cost out, because we've already computed it and stored it in the final action. Then for each of the successors, we figure out the total cost is the cost of the path that we already computed so far plus the bride cost of the individual action. Total cost so far plus cost for one more action, and then we just throw that into the path. The new path is equal to the old path plus the action total cost tuple plus the state that we end up with. Add that to the frontier and we're done. I just define this simple one-line function here. The final_state of a path is the last element of the path. I use that there. Here is adding to the frontier. Now, it could just be throwing it on there the way we did before, but there's a tricky part here. The complication that I want to deal with here that we haven't dealt with before was there may be two different paths that end up in the same state. If that's the case, we want to choose the best one. We don't want to get to the state from a path that's more expensive. We look at see–is there a path that gets to the state that is already on the frontier? If there is, then check to see which one has a better path cost and use that.
29. 19 l Generalizing
The moral of the story is this is tricky. There are a lot of cases to deal with in getting this kind of search just right, and we made a couple mistakes along the way. I sort of duplicated the history of the field. There a couple tools we can get to avoid mistakes. One tool is to write lots of tests, and I just didn't do enough testing. I wanted to go fast. I wanted to be able to show you some of the interesting ideas. I put in a few tests, but I really need more to have confidence that I've got this right. The second thing is to use, or better yet, reuse existing tools. Every time I do a search, I don't want to be rewriting this search routine from scratch, because it is tricky and I will make mistakes. Rather I want to write it once or have somebody else write it once and then reuse it. In order to do that, we're going to have to figure out how to generalize. I've written a a function that's good only for solve the bridge problem through search. I want to write a search function that can solve a wide variety of problems. Then I want to reuse that so that I'm not repeating mistakes, and I'm not introducing new errors.
30. 20 q Missionaries and Cannibals
Let's do an example to figure out how to do generalization. What do we generalize over? Well, we generalize over problems. So we're going to need another problem. Rather than have a problem dealing with costs, which we saw were complicated , let's just do a problem where we're finding the shortest path. That is, the least number of steps to a solution. I'm going to choose a classic problem called the "missionaries and cannibals" problem. It works like this: there's a river we have to cross, similar to the bridge but this time it's a river. We've got a boat, and on this side of the river, there are 6 people. No flashlight, but a boat and 6 people. Three of these people are missionaries, and three are cannibals. The goal is to get everybody over to the other side. What makes it hard is that there are two rules. One, at most 2 in the boat. One person can go in the boat and cross from one side to the other, but it takes either 1 or 2 people to get the boat from one side and to get it back. The other rule is that we don't want the cannibals eating the missionaries. If we leave more cannibals that missionaries on either side of the river– either on this side or over on this side– then the cannibals are going to gang up and eat the missionaries, and we won't be able to accomplish getting everybody across. We have to shuttle them back in forth in such a way that this never occurs. Now, let's try to come up with a good representation for state. One possibility would be to have a set of missionaries, a set of cannibals, and a boat–let's call that a Boolean, yes or no, saying what's on the starting side and leaving out what's on the other side, because we can figure that out. Given that we know we three missionaries, If there's a set of 2 on one side then the other side there must be 1. Another possibility is that we have 3 integers: the number of missionaries, the number of cannibals, and the number of boats that are on the starting side. These are all integers. Then the third possibility is that we have 6 numbers: the number of missionaries, cannibals, and boats on the first side, and the number of each of those on the other side. It may be subjective which of these is best, but I want you to tell me which of these would sufficient for representing the state.
31. 20 s Missionaries and Cannibals
The answer is that all of them would work. All of them have everything you need to know to solve this specific problem of three missionaries and three cannibals in the boat.
32. 21 q Generalized State
Now the next question is what representation for states should we use if we want to generalize this problem. So that we're given an initial state when there can be any number of missionaries, cannibals, and boats on one side of the river and any number on the other. Which of these representations is sufficient under those conditions?
33. 21 s Generalized State
In this case since we don't know that there's only three missionaries, we need to have both sets of numbers. We can't just say there's two missionaries on the left; therefore, there's one on the right. We don't know how many are going to be on the right. So this six-element tuple would do the job where these two wouldn't.
34. 22 p csuccessors
Now I want you to define the successor function for this problem. We'll give you a hint that a state is of that form. Return all the successors. The successors should be a dictionary as before. We want to include successor states that result in cannibals being able to eat, but such a state should have no successors itself. In other words, we're free to generate a successor state that has, say, two cannibals and one missionary in one location, but if we're given such a state then we should return the empty dictionary of successors.
35. 22 s csuccessors
Here's my solution. The key to my solution is a list of deltas, of differences in the states that correspond to these moves. What do I mean by that? One thing we can do is send two missionaries from a side with the boat to the other side. That would be a difference of 2 in the missionaries. We would add 2 to one side and subtract 2 from the other side and not change at all the number of cannibals and change the number of boats by 1. Or we could send 2 cannibals, or we could send one of each, or we could send only 1 missionary or cannibal. There are 5 possible moves, basically, depending on where the boat is. That's what csuccessors says. First we check for states with no successors. If there are more cannibals than missionaries but there are some missionaries, then they're going to get eaten, and so we return the empty dictionary as a result. Otherwise, we're going to collect up the number of items in our dictionary, and we're going to do that by going through these deltas and subtracting the deltas from the side where the boat is and adding them in to the other side. We have two directions we can go from left to right, start to the other side, or from the other side back to the original side. I made use here of vector addition and subtraction. I take the current state, which is 6 numbers, and I add or subtract these deltas. That's what these definitions say. Now, it would nice if this type of vector arithmetic was built into Python, and there are versions called "numeric Python" where you can do that, but here I had to write these functions myself.
36. 23 l mc problem
Now let's write a function to solve the missionary and cannibals problem. It takes a start state. Here's the normal problem: 3 missionary, 3 cannibals, and 1 boat on the start side. Nothing on the other side, and it takes a goal state. The goal state is not specified. It's just the opposite of that–3, 3, 1 on the other side. Nothing on the original side. The state is this 6-tuple, and we're trying to find a path from the initial state to the goal state. In fact, we're trying to find the path with the least number of steps. I'm not going to ask you to do this as a quiz. If you're enthusiastic, you can stop the video now and go ahead and solve it on your own, but now I'm going to go ahead and show it to you. Here's a solution that looks pretty much like the pouring water problem. We check to see if the goal is None, then we fix up a nice goal. We check to see if we've accidentally already reached the goal at the start. Then we just search for the shortest path.
37. 24 q Shortest Path Search
Now let's generalize. Let's take the specific solver–we had a specific one for the pouring problem and one for the missionaries and cannibals. Let's generalize them. I'm going to call the generalization "shortest_path_search." That's a search for the shortest path that reaches a goal. Let's take our inventory. The concepts we have to deal with–we've got paths, states, actions, successors. We have a start state. We have a goal. Now let's figure out how we're going to represent each of these concepts. Paths we already had. I don't see any reason to change. We have [state, action, state…]. Notice we're just doing shortest_path_search. We're not doing best_cost_search. We don't need to put in the total cost in here. We can just have the action by itself. We have states, and here the states can be atomic. We don't have to know anything about the states. In other words, a state can be anything that a particular problem wants to deal with. Shortest_path_search doesn't have to know about that. Now, why is that the case? Because shortest_path_search can interface with states through these two functions– through successors and through the goal function and through the start state. What do I mean by that? The start state is going to be some atomic state. We don't know anything more about that. Shortest_path doesn't know anything about that. When we go to use shortest_path_search for a particular problem, then we have to specify what a state looks like, but shortest_path_search itself doesn't have to know. All it has to know is that if you give the start state to the successor function– so successor will be a function which takes a state as input and returns a dictionary of state-action pairs. Now, given that initial state that we passed in, we can generate new states and new actions. So the actions also are atomic. Shortest_path_search doesn't have to know anything about the representation other than that this is where they come from–from the successor function. Now, what about the goal? Well, we could specify an exact state that we're looking for, but sometimes we're looking for multiple states. We could specify a set of states, but sometimes the set of states is really big. There's lots of states that satisfy the goal. Instead, let's have the goal be a function. Its's a function. When you pass it a state it returns a boolean. True or False? Is that the goal? With that now we're ready to specify shortest_path_search. Shortest_path_search is going to be a function. It's going to take some inputs, and it's going to return a path, and return failure as a path if it can't find a solution. Now the question is out of this inventory, which of these things do we have to pass into shortest_path_search to allow us to solve a problem? Check all those that apply.
38. 24 s Shortest Path Search
The answer is what we have to pass in is the start state– you've got to know where you're starting from, a successor function– you have to know where you can get to from the start state, and a goal function–you have to know when you're done applying successors. That's it. We don't need to pass in any other actions or states or paths, because those can all be generated from these three.
39. 25 p sps function
Let's see if you can write that function. I've left you with the missionary and cannibals problem as sort of a template, but I want you to generalize that to write shortest_path_search, which takes a start state, a successor function, and a is_goal function and returns the shortest path.
40. 25 s sps function
It's pretty easy. We just took the template that we had for missionaries and cannibals and just replace these general functions–is_goal and successors– put them in here rather than putting in the specific functions for the missionaries and cannibals.
41. 26 p Cleaning up MC Problem
Now let's complete the generalization. I'm going to define missionaries and cannibals problem, and we'll give it a 2 just so we can tell the two versions apart. It takes the same arguments as before. You may need some initialization code to get going. Then I want the body of the function, the main part, to just be a call to shortest_path_search with the appropriate arguments inserted. If you need to you can define other functions outside of here if that's necessary.
42. 26 s Cleaning up MC Problem
Here's my solution. I had to write some code to fix up the goal if it wasn't specified. Then it's just a single call. We call shortest_path_search with the start state we were given, with the csuccessors function that we've already defined, and then with a goal test. The goal test is that everybody is gone from the start side of the river. That we define this way.
43. 27 p Lowest Cost Search
Once again generalize. This time I want to go back to the bridge problem and generalize that. What we're going to come up with is lower_cost_search, and that'll take some arguments and again return a path, but let's figure out what we need. Yes, we're going to need the start state just like before. We're going to need a successor function, and we're going to need a goal function. In addition, we're going to need one more thing. We're going to need to know the cost of an action. That's going to be necessary. It's going to have to be a parameter to the function. We'll have the start, the successors, the goal, and the action cost and return from that a path. There's a notion of action_cost, and as part of our inventory of concepts, there's also the notion of path cost, but that won't have to be passed in as a prohibitor. Let's see if you can define for me lowest_cost_search, which takes these four parameters and should perform the same type of search as we saw previously with the bridge problem.
44. 27 s Lowest Cost Search
Here is my solution, and I got it by copying the code from the bridge problem and just generalizing it. Just replacing the B successors with successors and action_cost and so on.
45. 28 p Back to Bridge Problem
Now let's go ahead and redefine bridge problem in terms of lowest cost search, thereby generalizing it. In the initialization code you might need here a single call to lowest_cost_search. Any other functions you need to define here.
46. 28 s Back to Bridge Problem
Here's my solution. I have to define the start state given a set of people that are on the here side. I have to define the here side and just make sure that we throw in the flashlight there. Then on the other side there's nobody. Lowest_cost_search–starting from the start state, we've already defined the successor function. I'm defining a new function to test for a goal. We already defined the cost function. The new function to test for the goal is right here. It says if not here–in other words, if there's nothing here, if there's nobody here at all, it's the empty set, or if here is only the set of the flashlight. That normally wouldn't happen, but I guess it could happen if the initial problem was there's no people and just a flashlight. Then you've got a solution with doing nothing at all. I just wanted to make sure I covered that trivial case.
47. 29 l Summary
Congratulations. You made it to the end of the unit. What have we learned? Well, first of all, some problems require search. What I mean by search is you need to put together a sequence of steps, starting from a start and keep going. You don't know how many steps it's going to take, and you're trying to optimize some factor. There are different kinds of search. We just scratched the surface, believe me. It's a gigantic field with all sorts of different algorithms and different types of applicability for these different algorithms. There are many complications we didn't cover, but we covered two– the shortest_path and the least_cost search. These are two of the most useful. Third, search is really subtle. There are lots of possible problems lurking in there and many that we didn't even cover yet. What that means is where there is subtlety, there is likely to be bugs, and there are even some bugs where there is no subtlety. That means we have to be careful. We have these two tools for combating bugs. One is lots of tests, and the second is standardized tools. That is, we work really hard to make a tool that we know works and has got all the bugs out of it, and then we reuse that tool. Part of that reuse is generalization– to look at a specific problem and say, "Here we solved this specific problem this way," and to generalize it, to say here's part of that that I think we're going to use over and over again. Let's break that out, and now we'll have two parts to the solution. We want to be thinking about this specific problem, and we want to be thinking about the more general problem. We want to be allocating our work to one or the other appropriately. Congratulations again. You learned a lot of important concepts. You did a great job in writing some very complex programs.
Unit 1 Personal Notes
Write a Poker Program
Poker(hands) returns best hand
Hand rank takes a hand and maps to something like 2 pair (maybe something else)
n-kind (2 of a kind, 3 of a kind, etc)
need to accommodate for hands, cards, ranks, suits
max([3,4,-5],key=abs) == -5
return max(hands, key=hand_rank) return the max of the hands where the hand_rank function determines this ranking
Now create a test function to test out the function—need to test in order to know when you write this correctly:
def test():
"test cases for the functions in the poker program."
sf = "6C 7C 8C 9C TC".split()
fk= "9D 9H 9S 9C 7D".split()
fh="TD TC TH 7C 7D".split()
assert poker([sf,fk,fh]) == sf
return "tests pass"
print test()
# ———–
# User Instructions
# Modify the test() function to include two new test cases:
# 1) four of a kind (fk) vs. full house (fh) returns fk.
# 2) full house (fh) vs. full house (fh) returns fh.
# Since the program is still incomplete, clicking RUN won't do
# anything, but clicking SUBMIT will let you know if you
# have gotten the problem right.
def poker(hands):
"Return the best hand: poker([hand,…]) => hand"
return max(hands, key=hand_rank)
"Test cases for the functions in poker program"
sf = "6C 7C 8C 9C TC".split() # => ['6C', '7C', '8C', '9C', 'TC']
fk = "9D 9H 9S 9C 7D".split()
fh = "TD TC TH 7C 7D".split()
assert card_ranks(sf) == [10, 9 ,8 ,7, 6]
assert card_ranks(fk) == [9, 9, 9, 9, 7]
assert card_ranks(fh) == [10, 10, 10, 7, 7]
assert poker([sf, fk, fh]) == sf
assert poker([fk, fh]) == fk
assert poker([fh, fh]) == fh
assert poker([sf]_ == sf
assert poker([sf] + 99*[fh]) == sf
assert hand_rank(sf) == (8, 10)
assert hand_rank(fk) == (7, 9, 7)
assert hand_rank(fh) == (6, 10, 7)
# Add 2 new assert statements here. The first
# should check that when fk plays fh, fk
# is the winner. The second should confirm that
# fh playing against fh returns fh.
Extreme Values: Important part of testing is to test extreme values. Add a test to make sure if one hand is played, or zero, or 100, nothing is wrong
To rank hands, can use tuples: (7,9,5) (rank of hand, number for rank, kicker)
-lexigraphic: comparing lists of numbers of strings
ranks= ['–23456789TJQKA'.index(r) for r, s in hand] index value will change T into 10, J into 11, etc
use set to combine like items in a list
for r in ranks:
if ranks.count(r) == n: return r #count number of times element is found, if n then return it
lowpair = kind(2, list(reversed(ranks))) #reverse the order of the list
return [5,4,3,2,1] if (ranks == [14, 5, 4, 3, 2]) else ranks this hardcoded fix will correct the Ace as 1 (originally 14)
deck = [r+s for r in '23456789TJQKA' for s in 'SHDC']
def deal(numhands, n=5, deck = [r+s for r in '23456789TJQKA' for s in 'SHDC']):
random.shuffle(desk)
return [deck[n*i:n*(i+1)] for i in range(numhands)]
def hand_percentages(n=700*1000):
counts = [0] * 9
for i in range(n/10):
for hand in deal(10):
ranking = hand_rank(hand)[0]
counts[ranking] += 1
for i in reversed(range(9)):
print "%14s: %6.3f %%" % (hand_names[i], 100.*counts[i]/n)
Dimensions of Programming:
"Elegance is not optional"
"ity": clarity, simplicity, generality
Program will occupy a space in the multidimensional space above: will change over time
-Might want more features, might run slow, might crash, need to move it around
-Program that is more elegant it's easier to change/maintain—buying time in the future
"The best is the enemy of the good" if you are striving for perfection, you might be wasting time that could be doing something else. Make good trade-offs for whatever you change to program/space, always a cost (takes time. learn to make the correct trade-offs)
Refactoring: take a program and change it around, so it's clearer and easier to maintain
"DRY: Don't Repeat Yourself" Principal
def unzip: return zip(*pairs)
Understand the problem
Define the pieces (cards, hands, ranks, suits)
Reuse (try and reuse pieces that exist)
Test (write tests)
Explore the design space (determine where you want to go and know when to stop)
Correctness
for I in range(N-1)
random.randrange(I, N))
Analyze algorithms with test cases to display probabilities
Pure functions: computing code easier to test
Impure functions: do (subroutines, not functions—just have an affect)
Pure test: assert sorted([3,2,1]) == [1,2,3]
Subroutine/impure test:
input = [3,2,1]
input.sort
input = =[1,2,3]
List Comprehensions:
Bad code:
Using list comprehension can solve this in one line of code:
Uppercase_tas = [name.upper() for name in udacity_tas] name is what each iterable will be called in list udacity_tas
So this makes a list by iterating through each "iterable" called "name" in list udacity_tas and converts them to uppercase by using name.upper()
ta_facts = [name + ' lives in ' + country + ' and is the TA for ' + course for name, country, course in ta_data]
ta_facts = [name + ' lives in ' + country + ' and is the TA for ' + course for name, country, course in ta_data if country != 'USA'] putting an if statement at end
course.find('CS3') != -1
def best_hand(hand):
"From a 7-card hand, return the best 5 card hand."
return max(itertools.combinations(hand,5), key=hand_rank)
Set Theory:
Set: An unordered collection of elements with no duplicates
Use a Ven Diagram to show where the intersection of sets are
Fruit&Vegetable (where Fruit and Vegetable): Tomato
Fruit – Vegetable (where Fruit not vegetable): Apple, Orange
Fruit|Vegetable (where fruit or vegetable): Apple, Orange, Broccoli, Carrot, Tomato
Fruit^Vegetable (where fruit or vegetable but NOT both): Apple, Orange, Broccoli, Carrot
Sets do not care about order and do not duplicate
List Methods: Count, Sort, Reverse
list.count(x): returns number of occurrences of x in list
list.sort(): sorts the list in place
list.reverse(): reverses the elements of the list in place
in place: only modify existing list, do NOT make a new list (because lists are mutable)
Learn how to remove complexity, leave to simplest solution
Concept Inventory:
First middle
Two ways to solve this: Deduce (need to be clever, harder to write) or Try (make computer do all the work)
5! = 120 if 5 property and 5 assignments
Millions = easily done, fast
Trillions = need a better way than brute force
Billions = maybe
Permutation: all combinations of possible ordering
A == B checks that values are equal
A is B checks that whether two objects are identical (handles small integers as single identical objects)
Generator Expressions: only execute when you say "next"
Rewrite generator expression to include limitations earlier to eliminate redundancy
Timing program:
def timedcall(fn,*args):
t0 = time.clock()
fn (*args)
return t1-t0
timedcall(function,1,2,3)
Aspect-oriented programming, separate out the different aspects
yield 0
for i in ints(1):
yield +i
yield –i
infinite loop: while True:
For Statements (Clauses): The Whole Truth:
for x in items: calls this an iterable (strings, lists, generators)
print x
it = iter(items)
x=next(it)
except Stop Iteration:
def c(Sequence):
"""Generate items in sequence; keeping counts as we go. C.starts is the number of sequences started; c.items is number of items generated."""
c.starts += 1
for item in sequence:
c.items += 1
yield item
-concepts inventory
-refine ideas
-simple implementation
-back envelope of how long it will take to compute
-refine code by swapping clauses
-tools (timing, counts)
-aspects (keeping code clean)
Cryptarithmetic (alphuematics): odd + odd = even
10 digits, all possibilities: 10! = 3 million
table = string.maketrans('ABC','123')
f = 'A + B == C'
eval(f.translate(table)) '1 + 2 == 3'
return not re.search(r'\b0[0-9]',f) \b is a word boundary, need this to find things that start with 0 to omit from evaluating because it will be an octal number
except ZeroDivisionError:
except ArithmeticError:
Any number that starts with a 0 is an octal number, so 012 evaluates to 10 (2 * 1 + 1 * 8 + 0 = 10)
Def solve(formula):
For f in fill_in(formula):
If valid(f):
return f
# Complete the fill_in(formula) function by adding your code to
# the two places marked with ?????.
from __future__ import division
import string, re, itertools
"""Given a formula like 'ODD + ODD == EVEN', fill in digits to solve it.
Input formula is a string; output is a digit-filled-in string or None."""
def fill_in(formula):
"Generate all possible fillings-in of letters in formula with digits."
letters = ''.join(set(re.finall('[A-Z]',formula))) find all the letters in the formula and join them together in a string
for digits in itertools.permutations('1234567890', len(letters)):
table = string.maketrans(letters, ".join(digits))
yield formula.translate(table)
def valid(f):
"""Formula f is valid if and only if it has no
numbers with leading zero, and evals true."""
return not re.search(r'\b0[0-9]', f) and eval(f) is True
Python 2.x:
3/2 1 (integer division, truncated response)
½ 0
Python 3 will do this
from __future__ import division want Python 3 behavior in Python 2
Profiling:
In Command Line:
$ python-m cProfile crypt.py
In Python:
import cProfile
cProfile.run('test()')
Number of calls, total time, time per call, cumulative time want to make faster, look where the time is
Law of Diminishing Returns:
To speed up calls either make them fewer or easier
Divide and conquer and do multiple calls to make easier for the function to evaluate
Eval builds a tree of the string (parse) and then loads in the numbers (codegen) and returns the answer (executes)
lambda: branch of formal mathematics defining functions, anonymous function with no name
compile_formula into lambda function
if word.isupper():
terms = [('%s%s' % (10**I,d))
for (i,d) in enumerate(word[::-1})}
return '(' + '+' .join(terms) + ')'
return word
re.split('([A-Z]+)',formula) splitting up one or more letters in formula
List comprehensions
Generator expressions
Generator functions (yield)
Handling types *polymorphism* (timedcalls(n,) could be int or float) isinstance to determine
eval str object (result of evaluating a string) or function
Timing, time.clock()
Timedcalls routine
Counting (c)
Unit 3 – Personal Notes
Python: statements, functions, format, class – operator overloading, domain specific
Regular Expressions:
Grammar is a description of a language (regular expression)
Language is a set of strings
API: Application Programming Interface
UI: user interface
lit(s) literal('a')
seq(x,y)	seq(lit('a'),lit('b'))
alt(x,y)	alt(lit('a'),lit('b'))
star(x)	star(lit('a'))
oneof(c)	one of('abc')
Can use regular expression from re library (import re)
# The function, matchset, takes a pattern and a text as input
# and returns a set of remainders. For example, if matchset
# were called with the pattern star(lit(a)) and the text
# 'aaab', matchset would return a set with elements
# {'aaab', 'aab', 'ab', 'b'}, since a* can consume one, two
# or all three of the a's in the text.
# Your job is to complete this function by filling in the
# 'dot' and 'oneof' operators to return the correct set of
# remainders.
# dot: matches any character.
# oneof: matches any of the characters in the string it is
# called with. oneof('abc') will match a or b or c.
def matchset(pattern, text):
"Match pattern at start of text; return a set of remainders of text."
op, x, y = components(pattern)
if 'lit' == op:
return set([text[len(x):]]) if text.startswith(x) else null
elif 'seq' == op:
return set(t2 for t1 in matchset(x, text) for t2 in matchset(y, t1))
elif 'alt' == op:
return matchset(x, text) | matchset(y, text)
elif 'dot' == op:
return set([text[1:]]) if text else null
elif 'oneof' == op:
return set([text[1:]]) if any(text.startswith(c) for c in x) else null
elif 'eol' == op:
return set(["]) if text == " else null
elif 'star' == op:
return (set([text]) |
set(t2 for t1 in matchset(x, text)
for t2 in matchset(pattern, t1) if t1 != text))
raise ValueError('unknown pattern: %s' % pattern)
null = frozenset()
def components(pattern):
"Return the op, x, and y arguments; x and y are None if missing."
x = pattern[1] if len(pattern) > 1 else None
y = pattern[2] if len(pattern) > 2 else None
return pattern[0], x, y
assert matchset(('lit', 'abc'), 'abcdef') == set(['def'])
assert matchset(('seq', ('lit', 'hi '),
('lit', 'there ')),
'hi there nice to meet you') == set(['nice to meet you'])
assert matchset(('alt', ('lit', 'dog'),
('lit', 'cat')), 'dog and cat') == set([' and cat'])
assert matchset(('dot',), 'am i missing something?') == set(['m i missing something?'])
assert matchset(('oneof', 'a'), 'aabc123') == set(['abc123'])
assert matchset(('eol',),") == set(["])
assert matchset(('eol',),'not end of line') == frozenset([])
assert matchset(('star', ('lit', 'hey')), 'heyhey!') == set(['!', 'heyhey!', 'hey!'])
return 'tests pass'
# Fill out the API by completing the entries for alt,
# star, plus, and eol.
def lit(string): return ('lit', string)
def seq(x, y): return ('seq', x, y)
def alt(x, y): return ('alt', x, y)
def star(x): return ('star', x)
def plus(x): return seq(x, star(s))
def opt(x): return alt(lit("), x) #opt(x) means that x is optional
def oneof(chars): return ('oneof', tuple(chars))
dot = ('dot',)
eol = #??
assert lit('abc') == ('lit', 'abc')
assert seq(('lit', 'a'),
('lit', 'b')) == ('seq', ('lit', 'a'), ('lit', 'b'))
assert alt(('lit', 'a'),
('lit', 'b')) == ('alt', ('lit', 'a'), ('lit', 'b'))
assert star(('lit', 'a')) == ('star', ('lit', 'a'))
assert plus(('lit', 'c')) == ('seq', ('lit', 'c'),
('star', ('lit', 'c')))
assert opt(('lit', 'x')) == ('alt', ('lit', "), ('lit', 'x'))
assert oneof('abc') == ('oneof', ('a', 'b', 'c'))
# Complete the search and match functions. Match should
# match a pattern only at the start of the text. Search
# should match anywhere in the text.
def search(pattern, text):
"Match pattern anywhere in text; return longest earliest match or None."
for i in range(len(text)):
m = match(pattern, text[i:])
if m is not None:
return m
def match(pattern, text):
"Match pattern against start of text; return longest match found or None."
remainders = matchset(pattern, text)
if remainders:
shortest = min(remainders, key=len)
return text[:len(text)-len(shortest)]
return set([text[1:]]) if text.startswith(x) else null
def plus(x): return seq(x, start(x))
def opt(x): return alt(lit("), x)
eol = ('eol',)
assert match(('star', ('lit', 'a')),'aaabcd') == 'aaa'
assert match(('alt', ('lit', 'b'), ('lit', 'c')), 'ab') == None
assert match(('alt', ('lit', 'b'), ('lit', 'a')), 'ab') == 'a'
assert search(('alt', ('lit', 'b'), ('lit', 'c')), 'ab') == 'b'
Compiler vs Interpreter
Interpreter takes a pattern and text and performs process each time (repeated work each time), will have to look up any embedded things each time
Compiler takes the pattern and compiles it into compiled object, and then executes in second step, doesn't need to look up same information each time (lambda functions)
Compiler Levels: Python Functions, Virtual Machine, Machine Instruction
Changing to Compiler:
# Write the compiler for alt(x, y) in the same way that we
# wrote the compiler for lit(s) and seq(x, y).
def lit(s): return lambda text: set([text[len(s):]]) if text.startswith(s) else null
def seq(x, y): return lambda text: set().union(*map(y, x(text)))
def alt(x, y): return lambda text: x(text) | y(text)
null = frozenset([])
g = alt(lit('a'), lit('b'))
assert g('abc') == set(['bc'])
return 'test passes'
# Fill out the function match(pattern, text), so that
# remainders is properly assigned.
remainders = pattern(text)
def lit(s): return lambda t: set([t[len(s):]]) if t.startswith(s) else null
def seq(x, y): return lambda t: set().union(*map(y, x(t)))
def alt(x, y): return lambda t: x(t) | y(t)
def oneof(chars): return lambda t: set([t[1:]]) if (t and t[0] in chars) else null
dot = lambda t: set([t[1:]]) if t else null
eol = lambda t: set(["]) if t == " else null
def star(x): return lambda t: (set([t]) |
set(t2 for t1 in x(t) if t1 != t
for t2 in star(x)(t1)))
assert match(star(lit('a')), 'aaaaabbbaa') == 'aaaaa'
assert match(lit('hello'), 'hello how are you?') == 'hello'
assert match(lit('x'), 'hello how are you?') == None
assert match(oneof('xyz'), 'x**2 + y**2 = r**2') == 'x'
assert match(oneof('xyz'), ' x is here!') == None
Have accomplished a Recognizer, because recognizing whether text is in pattern (regular expression)
Generator: generates patterns of language based on pattern specified (combinations, factorial, permutation)
COMPILER OPTIMIZATION
# Complete the code for the compiler by completing the constructor
# for the patterns alt(x, y) and oneof(chars).
def lit(s): turquiose only happens once, anything before lambda
set_s = set([s]) putting this here only to do once instead of repeat every time
return lambda Ns: set_s if len(s) in Ns else null highlight happens every time
def alt(x, y): return lambda Ns: x(Ns) | y(Ns)
def star(x): return lambda Ns: opt(plus(x))(Ns)
def plus(x): return lambda Ns: genseq(x, star(x), Ns, startx=1) #Tricky
def oneof(chars): return lambda Ns: set(chars) if 1 in Ns else Null
def seq(x, y): return lambda Ns: genseq(x, y, Ns)
def opt(x): return alt(epsilon, x)
dot = oneof('?') # You could expand the alphabet to more chars.
epsilon = lit(") # The pattern that matches the empty string.
f = lit('hello')
assert f(set([1, 2, 3, 4, 5])) == set(['hello'])
assert f(set([1, 2, 3, 4])) == null
g = alt(lit('hi'), lit('bye'))
assert g(set([1, 2, 3, 4, 5, 6])) == set(['bye', 'hi'])
assert g(set([1, 3, 5])) == set(['bye'])
h = oneof('theseletters')
assert h(set([1, 2, 3])) == set(['t', 'h', 'e', 's', 'l', 'r'])
assert h(set([2, 3, 4])) == null
Patterns (Grammars) which described language
Interpreters and compilers (faster than interpreter) over language
Interpreters can be more expressive and natural to build own language than using python language
Composable: expressions and statements (composed by python programmer) whereas functions can be made dynamically to manipulate/apply functions together
Functions provide control over time (do some now, and some later)
Expressions/statements evaluate right away
MAKING CHANGES TO FUNCTIONS:
Refactoring to improve user-interface might require rewrite of functions. Best to create new function to prevent rewrites of rest of code
# Write a function, n_ary(f), that takes a binary function (a function
# that takes 2 inputs) as input and returns an n_ary function.
def n_ary(f):
"""Given binary function f(x, y), return an n_ary function such
that f(x, y, z) = f(x, f(y,z)), etc. Also allow f(x) = x."""
def n_ary_f(x, *args):
return x if not args else f(x, n_ary_f(*args))
return n_ary_f
## DECORATOR will apply the 'decoration' function to the function specified below
@n_ary
def seq(x,y): return('seq',x,y))
from functools import update_wrapper allows you to give documentation to function when asking for help
update_wrapper(function_updating, function_should_return)
@decorator this decorator will hold the update_wrapper code
blah, blah, blah
def decorator(d):
return lambda fn: update_wrapper(d(fn),fn)
decorator = decorator(decorator)
MEMOIZATION:
Sometimes with a recursive function, you'll have to call and compute the same thing multiple times
Instead, cache results and look them up
def fib(n):
if n is cache:
return cache[n]
cache[n]=result
@decorator
def memo(f):
To show time loss, can count the calls to a function using a decorator
Golden Ratio: (1 + sqrt5)/2
Types of Tools:
Countcalls Debug Tool
Trace Debug Tool
Memo Performance Tool (makes program run better)
n_ary Expression Tool
# Modify the function, trace, so that when it is used
# as a decorator it gives a trace as shown in the previous
# video. You can test your function by applying the decorator
# to the provided fibonnaci function.
# Note: Running this in the browser's IDE will not display
# the indentations.
from functools import update_wrapper
"Make function d a decorator: d wraps a function fn."
def _d(fn):
return update_wrapper(d(fn), fn)
update_wrapper(_d, d)
return _d
def trace(f):
indent = ' '
def _f(*args):
signature = '%s(%s)' % (f.__name__, ', '.join(map(repr, args)))
print '%s–> %s' % (trace.level*indent, signature)
trace.level += 1
result = f(*args)
print '%s<– %s == %s' % ((trace.level-1)*indent,
signature, result)
trace.level -=1
return # your code here
trace.level = 0
return _f
if n == 0 or n == 1:
return fib(n-1) + fib(n-2)
fib(6) #running this in the browser's IDE will not display the indentations!
Another Debug Tool:
def disabled(f): return f turn off decorators
trace = disabled so when @trace it will just use the exact function passed in
Back to Languages:
When writing programs, just "wish" you had function/language you want, then keep going assuming you do, and then write them later
Context-Free Languages
Recognizer yes/no part of language?
Parser yes/no part of language? If yes, gives you tree-structure of input
Tools and how to apply them to components of a domain
Grammars that describe the language, interpreters that interpret them, and compilers that interpret quicker
Functions are better than statements, because you can compose them as objects and don't have to copy and paste statements everywhere and then change them
Also learned about decoratorsUnit 4 Personal Notes
Water Pouring Problem:
Search program (search for shortest path)
Make program to have lists of explored paths and not-explored paths. Keep popping off not-explored and following and checking if goal reached.
Bridget Problem:
1, 2, 5, 10 to get across bridge with flashlight
Lists are NOT hashable because they are mutable
Ways to hold information: tuple, list, set, frozenset
Hashable types: Tuple and frozenset
Mutable (not hashable): list, set
here = {}
there = {}
(here, there, t) where t = elapsed time
Frozen set {1,2,5,10,'light'}
def path_states(path):
return path[0::2] starting at 0 and going to the end, returning every other one since that is the state in the path
def path_actions(path):
return path[1::2]
def elapsed_time(path):
return path[-1][2] the third part of the last element
Avoid mistakes: write lots of tests, use (re-use) tools, generalize function in order to reuse it so you don't have to repeat mistakes and introduce new errors by remaking it again
GENERALIZE:
Shortest_path_search search for the shortest path that reaches a goal
[state, action, state…]
Successors(state){state:action}
Some atomic state
Goal(state)bool
shortest_path_search(successors,start,goal)path
lowest_cost_search(start,successors,goal,cost)path
Learned:
Search problems
Different kinds of searches and algorithms
Shortest_path
Least_cost
Search is subtle, lots of possible problems
Likely to be bugs
Combat with tests
Use standardized tools that hopefully has bugs out of it
Reuse tools that are free of bugs
Two parts
Think about this problem
Think about general problems that you could break out to reuse
Unit 5: Personal Notes about Probability
P(x)
Search problems contain uncertainty: uncertain about current state, and when apply action you are uncertain what action will do and uncertain as to state you will get to
Dice: uncertainty Pig
Pig, 2 players, take turns, roll/hold, object is to score points—can roll each time and keep adding score of rolls as pending score (collect points when you hold), a 1 is a "pig out" and you lose all pending points and get only 1
Play-Pig: fn(A,B) A
Function with two players as input and output is winner of game
Strategy: fn(state)
Function that takes a state and returns an action
State of game (players scores, player's turn, pending score)
(p,me,you,pending)
Actions we can take (roll and hold)
Could be strings of 'roll' and 'hold'
roll(state){state} can have a set of possible results/actions
hold(state){state}
roll(state,d) state easier, because you give a die roll along with it to get single state returned
Die-int
Scores-int
Players / to move: strategy functions
Goal-int
# Write the two action functions, hold and roll. Each should take a
# state as input, apply the appropriate action, and return a new
# state.
# States are represented as a tuple of (p, me, you, pending) where
# p: an int, 0 or 1, indicating which player's turn it is.
# me: an int, the player-to-move's current score
# you: an int, the other player's current score.
# pending: an int, the number of points accumulated on current turn, not yet scored
def hold(state):
"""Apply the hold action to a state to yield a new state:
Reap the 'pending' points and it becomes the other player's turn."""
if state[0] == 1:
score = state[1] + state[3]
return (p, state[2], score, 0)
def roll(state, d):
"""Apply the roll action to a state (and a die roll d) to yield a new state:
If d is 1, get 1 point (losing any accumulated 'pending' points),
and it is the other player's turn. If d > 1, add d to 'pending' points."""
if d == 1:
return hold((state[0],state[1],state[2],1))
return (state[0],state[1],state[2],state[3] + d)
assert hold((1, 10, 20, 7)) == (0, 20, 17, 0)
assert hold((0, 5, 15, 10)) == (1, 15, 15, 0)
assert roll((1, 10, 20, 7), 1) == (0, 20, 11, 0)
assert roll((0, 5, 15, 10), 5) == (0, 5, 15, 15)
(p, me, you, pending) = state can use this to create variables instead of always using state[0], state[1], etc
Use dictionaries to help map variables in code:
other[p]
other = {1:0,0:1}
NAMING A TUPLE (class):
from collections import namedtuple
State = namedtuple('State','p me you pending')
s = State(1,2,3,4)
s.p 1
s.me 2
Interested in learning more about namedtuples? Check out the python documentation.
Strategies:
function(state)['roll','hold']
# Write a strategy function, clueless, that ignores the state and
# chooses at random from the possible moves (it should either
# return 'roll' or 'hold'). Take a look at the random library for
# helpful functions.
possible_moves = ['roll', 'hold']
def clueless(state):
"A strategy that ignores the state and chooses at random from possible moves."
return choice(possible_moves)
hold_at(number), where number is pending to determine hold or when you win by holding
# In this problem, you will complete the code for the hold_at(x)
# function. This function returns a strategy function (note that
# hold_at is NOT the strategy function itself). The returned
# strategy should hold if and only if pending >= x or if the
# player has reached the goal.
def hold_at(x):
"""Return a strategy that holds if and only if
pending >= x or player reaches goal."""
def strategy(state):
(p,me,you,pending) = state
return 'hold' if (pending >= x or (me + pending) >= goal) else 'roll' condescending if/else statement
strategy.__name__ = 'hold_at(%d)' % x
return strategy
goal = 50
assert hold_at(30)((1, 29, 15, 20)) == 'roll'
assert hold_at(30)((1, 29, 15, 21)) == 'hold'
assert hold_at(15)((0, 2, 30, 10)) == 'roll'
assert hold_at(15)((0, 2, 30, 15)) == 'hold'
def play_pig(A,B):
-keep score, pending, take turns, call strategyaction, then do actionstate, roll die
# Write a function, play_pig, that takes two strategy functions as input,
# plays a game of pig between the two strategies, and returns the winning
# strategy. Enter your code at line 41.
# You may want to borrow from the random module to help generate die rolls.
other = {1:0, 0:1}
return random.choice(possible_moves)
(p, me, you, pending) = state
return (other[p], you, me+pending, 0)
return (other[p], you, me+1, 0) # pig out; other player's turn
return (p, me, you, pending+d) # accumulate die roll in pending
def play_pig(A, B):
"""Play a game of pig between two players, represented by their strategies.
Each time through the main loop we ask the current player for one decision,
which must be 'hold' or 'roll', and we update the state accordingly.
When one player's score exceeds the goal, return that player."""
strategies = [A,B]
state = (0,0,0,0)
If me>= goal:
return strategies[p]
elif you >= goal:
return strategies[other[p]]
elif strategies[p](state) == 'hold':
state = hold(state)
state = roll(state, random.randint(1,6))
def always_roll(state):
return 'roll'
def always_hold(state):
return 'hold'
for _ in range(10):
winner = play_pig(always_hold, always_roll)
assert winner.__name__ == 'always_roll'
Way to test: Dependency Injection
Add in element like dierolls to be used in place of random in order to test the function (optional, can pass in a list)
Change the following:
def dierolls():
whie True:
yield random.int(1,6)
def play_pig(A, B,dierolls = dierolls()): can now "inject" dierolls
state = roll(state, next(dierolls))
# Modify the rolls variable in the test() function so that it
# contains the fewest number of valid rolls that will cause
# the hold_at(50) strategy to win. Enter your rolls at line 63
return 'hold' if (pending >= x or me + pending >= goal) else 'roll'
"Generate die rolls."
yield random.randint(1, 6)
def play_pig(A, B, dierolls=dierolls()):
strategies = [A, B]
state = (0, 0, 0, 0)
if me >= goal:
A, B = hold_at(50), clueless
rolls = iter([6,6,6,6,6,6,6,6,6]) # <– Your rolls here
assert play_pig(A, B, rolls) == A
Better Strategy: Way to leap to Best Strategy? Optimal Strategy
Value of a state is its Utility(state) number
End and win, value = 1
End and Lose, value = 0
Quality(state,action)number
What's the quality of the action in this particular state?
Hold and Roll
Values calculated for self, opponents and dice
Can go backwards from end state to calculate utility of each state, and then collect values to get utility of start state and quality of rolling/holding at each state (choose move with highest quality from each state)
Game Theory: Decision under Uncertainty (having an opponent)
Q: quality
U: utility
million = 1000000
def Q(state, action, U):
if action == 'hold':
return U(state + 1*million)
if action == 'gamble':
return U(state + 3*million) *.5 + U(state) * .5
def actions(state): return ['hold', 'gamble']
def identity(x): return x
U = identity
def best_action(state, actions, Q, U):
def EU(action): return Q(state, action, U)
return max (actions(state), key=EU)
Best is to gamble: but this assumes values of money to people is linear
Actually is logarithmic
best_action(100, actions, Q, math.log)
'hold'
best_action(10*million,actions,Q,math.log)
'gamble'
Break Even Point: where outcome is evenly weighted between two
##Optimal Pig
def Q_pig(state, action, Pwin): probability of winning
"The expected value of choosing action in state."
return 1 – Pwin(hold(state))
if action == 'roll':
return (1 – Pwin(roll(state,1))
+ sum(Pwin(roll(state,d)) for d in (2,3,4,5,6))) / 6
raise ValueError
def pig_actions(state):
_,_,_,pending = state
return ['roll','hold'] if pending else ['roll']
def Pwin(state):
if me + pending >= goal:
return max(Q_pig(state,action,Pwin)
for action in pig_actions(state))
OPTIMAL PIG STRATEGY QUIZ
# Write the max_wins function. You can make your life easier by writing
# it in terms of one or more of the functions that we've defined! Go
# to line 88 to enter your code.
"""Decorator that caches the return value for each call to f(args).
Then when called again with same args, we can just look it up."""
cache = {}
return cache[args]
cache[args] = result = f(*args)
# some element of args can't be a dict key
return f(args)
def Q_pig(state, action, Pwin):
return (1 – Pwin(roll(state, 1))
+ sum(Pwin(roll(state, d)) for d in (2,3,4,5,6))) / 6.
"Return the optimal action for a state, given U."
return max(actions(state), key=EU)
"The legal actions from a state."
_, _, _, pending = state
return ['roll', 'hold'] if pending else ['roll']
@memo
"""The utility of a state; here just the probability that an optimal player
whose turn it is to move can win from the current state."""
# Assumes opponent also plays with optimal strategy.
return max(Q_pig(state, action, Pwin)
def max_wins(state):
"The optimal pig strategy chooses an action with the highest win probability."
return best_action(state, pig_actions, Q_pig, Pwin)
assert(max_wins((1, 5, 34, 4))) == "roll"
assert(max_wins((1, 18, 27, 8))) == "roll"
assert(max_wins((0, 23, 8, 8))) == "roll"
assert(max_wins((0, 31, 22, 9))) == "hold"
assert(max_wins((1, 11, 13, 21))) == "roll"
assert(max_wins((1, 7, 26, 34))) == "hold"
assert(max_wins((0, 30, 23, 11))) == "hold"
assert(max_wins((0, 1, 13, 21))) == "roll"
assert(max_wins((0, 11, 0, 24))) == "roll"
This Optimal Strategy is using Pwin as the probability of winning, but maybe utility function might not just be to win the game but also to maximize the differential
Use @memo to only have to do each state computation once
# Write a function, max_diffs, that maximizes the point differential
# of a player. This function will often return the same action as
# max_wins, but sometimes the strategies will differ.
# Enter your code at line 101.
def win_diff(state):
"The utility of a state: here the winning differential (pos or neg)."
if me + pending >= goal or you >= goal:
return (me + pending – you)
return max(Q_pig(state, action, win_diff)
def max_diffs(state):
"""A strategy that maximizes the expected difference between my final score
and my opponent's."""
return best_action(state, pig_actions, Q_pig, win_diff)
# The first three test cases are examples where max_wins and
# max_diffs return the same action.
assert(max_diffs((1, 26, 21, 15))) == "hold"
assert(max_diffs((1, 23, 36, 7))) == "roll"
assert(max_diffs((0, 29, 4, 3))) == "roll"
# The remaining test cases are examples where max_wins and
# max_diffs return different actions.
assert(max_diffs((0, 7, 9, 18))) == "hold"
assert(max_diffs((1, 0, 35, 35))) == "hold"
Made mistake of naming utility and strategy functions too similarly, will edit play_pig function to check both outputs for hold or roll and will disqualify if neither
# Update the play_pig function so that it looks at the result
# that comes from the strategy function and makes sure that
# it is either 'hold' or 'roll' and if it's not one of those
# then that strategy should immediately lose and the other
# strategy should be declared the winner.
elif strategies[p](state) == 'roll':
else: return strategies[other[p]]
def bad_strategy(state):
"A strategy that could never win, unless a player makes an illegal move"
def illegal_strategy(state):
return 'I want to win pig please.'
print play_pig(bad_strategy, illegal_strategy).__name__
winner = play_pig(bad_strategy, illegal_strategy)
assert winner.__name__ == 'bad_strategy'
Simulation vs Enumeration:
P(A|B)
Probability of event A given sample B
P(2 boys|at least 1 boy and 2 children total in family)
.5 B, .5 G
BB, BG, GB, GG all possibilities of 2 children families, GG is excluded and BB is the qualified one so answer is 1/3
Gender of child
Sample space
Families of 2 kids with at least 1 boy
List of Sample Points
BB, BG, GG
Add in day of week that one boy born on Tuesday, will this change probability? Yes: will be 13/27
Set then becomes (gender,day,gender,day): BMBM, BMBT,BMBW,BMBt,BMBF…etc (huge list, more than 4)
Because "at least 1 boy born on Tuesday" has 27 elements, and of those, 13 are "2 boys".
BtG = 7, GBt = 7, BtB = 7, BBt = 6 (would be 7, but then would double-count BtBt), so 27 total and 6+7=13, so 13/27
Probability is powerful
Search with uncertainty
Powerful way of solving general approach
Separates how from what
Don't need to tell it 'how' to do something, just tell it what you want
Simulate and extrapolate
Enumerate if small enough sample (just do all options and count them up)
Wrapper function
Inject into program
Aspect oriented programming
Exploratory data analysis
Used to tell story of what different strategies did
Can pop up in types of functions that they except and return
Python does not protect from errors, must be vigilant
Unit 6: Word Games
2D array (list of lists)
Legal Play
function that generates all the plays possible: tuple (position,direction,'word')
Score – function to compute
dictionary {'Z':10}
Function to compute plays
position on boardDoubleWord/Triple Letter (mapping)
set(words)
Blank Tile
str ' '
scoring will have to know it's blank worth 0
WORDS = set(file('words4k.txt').read().upper().split())
# Write a function, readwordlist, that takes a filename as input and returns
# a set of all the words and a set of all the prefixes in that file, in
# uppercase. For testing, you can assume that you have access to a file
# called 'words4k.txt'
def prefixes(word):
"A list of the initial sequences of a word, not including the complete word."
return [word[:i] for i in range(len(word))]
def readwordlist(filename):
"""Read the words from a file and return a set of the words
and a set of the prefixes."""
file = open(filename) # opens file
text = file.read() # gets file into string
wordset = set(file(filename).read().upper().split())
prefixset = set(p for word in wordset for p in prefixes(word))
return wordset, prefixset
WORDS, PREFIXES = readwordlist('words4k.txt')
assert len(WORDS) == 3892
assert len(PREFIXES) == 6475
assert 'UMIAQS' in WORDS
assert 'MOVING' in WORDS
assert 'UNDERSTANDIN' in PREFIXES
assert 'ZOMB' in PREFIXES
# Write a function, extend_prefix, nested in find_words,
# that checks to see if the prefix is in WORDS and
# adds that to results if it is.
# If not, your function should check to see if the prefix
# is in PREFIXES, and if it is should recursively add letters
# until the prefix is no longer valid.
def removed(letters, remove):
"Return a str of letters, but with each letter in remove removed once."
for L in remove:
letters = letters.replace(L, ", 1)
return letters
file = open(filename)
text = file.read().upper()
wordset = set(word for word in text.splitlines())
def find_words(letters):
results = set()
def extend_prefix(w, letters):
if w in WORDS: #your code here
if w not in PREFIXES: return
for L in letters:
extend_prefix(w+L, removed(letters, L))
extend_prefix(", letters)
return results
# Write a function, add_suffixes, that takes as input a hand, a prefix we
# have already found, and a result set we'd like to add to, and returns
# the result set we have added to. For testing, you can assume that you
# have access to a file called 'words4k.txt'
def find_words(letters, pre=", results=None):
if results is None: results = set()
if pre in WORDS: results.add(pre)
if pre in PREFIXES:
find_words(letters.replace(L, ", 1), pre+L, results)
def word_plays(hand, board_letters):
"Find all word plays from hand that can be made to abut with a letter on board."
# Find prefix + L + suffix; L from board_letters, rest from hand
for pre in find_prefixes(hand, ", set()):
for L in board_letters:
add_suffixes(removed(hand, pre), pre+L, results)
def find_prefixes(hand, pre=", results=None):
"Find all prefixes (of words) that can be made from letters in hand."
results.add(pre)
for L in hand:
find_prefixes(hand.replace(L, ", 1), pre+L, results)
def add_suffixes(hand, pre, results):
"""Return the set of words that can be formed by extending pre with letters in hand."""
If pre in WORDS: result.add(pre)
def timedcall(fn, *args):
"Call function with args; return the time in seconds and result."
result = fn(*args)
return t1-t0, result
hands = { ## Regression test
'ABECEDR': set(['BE', 'CARE', 'BAR', 'BA', 'ACE', 'READ', 'CAR', 'DE', 'BED', 'BEE',
'ERE', 'BAD', 'ERA', 'REC', 'DEAR', 'CAB', 'DEB', 'DEE', 'RED', 'CAD',
'CEE', 'DAB', 'REE', 'RE', 'RACE', 'EAR', 'AB', 'AE', 'AD', 'ED', 'RAD',
'BEAR', 'AR', 'REB', 'ER', 'ARB', 'ARC', 'ARE', 'BRA']),
'AEINRST': set(['SIR', 'NAE', 'TIS', 'TIN', 'ANTSIER', 'TIE', 'SIN', 'TAR', 'TAS',
'RAN', 'SIT', 'SAE', 'RIN', 'TAE', 'RAT', 'RAS', 'TAN', 'RIA', 'RISE',
'ANESTRI', 'RATINES', 'NEAR', 'REI', 'NIT', 'NASTIER', 'SEAT', 'RATE',
'RETAINS', 'STAINER', 'TRAIN', 'STIR', 'EN', 'STAIR', 'ENS', 'RAIN', 'ET',
'STAIN', 'ES', 'ER', 'ANE', 'ANI', 'INS', 'ANT', 'SENT', 'TEA', 'ATE',
'RAISE', 'RES', 'RET', 'ETA', 'NET', 'ARTS', 'SET', 'SER', 'TEN', 'RE',
'NA', 'NE', 'SEA', 'SEN', 'EAST', 'SEI', 'SRI', 'RETSINA', 'EARN', 'SI',
'SAT', 'ITS', 'ERS', 'AIT', 'AIS', 'AIR', 'AIN', 'ERA', 'ERN', 'STEARIN',
'TEAR', 'RETINAS', 'TI', 'EAR', 'EAT', 'TA', 'AE', 'AI', 'IS', 'IT',
'REST', 'AN', 'AS', 'AR', 'AT', 'IN', 'IRE', 'ARS', 'ART', 'ARE']),
'DRAMITC': set(['DIM', 'AIT', 'MID', 'AIR', 'AIM', 'CAM', 'ACT', 'DIT', 'AID', 'MIR',
'TIC', 'AMI', 'RAD', 'TAR', 'DAM', 'RAM', 'TAD', 'RAT', 'RIM', 'TI',
'TAM', 'RID', 'CAD', 'RIA', 'AD', 'AI', 'AM', 'IT', 'AR', 'AT', 'ART',
'CAT', 'ID', 'MAR', 'MA', 'MAT', 'MI', 'CAR', 'MAC', 'ARC', 'MAD', 'TA',
'ARM']),
'ADEINRST': set(['SIR', 'NAE', 'TIS', 'TIN', 'ANTSIER', 'DEAR', 'TIE', 'SIN', 'RAD',
'TAR', 'TAS', 'RAN', 'SIT', 'SAE', 'SAD', 'TAD', 'RE', 'RAT', 'RAS', 'RID',
'RIA', 'ENDS', 'RISE', 'IDEA', 'ANESTRI', 'IRE', 'RATINES', 'SEND',
'NEAR', 'REI', 'DETRAIN', 'DINE', 'ASIDE', 'SEAT', 'RATE', 'STAND',
'DEN', 'TRIED', 'RETAINS', 'RIDE', 'STAINER', 'TRAIN', 'STIR', 'EN',
'END', 'STAIR', 'ED', 'ENS', 'RAIN', 'ET', 'STAIN', 'ES', 'ER', 'AND',
'ANE', 'SAID', 'ANI', 'INS', 'ANT', 'IDEAS', 'NIT', 'TEA', 'ATE', 'RAISE',
'READ', 'RES', 'IDS', 'RET', 'ETA', 'INSTEAD', 'NET', 'RED', 'RIN',
'ARTS', 'SET', 'SER', 'TEN', 'TAE', 'NA', 'TED', 'NE', 'TRADE', 'SEA',
'AIT', 'SEN', 'EAST', 'SEI', 'RAISED', 'SENT', 'ADS', 'SRI', 'NASTIER',
'RETSINA', 'TAN', 'EARN', 'SI', 'SAT', 'ITS', 'DIN', 'ERS', 'DIE', 'DE',
'AIS', 'AIR', 'DATE', 'AIN', 'ERA', 'SIDE', 'DIT', 'AID', 'ERN',
'STEARIN', 'DIS', 'TEAR', 'RETINAS', 'TI', 'EAR', 'EAT', 'TA', 'AE',
'AD', 'AI', 'IS', 'IT', 'REST', 'AN', 'AS', 'AR', 'AT', 'IN', 'ID', 'ARS',
'ART', 'ANTIRED', 'ARE', 'TRAINED', 'RANDIEST', 'STRAINED', 'DETRAINS']),
'ETAOIN': set(['ATE', 'NAE', 'AIT', 'EON', 'TIN', 'OAT', 'TON', 'TIE', 'NET', 'TOE',
'ANT', 'TEN', 'TAE', 'TEA', 'AIN', 'NE', 'ONE', 'TO', 'TI', 'TAN',
'TAO', 'EAT', 'TA', 'EN', 'AE', 'ANE', 'AI', 'INTO', 'IT', 'AN', 'AT',
'IN', 'ET', 'ON', 'OE', 'NO', 'ANI', 'NOTE', 'ETA', 'ION', 'NA', 'NOT',
'NIT']),
'SHRDLU': set(['URD', 'SH', 'UH', 'US']),
'SHROUDT': set(['DO', 'SHORT', 'TOR', 'HO', 'DOR', 'DOS', 'SOUTH', 'HOURS', 'SOD',
'HOUR', 'SORT', 'ODS', 'ROD', 'OUD', 'HUT', 'TO', 'SOU', 'SOT', 'OUR',
'ROT', 'OHS', 'URD', 'HOD', 'SHOT', 'DUO', 'THUS', 'THO', 'UTS', 'HOT',
'TOD', 'DUST', 'DOT', 'OH', 'UT', 'ORT', 'OD', 'ORS', 'US', 'OR',
'SHOUT', 'SH', 'SO', 'UH', 'RHO', 'OUT', 'OS', 'UDO', 'RUT']),
'TOXENSI': set(['TO', 'STONE', 'ONES', 'SIT', 'SIX', 'EON', 'TIS', 'TIN', 'XI', 'TON',
'ONE', 'TIE', 'NET', 'NEXT', 'SIN', 'TOE', 'SOX', 'SET', 'TEN', 'NO',
'NE', 'SEX', 'ION', 'NOSE', 'TI', 'ONS', 'OSE', 'INTO', 'SEI', 'SOT',
'EN', 'NIT', 'NIX', 'IS', 'IT', 'ENS', 'EX', 'IN', 'ET', 'ES', 'ON',
'OES', 'OS', 'OE', 'INS', 'NOTE', 'EXIST', 'SI', 'XIS', 'SO', 'SON',
'OX', 'NOT', 'SEN', 'ITS', 'SENT', 'NOS'])}
def test_words():
assert removed('LETTERS', 'L') == 'ETTERS'
assert removed('LETTERS', 'T') == 'LETERS'
assert removed('LETTERS', 'SET') == 'LTER'
assert removed('LETTERS', 'SETTER') == 'L'
t, results = timedcall(map, find_words, hands)
for ((hand, expected), got) in zip(hands.items(), results):
assert got == expected, "For %r: got %s, expected %s (diff %s)" % (
hand, got, expected, expected ^ got)
print test_words()
# Write the function show that takes a board
# as input and outputs a pretty-printed
# version of it as shown below.
## Handle complete boards
def a_board():
return map(list, ['|||||||||||||||||',
'|J…………I.|',
'|A…..BE.C…D.|',
'|GUY….F.H…L.|',
'|||||||||||||||||'])
def show(board):
"Print the board."
for row in board:
for sq in row:
print sq,
# >>> a_board()
# [['|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|'],
# ['|', 'J', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', 'I', '.', '|'],
# ['|', 'A', '.', '.', '.', '.', '.', 'B', 'E', '.', 'C', '.', '.', '.', 'D', '.', '|'],
# ['|', 'G', 'U', 'Y', '.', '.', '.', '.', 'F', '.', 'H', '.', '.', '.', 'L', '.', '|'],
# ['|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|', '|']]
# >>> show(a_board())
# | | | | | | | | | | | | | | | | |
# | J . . . . . . . . . . . . I . |
# | A . . . . . B E . C . . . D . |
# | G U Y . . . . F . H . . . L . |
# The all_plays function takes a hand and a current board as input.
# Modify the all_plays function to return the resulting set of
# all plays.
def transpose(matrix):
"Transpose e.g. [[1,2,3], [4,5,6]] to [[1, 4], [2, 5], [3, 6]]"
# or [[M[j][i] for j in range(len(M))] for i in range(len(M[0]))]
return map(list, zip(*matrix))
"Return a pair of sets: all the words in a file, and all the prefixes. (Uppercased.)"
class anchor(set):
"An anchor is where a new word can be placed; has a set of allowable letters."
LETTERS = list('ABCDEFGHIJKLMNOPQRSTUVWXYZ')
ANY = anchor(LETTERS) # The anchor that can be any letter
def is_letter(sq):
return isinstance(sq, str) and sq in LETTERS
def is_empty(sq):
"Is this an empty square (no letters, but a valid position on board)."
return sq == '.' or sq == '*' or isinstance(sq, set)
def add_suffixes(hand, pre, start, row, results, anchored=True):
"Add all possible suffixes, and accumulate (start, word) pairs in results."
i = start + len(pre)
if pre in WORDS and anchored and not is_letter(row[i]):
results.add((start, pre))
sq = row[i]
if is_letter(sq):
add_suffixes(hand, pre+sq, start, row, results)
elif is_empty(sq):
possibilities = sq if isinstance(sq, set) else ANY
if L in possibilities:
add_suffixes(hand.replace(L, ", 1), pre+L, start, row, results)
def legal_prefix(i, row):
"""A legal prefix of an anchor at row[i] is either a string of letters
already on the board, or new letters that fit into an empty space.
Return the tuple (prefix_on_board, maxsize) to indicate this.
E.g. legal_prefix(a_row, 9) == ('BE', 2) and for 6, (", 2)."""
while is_letter(row[s-1]): s -= 1
if s < i: ## There is a prefix
return ".join(row[s:i]), i-s
while is_empty(row[s-1]) and not isinstance(row[s-1], set): s -= 1
return (", i-s)
prev_hand, prev_results = ", set() # cache for find_prefixes
## Cache the most recent full hand (don't cache intermediate results)
global prev_hand, prev_results
if hand == prev_hand: return prev_results
if pre == ": prev_hand, prev_results = hand, results
# Now do the computation
if pre in WORDS or pre in PREFIXES: results.add(pre)
def row_plays(hand, row):
"Return a set of legal plays in row. A row play is an (start, 'WORD') pair."
## To each allowable prefix, add all suffixes, keeping words
for (i, sq) in enumerate(row[1:-1], 1):
if isinstance(sq, set):
pre, maxsize = legal_prefix(i, row)
if pre: ## Add to the letters already on the board
start = i – len(pre)
add_suffixes(hand, pre, start, row, results, anchored=False)
else: ## Empty to left: go through the set of all possible prefixes
for pre in find_prefixes(hand):
if len(pre) <= maxsize:
add_suffixes(removed(hand, pre), pre, start, row, results,
anchored=False)
def find_cross_word(board, i, j):
"""Find the vertical word that crosses board[j][i]. Return (j2, w),
where j2 is the starting row, and w is the word"""
sq = board[j][i]
w = sq if is_letter(sq) else '.'
for j2 in range(j, 0, -1):
sq2 = board[j2-1][i]
if is_letter(sq2): w = sq2 + w
else: break
for j3 in range(j+1, len(board)):
sq3 = board[j3][i]
if is_letter(sq3): w = w + sq3
return (j2, w)
def neighbors(board, i, j):
"""Return a list of the contents of the four neighboring squares,
in the order N,S,E,W."""
return [board[j-1][i], board[j+1][i],
board[j][i+1], board[j][i-1]]
def set_anchors(row, j, board):
"""Anchors are empty squares with a neighboring letter. Some are resticted
by cross-words to be only a subset of letters."""
neighborlist = (N,S,E,W) = neighbors(board, i, j)
# Anchors are squares adjacent to a letter. Plus the '*' square.
if sq == '*' or (is_empty(sq) and any(map(is_letter, neighborlist))):
if is_letter(N) or is_letter(S):
# Find letters that fit with the cross (vertical) word
(j2, w) = find_cross_word(board, i, j)
row[i] = anchor(L for L in LETTERS if w.replace('.', L) in WORDS)
else: # Unrestricted empty square — any letter will fit.
row[i] = ANY
def horizontal_plays(hand, board):
"Find all horizontal plays — ((i, j), word) pairs — across all rows."
for (j, row) in enumerate(board[1:-1], 1):
set_anchors(row, j, board)
for (i, word) in row_plays(hand, row):
results.add(((i, j), word))
ACROSS, DOWN = (1, 0), (0, 1) # Directions that words can go
def all_plays(hand, board):
"""All plays in both directions. A play is a (pos, dir, word) tuple,
where pos is an (i, j) pair, and dir is ACROSS or DOWN."""
hplays = horizontal_plays(hand, board) # set of ((i, j), word)
vplays = horizontal_plays(hand, transpose(board)) # set of ((j, i), word)
###Your code here.
# The show function takes a board as input.
# Modify the show function to show the bonus if
# there is no letter in a square.
return sq == '.' or sq == '*' or isinstance(sq, anchor)
possibilities = sq if isinstance(sq, anchor) else ANY
while is_empty(row[s-1]) and not isinstance(row[s-1], anchor): s -= 1
if isinstance(sq, anchor):
def calculate_score(board, pos, direction, hand, word):
"Return the total score for this play."
total, crosstotal, word_mult = 0, 0, 1
starti, startj = pos
di, dj = direction
other_direction = DOWN if direction == ACROSS else ACROSS
for (n, L) in enumerate(word):
i, j = starti + n*di, startj + n*dj
b = BONUS[j][i]
word_mult *= (1 if is_letter(sq) else
3 if b == TW else 2 if b in (DW,'*') else 1)
letter_mult = (1 if is_letter(sq) else
3 if b == TL else 2 if b == DL else 1)
total += POINTS[L] * letter_mult
if isinstance(sq, set) and sq is not ANY and direction is not DOWN:
crosstotal += cross_word_score(board, L, (i, j), other_direction)
return crosstotal + word_mult * total
def cross_word_score(board, L, pos, direction):
"Return the score of a word made in the other direction from the main word."
i, j = pos
(j2, word) = find_cross_word(board, i, j)
return calculate_score(board, (i, j2), DOWN, L, word.replace('.', L))
"Find all horizontal plays — (score, pos, word) pairs — across all rows."
score = calculate_score(board, (i, j), ACROSS, hand, word)
results.add((score, (i, j), word))
"""All plays in both directions. A play is a (score, pos, dir, word) tuple,
where pos is an (i, j) pair, and dir is a (delta-_i, delta_j) pair."""
hplays = horizontal_plays(hand, board)
vplays = horizontal_plays(hand, transpose(board))
return (set((score, (i, j), ACROSS, w) for (score, (i, j), w) in hplays) |
set((score, (i, j), DOWN, w) for (score, (j, i), w) in vplays))
def bonus_template(quadrant):
"Make a board from the upper-left quadrant."
return mirror(map(mirror, quadrant.split()))
def mirror(sequence): return sequence + sequence[-2::-1]
SCRABBLE = bonus_template("""
|||||||||
|3..:…3
|.2…;..
|..2…:.
|:..2…:
|….2…
|.;…;..
|..:…:.
|3..:…*
WWF = bonus_template("""
|…3..;.
|..:..2..
|.:..:…
|3..;…2
|.2…3..
|;…:…
|…:…*
BONUS = WWF
DW, TW, DL, TL = '23:;'
# The best_play function takes a hand and a current board as input.
# Modify the best_play function to return the highest-scoring
# play. If no play exists, return None.
POINTS = dict(A=1, B=3, C=3, D=2, E=1, F=4, G=2, H=4, I=1, J=8, K=5, L=1, M=3, N=1, O=1, P=3, Q=10, R=1, S=1, T=1, U=1, V=4, W=4, X=8, Y=4, Z=10, _=0)
def make_play(play, board):
"Put the word down on the board."
(score, (i, j), (di, dj), word) = play
board[j+ n*dj][i + n*di] = L
return board
NOPLAY = None
def best_play(hand, board):
"Return the highest-scoring play. Or None."
Previous postUdacity Classes 4: Intro to Computer Science
Next postSetting Up My Ender3 Printer with an Octopi Touch Screen & Printing Covid-19 Masks
Neural Network Cognitive Behavioral Therapy Project 3: Inner Demon, Outer Angel, & Inner Angel
Neural Network Cognitive Behavioral Therapy Project 2: My Mental Health Story
Instagram & Neural Network Cognitive Behavioral Therapy Project 1: The Premise
Dome Audio: The Future of Bone Conducting Headphones
How to Use MS Flow to Refresh a PowerBI Dataflow and a PowerBI Dataset upon completion
How to Make a Google Maps API Call
Udacity Classes 9: Intro to Relational Databases Part 1
SAS Programming: Advanced Notes
SAS Programming: Beginner Notes
Udacity Classes 8: Applied Cryptography
Udacity Class
Virtual Frontier
Wellness Journey
I'm just your average video and board game geek living with a loud and goofy persian cat named Madam Crookshanks (Smoosh) that I like to annoy with my tech projects. I'm currently learning about 3D Printing, Digital Photography, Photoshop, Neural Networking, and Ubuntu. I keep up a blog to remind my future self what I've learned, and also use it as a place to share pictures of my cats like any good cat mom would. Email me at: contact@thecatgeek.com
thecatgeek
Words of wisdom as told by cats
😻Madam Crookshanks (smooshy) 😻
👑 Princess Marshmallow 👑
🐱Jezebel 🐱
My babies love me #MadamCrookshanks #persiancat #s
Me and mah bestie #MadamCrookshanks #persiancat #s
She wrapped herself #MadamCrookshanks #persiancat
She made a friend #MadamCrookshanks #persiancat #s
A lone peet amongst the sea of blankets #MadamCroo
Love this little nugget #MadamCrookshanks #persian
Who says cat led yoga isn't a thing #MadamCrooks
Just hanging out #MadamCrookshanks #cats #catsofin
Sleepy gromp #MadamCrookshanks #cats #catsofinstag
Picture of Smoosh by the highly talented @eziekbby