I usually don’t speak about my day job on this blog, mostly because it has nothing whatsoever to do with adventures in Eastern Bloc cuisine. Or rather, it didn’t until recently…
About two years ago, a small team of researchers at IBM (including yours truly) started working on computational creativity. By winning on Jeopardy, IBM has shown that computers can make inferences about the world as it is. But could they also be creative, and produce quality artifacts that have never been seen before? To investigate, we built a cognitive cooking system.
I won’t get into too many details (you can check out our short project page here), but our application allows users to engage in a dialog with the machine, input a key ingredient, cuisine, and dish type, and create billions or more of never-before-seen ingredient combinations. These results are then filtered using predicted surprise and pleasantness factors as well as personal preferences. Once users have picked the recipe they want to make, the system generates ingredient proportions and basic preparation instructions. We’ve partnered with my culinary alma mater, the Institute of Culinary Education to pick their chefs’ brains, have them try some of those computer-generated recipes, and give us some feedback.
This week, we finally started sharing the results with a wider public audience. IBM sent a cognitive cooking food truck to the Pulse Conference in Vegas. Every day, people can vote, via on a dish they’d like to try, and the ICE chefs use the cognitive cooking system to create a recipe that’s served the next day. You can take a look at what’s been served so far on the web site.
Which brings us to the Eastern Bloc connection: on the opening day, the plat du jour was a Baltic apple pie. Since I don’t often have the chance to see an Eastern European recipe created by the collaboration of a computer and three big names like Michael Laiskonis, James Briscione, and Michael Garrett, I wanted to share it here.
Let’s take a look at how the Baltic apple pie was born. The screenshots that follow are taken from our old, “straight from Research” user interface (a more user-friendly app is in the works). On the first day of the conference, the most tweeted dish was pie. We therefore entered pie as the dish we wanted to make. The system first told us what ingredients are frequently used in pies, and what cuisines make pies the most often, inviting us to choose at least one key ingredient and one cuisine. Chef Michael Laiskonis picked apple as the key ingredient, thus reimagining the classic apple pie. Maybe because of his Lithuanian ancestry (check out his takes on Lithuanian desserts: rye bread ice cream with poached apples and caramel, and poppyseed spurgos), he opted for a Baltic recipe (Estonian, Lithuanian, Latvian and Swedish cuisines selected below), and I was very glad he did.
Next, the system looked at its database of existing recipes to figure out what is needed to make a pie. It came back with what I call a “bill of ingredients” — the ingredient categories found in the typical pie — and suggestions as to how many ingredients of each type one is most likely to require. Users are can accept the recommended bill of ingredients, or tweak the specifications. Michael wanted to make a pie with both sweet and savory ingredients. You’ll notice that we have over 80 quintillion possible recipes for our Baltic apple pie — this is 6 trillion times as many results as you’ll get on Google.
Now we can really get into the creative part of the process. The system took our inputs, tried all the combinations of ingredients that satisfied our specifications, excluded the ingredients that aren’t found in Baltic cuisines or don’t pair well with apple, and presented the top 5,000 results. To score these results, we designed three evaluators that predict the novelty of a given ingredient combination, its pleasantness based on psychophysical theories of people’s likes and dislikes, and its flavor pairing count (inspired by the flavor pairing hypothesis). Please note that the result I’m showing in the following image isn’t exactly the recipe that was made at the food truck — there are so many possible combinations that we introduced an element of randomness when displaying the results, and thus I could not reproduce the exact same screenshot.
For each result, the system also explained why it chose this particular combination from amongst the quintillion other possibilities. You can click on the image below to expand it. Dill is used prominently in Lithuanian, Swedish and Latvian cuisines, hence the thick connecting lines. Similarly, apple is often paired with onion, raisins, vanilla, and (like nearly every other food on earth) salt.
After being the Executive Pasty Chef at Le Bernardin for nearly a decade, Michael didn’t need a computer to figure out the ingredient proportions or recipe instructions 🙂 Here are his comments about the recipe:
As a pastry chef, I was motivated to put a traditional ‘dessert’ forms into the system, but with the added challenge of savory and sweet elements, and the cold-weather cuisines of the Baltic and Scandinavia offer some great inspiration. What I think is interesting here is how the balance can easily shift from ‘sweet pie with savory elements’ to ‘savory pie with sweet elements’ simply with the ratio and treatment of the ingredients. The approach here will be to find that elusive middle that highlights the qualities of both elements – to play with the guest’s expectations through presentation.
Finally, the pie was served at the IBM food truck to the 10,000 or so attendees of the 2014 IBM Pulse Conference. In the short video below, James Briscione shows the making of the pie behind the scenes:
Oh, and if you missed us in Vegas (what, you don’t travel the country to attend conferences on cloud computing???), the truck’s going to South by Southwest next week! And so am I!
The recipe below yields about 6 servings.
Pork and apple filling
1 lb pork tenderloin, trimmed and cut into 1/4 inch dice
2 cups water
3 tablespoons smoked salt
1 Granny Smith apple, peeled, cored and diced
2 tablespoons onion, diced
margarine, as needed
light brown sugar, as needed
fine sea salt, as needed
- Brine the pork in the water and smoked salt for 30 minutes, then drain and reserve for assembly.
- Meanwhile, gently sweat the apple and onion in a small amount of margarine until softened, allowing some of the excess moisture from the apples to evaporate. Season lightly with brown sugar and salt; bear in mind that the pork will be slightly salty from brining.
- Allow the apple mixture to cool, and combine thoroughly with the drained diced pork; reserve for assembly.
Blueberry apricot preserves
1/4 cup blueberries
2 tablespoons dried apricot, diced
1/2 cup water
1 tablespoon light brown sugar
1 teaspoon fresh ginger, peeled and finely chopped
Fine sea salt, as needed
- Combine all of the ingredients in a sauce pan and bring to a simmer; cook on low heat for approximately ten minutes, until the fruits have softened. Remove from heat and cool thoroughly. Blend very well, then strain and chill.
Allspice “mayo”
1/2 cup whole milk
1/2 vanilla bean, split and scraped
1/4 teaspoon agar agar
1/4 cup heavy cream (35% fat)
1 egg yolk
1 teaspoon granulated sugar
fine sea salt, to taste
1/4 teaspoon allspice, ground
- Combine the milk, vanilla (seeds and pod), and agar agar in a small saucepan. Gently bring to a boil; reduce heat while maintaining a simmer for two minutes. Remove from heat and whisk in cream, egg yolk, sugar, salt, and allspice. Remove vanilla pod and allow to cool completely at room temperature and set.
- Process the gel in blender until smooth, and chill.
Puff pastry “vol-au-vent”
8 ounces puff pastry dough (frozen), as needed
1 whole egg
2 teaspoons whole milk
pork and apple filling
confectioner’s sugar, as needed
- Temper the frozen puff pastry dough in the refrigerator to allow for cutting. Working quickly, cut 12 pieces of puff pastry discs to a diameter of 3 inches. Cut 6 of the discs a second time — with a 2 1/2 inch cutter — to create a thin even loop, and place back under refrigeration to avoid excess softening or loss of shape.
- Blend the eggs with just enough milk to create a loose egg wash. Brush each of the full discs with a very light coating of egg wash; gently place one of the puff pastry loops on top, ensuring that the two pieces are cleanly aligned. Brush this top loop with a very light coating of egg wash.
- Place about 1/4 cup of the pork and apple filling into the center of each “vol-au-vent” and arrange on parchment or pans, allowing ample space in between. Bake at 350 F for ten minutes; reduce heat to 300 F and continue to bake until uniformly browned and pork filling has thoroughly cooked. Remove from heat and dust very lightly with confectioner’s sugar; return to oven briefly for 30 seconds to “melt” the sugar, and cool.
Garlic chips
4-5 cloves of garlic, cloves separated and peeled
whole milk, as needed
vegetable oil, as needed
- Slice the garlic very thinly, and place in a sauce pan with milk, just to cover. Bring to a boil, drain the milk, and repeat this blanching process two more times. Gently dry the sliced garlic on paper towels.
- Heat the oil to 350 F and lightly fry the garlic chips until crisp.
Granny Smith apple, as needed
chives, as needed
allspice “mayo”
blueberry apricot preserves
puff pastry “vol-au-vent”
parsley leaves, as needed
garlic chips
- Slice the peeled and cored apple into a very fine julienne, approximately 1 inch in length; cut the chives into “sticks” of similar size.
- Place a few small dollops of the allspice “mayo” and blueberry preserves onto each individual “vol-au-vent”, and garnish with the apple julienne, chive sticks, parsley leaves, and garlic chips.
The recipes and pictures of the pie are courtesy of Michael Laiskonis.
The posts on this site are my own, and do not necessarily represent IBM’s positions, strategies, or opinions.
11 comments
I don’t believe in computers cooking with margarine…
Haha, I must say that usually, the first thing we do when the computer returns the results is filter out margarine 🙂 In this recipe, however, the small amount of margarine used to cook the apple and onion won’t be an issue. If you insist, replace it with butter.
Florian, I have never recovered from the cultural shock when I had caviar served with margarine at the Mariinsky Theatre which is otherwise pretty civilized…
Just to let you know, I utterly despise computers and margerine… and I don’t think that computers should interfere with cooking… in any case, they should provide information on how bad margarine is for you … otherwise they’re not much of a computer… not to mention the foul taste… if you want interesting recipes, it’s better to have tea with a loving granny…
I will be the first to admit that I know next to nothing about the field of computational creativity, but the way this system seems to work does not correspond to anything I think of as creativity. If I understand correctly, all of the inputs are human-generated, based on theories, habits, and identifications already well-worn by humans. “The system took our inputs, tried all the combinations of ingredients that satisfied our specifications, excluded the ingredients that aren’t found in Baltic cuisines or don’t pair well with apple, and presented the top 5,000 results.” The only parts of this description I would consider “creative” are in devising & deciding upon the inputs — the rest sounds like very standard data-crunching, just following human instructions. The initial input (the decision to try a pie) was by the human team. The system acquired its knowledge of Baltic cuisines via, I’m guessing, humans who told it where to find information about Baltic cuisines. It does not possess any knowledge of Baltic cuisines that hasn’t yet been put into a form accessible to computers — for example, a memory stored in someone’s Baltic grannie. The system’s ideas about things that do or do not “pair well with apple” is based on human habit & study. The decision as to which of the many thousands of results to proceed with was made by the human team. Unless I am missing something, the system is not capable of surprising us.
Why margarine, is that a choice the system made or the chef? Likewise, who decided that the allspice sauce is “mayo” when it sounds very much like creme anglaise?
Chefs, professional or amateur, acquire their knowledge via humans and memorize it in a form they can access later. Their ideas about things that do or do not pair well together are based on human habit and study. For example, they have people try their dishes and draw their conclusions. Or they write down combinations that they tasted and enjoyed. This is how cognition works, and this is the basis that any human or machine needs to be creative. Would you say that your favorite chefs are not creative just because they were trained by other humans and you had to order from a menu before they cooked for you?
What makes our system creative is that 1) it produces novel recipes that have never been seen before (have you seen a recipe similar to the Baltic apple pie?), and 2) the recipes taste good and produce complex ingredient pairings that are very hard for humans to consider. You can buy the Flavor Bible and get a (not very exhaustive) list of ingredient pairs that work together. Professional chefs can produce dishes with interesting ingredient triplets. Computers will try combinations of 4, 5, 6 or more ingredients, and evaluate them.
The margarine was selected by the system, because its teachers (people who post recipes on the internet) use a lot of margarine. Calling the allspice sauce a ‘mayo’ was Michael Laiskonis’ choice. It is indeed not a typical mayonnaise, hence the quotes. However, the agar creates a texture that’s similar to a mayo.
Thanks for the thoughtful response, Florian. It is helping me understand this system’s similarity to human creativity, but I still don’t agree that it is itself creative. And I would not say that the creativity of my favorite chefs is captured in the inputs and processes you’ve described, no — I strongly believe creativity is in the spaces between them, and the mental facility that navigates them and makes choices about them. That is an important difference between the system and the human chef example — the human chef is not limited to spitting out recipes strictly from food-related inputs; she might devise a recipe that is influenced by a combination of recipes she’s encountered in the past, other cooks she has worked with, and something non-edible she saw in the street on the way to her restaurant. Or something else she saw or experienced. She herself may not be able to explain where the idea for the dish came from. The system, as far as I can tell, will always have an explanation. Also, novelty is part of being creative, but on its own it’s not enough. I don’t think novelty for its own sake tends to be meaningful. To be able to decide whether a particular novelty is interesting / worthwhile / valuable / pleasing / useful or not is every bit as important, if not more so, than coming up with it in the first place. Novelty is also problematic because it’s so closely related to time and history — something that is known by a small or distant group can be novel to a larger or more social group. The system can decide the value of a particular novelty to some extent using ideas about flavor pairing, but I wouldn’t call that creativity — I would call it analysis & data-crunching being used to produce novelties. I have not seen a recipe before for any sort of Baltic pie quite like this one, no, but I think the most creative aspects of it are in its presentation, which was decided upon by the chef. For example, presenting a stabilized creme anglaise as “mayo.” The system does not seem to have a wit input that would allow it to have suggested that on its own.
I agree with you that our system still has a limited set of inputs. I think we’re only beginning our journey. With more time, we’d love to add information about the visual appearance of dishes, combinations of dish elements (e.g. pie crust + pie filling + “mayo” + chips), or personal memories and experiences shared via social media. However, maybe even more than a technical challenge, there’s a practical issue with this. If I write down interesting ideas while reading cookbooks, browsing the web, or traveling, I’m going through the normal process of looking for inspiration in my creative work. If a computer stores information it found in cookbooks (pictures or text), it’s called copyright violation. If a computer browses a web site, in most cases it’s a violation of its terms of use. If a computer records memories on the street or scans personal pages on social media, it can be a violation of privacy. We have to secure the rights for every data source we want to use.
This doesn’t mean that the system doesn’t demonstrate creativity in the field it’s operating on right now. It doesn’t just look for novelty, it does decide if something is valuable and pleasing – that’s the pleasantness score I talk about in my post. We could come up with more evaluators. Grant me access to your Twitter and Pinterest accounts, or your grocery receipts, and the machine will predict if a particular recipe is a good fit for you specifically.
We’re also not trying to create a machine that replaces humans. We want the system to help people be more creative in what they do. There’s a dialog between the user and the computer: the system helps users define what they want to create, makes suggestions, asks for further inputs to narrow down the options, and recommends what it thinks are the best results, which can be filtered again.
As for novelty being related to time and history, this is true for both humans and computers. Something that was creative 10 or 100 years ago typically isn’t creative today anymore, given the changes in the state of the art.
Great post. Great website. I am inspired! Thanks 🙂
[…] I introduced the Cognitive Cooking technology, I explained how computers could be creative, and create novel and tasty recipes. It’s worth […]
As Florian says “the article is showcasing what the technology can do today”. A quick Google from before this article was written returned zero results for a “Baltic Apple Pie”. Now there is a new recipe on the menu. The resulting recipe sounds very nice indeed
[…] might remember that apart from this blog, my day job as a software engineer has been all about Chef Watson — the app that helps you discover never-seen-before recipe ideas — and I have a couple of […]