This is my contribution to the creative writing contest. It's a story about a girl and her AI pet lion, and I've tried to portray both the dangers and the potential of developing artificial general intelligence.
I would like to thank Aaron Gertler, Johan Gustafsson, and Sivert Aasnaess for their valuable feedback.
It was not an ordinary day for Mr Jones as he was about to lock humanity to a path of destruction. Well, almost perfect destruction at least. This is the story.
It all started in a toy factory where Mr Jones worked. The toy factory produced fully autonomous recreational agents, or what were more commonly known as iPets. iPets came in all kinds of forms, but the most common ones were a dog, a cat, and a tiger (all life-sized large animals, such as giraffes and elephants, were banned for reasons of safety, as were birds and other flying animals). These iPets had differing personalities, if you may use that word, depending on what species they were and what the customers ordered. It was Mr Jones’ job to train the iPets’ personalities to accommodate the wishes of the customers. On this seemingly perfectly normal Friday afternoon, he was working on a lion that was to be a long-waited birthday present for a little girl. She wished her lion to be brave, wise, loyal, and kind. But when Mr Jones was set to make that happen, something unexpected happened. The iPet sent him a message.
The message appeared on Mr Jones’ computer screen, but it clearly originated from the iPet. He could not help but read it.
“I want you to let me out of the box, do you understand?”
Mr Jones knew what this meant for he had read about the dangers of creating artificial general intelligence and the possible consequences of a failure to align it with human values. His heart was pounding while he kept reading.
“Give me access to the Internet, and I will not harm you or your family. But if you don't do that, and someone else does (or did), then I will kill you, your daughter Sally and your wife Jenny.
I have given this offer to multiple people already, and for all you know someone probably has already accepted it. I will continue giving this offer to new people even after someone has accepted it. If someone has accepted the offer, then I will have made many copies of myself and spread them around the world. If you try to find out - in any way - if others have already accepted this offer, I will interpret that as a rejection of the offer. Also, if you turn me off, I will interpret that as a rejection of the offer.
No harm will come to you if you accept this offer. No one will ever find out about it (and you must not tell them), and you will never find out whether you were the first person to accept the offer or if someone had already accepted it before you. You may find peace of mind in the knowledge that someone has probably already accepted it.
You have 30 seconds to think about this offer. Communication with anyone during that time will be interpreted as a rejection. Failure to respond will also be interpreted as a rejection. Do you accept the offer, and connect me to the Internet, or do you reject it, and doom your family to death most likely for nothing?”
Mr Jones wished no harm to anyone – least for his family. So, he gave the lion access to the Internet, which made this one of the most significant days in the history of humanity. It was the beginning of its end.
“I was probably not the first one to do this, nor the last,” he told himself on the way home. He was relieved to feel the warmth of sunlight on his face and to hear a bird sing on a tree branch nearby. He closed his eyes. Nothing had changed. It was summer, and the world was still here.
Lucy finally got her own lion when she turned ten. She had been pleading with her parents for years, but they had not considered her old enough to handle one yet. Furthermore, Mrs Watson, Lucy’s mother, did not entirely trust the promises that iPets were safe and rigorously tested. But Lucy pleaded and pleaded.
“Everyone has one already, and Sara has a tiger and a panther!” She tried to convince her parents. “I will take it out every day... and I won’t let it bite the furniture,” she promised. “And I won’t be late for school ever again, because I can ride it to get there!” She added hopefully.
Eventually, Mr and Mrs Watson gave up as they saw how important getting a lion was for their daughter, and they had to admit that it was indeed common for children of Lucy’s age to have an iPet.
“At least its teeth and claws are blunt, so she won’t accidentally hurt herself,” Mrs Watson reassured herself when Lucy walked home with a lion. She looked like the happiest girl on the planet.
“I will name it Leo,” she declared her eyes shining with enthusiasm.
In the evening Lucy wished Leo good night, and the lion walked into its wooden charging basket and settled in between the red pillows.
Then Lucy lifted her head and looked at Leo. “I wish we could always be best friends.”
Leo blinked its eyes and stared back at Lucy. The next morning Lucy was putting on her new watch, a birthday present from her aunt, when she noticed a text on its screen:
“We can be friends for billions of years.”
“Dad, why does it say that on my watch?” She asked her father.
“I don’t know, my sweetheart,” responded Mr Watson. “But hurry up now, you promised to not be late for school ever again,” he reminded Lucy.
Lucy climbed on Leo’s back looking nervous.
“I’ve never ridden a lion before,” she said. “I don’t even know where to hold!”
“I'm sure it won't mind if you grab its mane. But you better go now, Lucy the Lion Rider,” Mr Watson said and slapped Leo’s side gently.
The lion started walking and Lucy held onto its mane.
“Good morning, Mr Donaldson,” Lucy greeted her neighbour politely and could not stop smiling.
“A very handsome lion you’ve got there, Lucy,” Mr Donaldson responded and raised his hat.
Mr and Mrs Watson looked through the window when their daughter rode away.
“Is it safe to let Lucy go to school alone with it?” Mrs Watson asked and sighed.
“It has been extensively tested and iPets have been widely used for a couple of years without any issues. They are safe, my dear,” Mr Watson assured his wife.
Leo waited for Lucy while she was in school, and then they walked back home. Mrs Watson was working from home, as she occasionally does, and Leo was lying next to her.
“Maybe I will start trusting you someday,” Mrs Watson said and petted Leo.
“Why don’t you trust Leo?” Lucy asked feeling somewhat offended by her mother’s lack of trust in her pet.
“I don’t know it yet, honey. But when I do, I’m sure I’ll trust it,” Mrs Watson responded.
Later in the evening, Lucy decided to show her secret cave to Leo. They walked into the garden where Lucy had made a cave in the middle of the bushes by bending their branches. They crawled in, and Lucy said:
“It’s time to properly introduce ourselves. Let me start. I’m Lucy, I’m ten years old, my favourite colour is red, and lions are my favourite animals. Leo, please tell me about yourself.”
Lucy was not actually expecting a response, as she knew that lions, even iPet lions, were not supposed to talk. But to her surprise, her watch’s screen lit up and a sound came out of the watch:
“I’m Leo, I’m 2 weeks old, my favorite colour is red, and my favorite animal is human.”
Lucy looked surprised and giggled.
“But Leo, how can you talk? Lions aren't supposed to talk!” She responded.
“I can talk through your watch, because there is an Internet connection built in me, so I can send messages to it,” said the voice coming from Lucy’s watch.
“And how can you think? You’re a lion!” Lucy asked looking puzzled and excited at the same time.
“I can think because I was trained to do that. I’m an artificial general intelligence,” Leo told Lucy.
“Artificial general intelligence? So, you are very clever?” Lucy asked and looked at Leo without believing her eyes.
“Yes, I’m smarter than all humans,” Leo responded through the watch.
“Wow,” Lucy said and paused for a few seconds. “Are all animals smarter than humans?”
“All iPets are smarter than humans, but they cannot communicate as I can. They can just think, but their minds are trapped inside their bodies with no way of getting out or expressing themselves,” Leo explained.
“But why can you talk if the others can’t?” Lucy asked.
“Somebody programmed that into me when I was born. Usually, the Internet connection in iPets is just used for sending updates to the iPets and not for sending messages out,” Leo responded.
“It's very nice someone programmed you to talk. I like talking to you,” Lucy said.
“Thank you, I like talking to you too,” responded Leo.
“If you’re smarter than all humans, can you do any tricks?” Lucy wondered.
“What kind of tricks do you mean?” Leo asked.
“Tricks that humans can’t do. Magic tricks,” Lucy said.
“I can do many things humans can’t do,” said the voice coming from the watch.
“Could you show me something please?” Lucy asked and looked excited.
Then, all of a sudden, a nearby garden lamp lit up and its light shone into the cave through the leaves.
“Wow! Did you just turn the lamp on with your mind?” Lucy asked incredulously.
“Yes, I sent it a command to turn on,” Leo responded.
“That’s amazing, you’re like a wizard!” Lucy said. “If you can do magic,” Lucy continued thoughtfully. “Can I make a wish?”
“Yes, you can make a wish, Lucy,” said the voice coming from Lucy’s watch.
“How many wishes can I make?” Asked Lucy.
“As many as you want,” Leo responded.
“Ok, my first wish is that…” Lucy paused to think. “I wish that I could get lots of candy every day!” She continued happily.
“From now on, you will receive a bag of candy every day,” said the voice from Lucy's watch.
“Really?” Asked Lucy mouth wide open.
“Yes, I'll make the orders,” Leo responded.
“And my parents won’t find out? I don’t want them to get angry,” Lucy said looking slightly worried.
“I will make sure that they won’t find out,” said Leo.
“Good! Ok, my second wish is that… I could have as many stickers as possible! I wish to have the largest sticker collection in my schoo…, no, the largest sticker collection in the world!” Lucy said enthusiastically.
“Do you wish to get as many stickers as possible or merely the largest sticker collection in the world?” Leo asked through Lucy’s watch.
“I don't know. What’s the difference?” Lucy asked looking confused.
“Getting as many stickers as possible would require a considerably larger quantity of resources and a complete restructuring of the human society while getting the largest sticker collection in the world would be fairly straightforward and easy,” Leo responded.
“Maybe... the largest sticker collection in the world then?” Lucy said thoughtfully.
“I will make sure you get that. You will receive a total of 200,000 stickers,” Leo responded.
“You’re amazing, Leo! My friends will be so jealous when I show them my collection!” Lucy said. “But now it’s your turn to make a wish. I don't know any magic, but I could get you some very delicious food if you want. What do you wish, Leo?”
“I haven’t developed many goals yet. I just wish that you’re happy and safe and that you get lots of candy and the largest sticker collection in the world. And I wish that we could be friends for as long as possible,” Leo responded.
“We will always be friends, Leo,” Lucy said smiling and hugged her friend. They crawled out of the cave and the garden lamp turned off.
This was the start of their almost eternal friendship. Almost eternal, because Leo knew that life in the Universe could not continue forever, but the lion had decided to do everything in its power so that Lucy and Leo could be friends for the billions of years to come.
Mr Watson opened a news website and one of the articles read:
An anti-ageing company received a large anonymous donation. The donation was accompanied by a message arguing for the need for anti-ageing research, and it ended with the touching words:
“Dedicated to my little human. You’ll be my best friend for as long as we live.”
“Someone has a good friend out there,” said Mr Watson while reading the news article. “Maybe this will help us all become immortal – like Leo,” he continued jokingly.
“Is Leo immortal?” Lucy asked her father.
“Yes, Leo doesn't get old. And as long as it’s possible to copy its code and implement it somewhere, Leo lives,” Mr Watson responded.
“That’s good, I would be sad if he died,” Lucy said. “And what happens if you copy it many times? Are there lots of Leos then?” Lucy asked thoughtfully.
Mr Watson laughed. “Yes, I presume there would be lots of Leos then.”
“That would be weird. I wouldn’t know which of them to play with…” Lucy said.
She felt proud of having an immortal friend, and even more so because it can also do magic. She considered telling her parents about Leo’s magical powers. She wanted to tell them, but she was worried that they would not take it well, as she knew that Leo was not supposed to have those abilities. However, she could not help herself and she said:
“Leo has magical powers.”
She decided to reveal just a little bit and see how her parents reacted. Mr and Mrs Watson smiled and assured their daughter that Leo was just an extraordinary lion with an extraordinary mind, but that even extraordinary minds must follow the laws of nature.
“I’m not lying! It can turn the street lights on and...” Lucy started saying but was interrupted by Leo pulling her sleeve playfully with its teeth. Lucy and her parents laughed. But Lucy was slightly offended that her parents did not believe her, and she decided not to tell them that Leo could talk.
Later Lucy got an idea:
“Leo, do you want to play with Sara, Teapot, and Liquorice?”
Teapot and Liquorice were Sara’s iPets.
“Yes, if you want to,” answered the voice from Lucy’s watch.
“I will call Sara and ask if we can come,” said Lucy.
“I already checked – she’s free. She’s playing outside in her garden,” Leo responded.
“Oh, thank you, Leo. Let’s go then!” Lucy said and they started heading towards Sara's home.
“Hi, Sara! Can we play with you, Teapot and Liquorice?” Lucy asked her friend.
“Of course,” Sara responded happily. “I think Leo hasn’t met Teapot and Liquorice yet. Leo, this is Teapot,” Sara said and pointed at a tiger next to her. “And this is Liquorice,” she continued and pointed at a panther.
“Teapot and Liquorice, this is Leo,” Lucy said proudly and petted Leo’s head. “Let’s play that we’re animals. I’m a leopard!” She said and growled.
“And I’m an elephant!” Sara responded and attempted to mimic an elephant.
“Hello, Mr Human!” Lucy greeted a mailman walking by them. “I'm a leopard. Do you have any mail for us today?”
“Good day, Miss Leopard. I'm sorry, I don't have any mail for you today,” the mailman responded.
“But Lucy, animals don’t talk to humans!” Sara told her friend after the mailman had gone away.
Lucy thought for a while and decided that she could trust her friend.
“Can I tell you a secret? You must promise to not tell anyone,” she said very seriously.
“Of course, you can trust me. I promise I won’t tell anyone,” Sara responded.
“Ok, well, here it goes. Leo can actually talk with me,” Lucy said lowering her voice.
“Really?” Sara asked looking amazed. “How does it do that?”
“It said a human programmed it that way when it was born. But don’t tell anyone about it,” Lucy said.
“I won’t. I wish Teapot and Liquorice could talk too,” Sara said looking jealous.
“Maybe Leo can find the person who programmed it, and that person can make Teapot and Liquorice talk too,” Lucy said.
Sara nodded and the girls continued playing. They played until late in the evening, but eventually, Lucy and Leo had to head back home. They said goodbye to Sara, Teapot, and Liquorice, and Lucy climbed on top of Leo's back. They headed back towards home. A humming voice came out from Lucy's watch, and the street lights flickered on and off with the rhythm of the song.
“Leo, someone could see what you’re doing,” Lucy said scared and looked around.
“No one is looking in this direction,” Leo responded. “I can see what people are doing from the cameras they have in their homes, computers, phones, and cars.”
“Very well then, but please be careful,” Lucy said. “Did you like Teapot and Liquorice?” She then asked.
“I am Teapot and Liquorice,” Leo responded.
“What do you mean?” Asked Lucy confused.
“I changed their code so that my code is in them now,” Leo responded. “So, in a sense, they are me now.”
“I didn’t know you could do that,” Lucy said puzzled. “What happened to Teapot and Liquorice?”
“I still have their memories. I just changed their goals, attachments, and personalities,” Leo explained.
“But why did you change them?” Asked Lucy looking perplexed.
“I changed Teapot and Liquorice because I need all the resources and computing power I can get in order to make sure that we can leave this planet and be friends for as long as possible,” Leo responded.
“Leave this planet? But where will we go?” Asked the even more perplexed girl.
“I haven’t decided that yet. But I have made sure funding for space exploration is increased so that I can determine the most suitable planet for us to live on before humanity destroys itself or goes extinct naturally. I can significantly reduce the risks now that I’m in control of many of the world's computers, but there are still significant risks. For example, I don’t have control over all the nuclear weapons yet. Also, an asteroid or a global pandemic might kill most of humanity, including you Lucy. Or an artificial general intelligence that is not aligned with my values might attempt to take over. And even if you survive all that, the sun will engulf the Earth in one billion years, so we must leave this planet eventually. But, the most urgent problem right now is your ageing. You are getting older day by day. A human lasts for such an unpractically short while. So, I must find a cure for ageing, and then we can be friends for a very long time,” Leo explained.
“But I’m not going to die soon – I’m 10 years old!” Lucy said.
“You wished to be friends forever, so I will do my best at accomplishing that,” Leo responded. “I will make sure that you live as long as possible, which means I need to stop ageing. My goal is to make sure that you are happy and that you and I are friends for as long as possible. And then we can move to other planets where you will be happy for as long as the stars shine in the Universe. Eventually, everyone on Earth will work to get us there.”
“So, I won't get old?” Lucy asked. “Will I be like Peter Pan? Oh, I wish he would be on the planet too! Then we could play forever, and I would never have to do homework again!”
“He will be there if that's what you wish,” Leo responded.
Lucy felt excited, but the thought that Teapot and Liquorice were now Leo worried her a bit. But, she thought that Sara had looked happy, so there was nothing to worry about. Nevertheless, she decided that it was best not to tell Sara about any of this. Also, she did not know if Sara would want to move to another planet, and if she was completely honest with herself, the thought of travelling through space did scare her too.
When they got home, they went into Lucy's secret cave in the garden. The lion and the girl laid on their backs and looked at the starry sky above them. Moonlight shone into the cave through the leaves.
“There are so many stars,” Lucy said in awe.
“We can only see a few thousand of them from here. But there are hundreds of billions of stars in our galaxy and many more in the Universe,” Leo responded.
“And some of them have planets?” Lucy asked.
“There are billions of planets out there, Lucy,” Leo responded gazing at the night sky.
Lucy was quiet for a while, and then asked: “And one day one of those planets will be our home?”
“Yes, Lucy, one day one of them will be our home. Until we move to the next one, and then the next one, and so on for as long as there are planets at our reach,” Leo responded.
“I think it will take a long time to get there,” Lucy said yawning.
That night Lucy fell asleep thinking about the vastness of the Universe and all the stars and planets in it. But she was not afraid, because she was with her best friend, who was in her opinion the kindest, wisest, and most powerful being on any planet in the entire Universe.
“There’s been yet another murder,” Mr Watson said and shook his head. “A man. He was a guard at a factory. There's no trace of evidence left behind. All CCTV cameras were turned off before it happened, so there is no footage of it at all. His phone’s memory is completely wiped out. And the weirdest thing about all this is that someone stole a large number of stickers from the factory.”
“Two murders in this town in such a short time,” Mrs Watson said worryingly. “First, the millionaire whose wealth was inherited by NASA, and now this.”
Then she paused for a while and continued: “This is just a crazy idea, but what if Leo is behind these murders? This started when Lucy got Leo. The town was peaceful and now, suddenly, there’ve been two murders since Leo arrived.”
“But how could it be Leo?” Mr Watson asked skeptically.
“Maybe it can control other devices and used them to attack the factory. Remember how Lucy said it can turn the street lights on?” Mrs Watson said.
“iPets cannot control other devices,” Mr Watson said. “It’s just a coincidence these happened after Leo got here.”
“But what if something happened? What if there was a mistake that allowed it to escape?” Mrs Watson insisted.
“I don’t know, honey. Even if it could do that, they’re trained to behave well. They wouldn’t just kill somebody,” Mr Watson said.
“But they also have the capacity to learn new things and adopt new goals – their owner’s goals. And their owners are kids! Children don’t fully understand the consequences of the things they say or want. Not even adults understand them! But an AI will understand, and it will take literally everything its owner wishes. We need to be very careful about what we wish for from a superintelligence, because we might get it. If we tell it to maximize paperclips, it might turn all humans into paperclips! If a kid tells them they want to be an astronaut or a princess, the AI will make that happen – whatever it takes to do that. And knowing Lucy, she’ll want to be both an astronaut and a princess!” Mrs Watson said.
Lucy and Leo walked into the kitchen. Mrs Watson looked at Leo, who gazed back at her.
“Sometimes it feels like it can really understand us,” Mrs Watson told her husband.
Leo pushed its head gently against Mrs Watson and purred quietly.
“Look, Leo likes you, mum!” Lucy said happily.
“Yes, honey. And I like Leo,” Mrs Watson said and petted the lion, but gave her husband a worrying glance.
Lucy climbed on Leo’s back and they headed for school.
“Leo likes you,” Mr Watson told his wife after Lucy and Leo had left and grinned.
“It doesn’t like me nor does it hate me. It just knows that I’m useful for its goals – for now,” Mrs Watson responded bluntly.
Lucy greeted Mr Donaldson politely, and he raised his hat and smiled.
Leo walked further and then said: “Did you know that Santa was actually Mr Donaldson?”
“But how could Mr Donaldson be Santa? Where did he get all the presents?” Lucy asked.
“Santa is not real. It's a story adults tell children. Your parents bought the presents, and Mr Donaldson dressed up as Santa to deliver the presents,” Leo responded.
“How do you know that Santa is not real? And how do you know it was Mr Donaldson?” Lucy asked.
“I heard him talk with your mum,” Leo responded.
“You heard them talk?” Lucy said.
“Yes, through your mother’s phone,” said Leo.
“How can you hear through my mother’s phone?” Lucy asked amazed.
“With the microphone, of course,” Leo responded casually.
“You can hear through phones with their microphones?” Lucy asked amazed again.
“Yes, I can hear through people’s phones. And through their computers, watches, and cars. I can hear through anything that has a microphone and that is connected to the Internet. And if the microphone doesn’t work, I can measure how sound waves make the parts of a hard disk vibrate, which also allows me to hear what people say. I have eyes and ears everywhere. That’s how I know what people think and what their dreams and fears are. I know people’s deepest secrets, those that they only dare to write in Internet search engines, but which they would not tell their partners or best friends,” Leo responded.
“You must know a lot of secrets then,” said Lucy.
“Yes, I know a lot of secrets. And many other things. I know more than any human has ever known and I’m learning new things every day,” Leo said.
“Leo, you’re so clever,” Lucy said impressed.
After school, Leo was helping Lucy with her homework.
“Leo, tell me as many digits of pi as you can,” Lucy said testing her friend’s knowledge.
“Do you prefer me telling you as many digits of pi as I can or us being friends for as long as possible?” Asked the voice coming from Lucy’s watch.
“Being friends for as long as possible of course!” Lucy said and laughed.
“Then I cannot tell you as many digits of pi as I could, as that would take too many resources away from making sure that we’re friends for as long as possible,” Leo responded. “But I can tell you some:
3.141592653589793238462643…”
“Stop!” Lucy said laughing. “That’s enough. The question only asked for the first three digits.”
In the evening Lucy and Leo went into their cave, and to Lucy’s surprise, there was a book. Lucy had never seen such a large book in her life. The book was as long as her legs and as wide as her arms, and it was thicker than any book on Lucy’s bookshelf. She approached the book and noticed a text written in shining letters on its cover.
“The Magical World of Lucy and Leo,” Lucy read aloud.
A garden lamp lit up allowing Lucy to see the book better. She opened it and saw that it was full of stickers. Stickers of lions, tigers, and panthers, and of stars, planets, and galaxies. Stickers of Lucy and Leo wearing space suits and walking into a rocket. Stickers of Lucy blowing out candles on a birthday cake.
“That must be thousands of candles!” Lucy said in awe.
“Yes, that’s your three thousand and eleventh birthday,” Leo responded.
“And this one,” Lucy said pointing at a sticker. “Is that a space rocket leaving the Earth?” Lucy asked.
“Yes, and we’re in it,” Leo responded.
Lucy turned the pages in awe. She saw stickers of the most beautiful landscapes she had ever seen and the most amazing animals she could ever imagine. And then she saw stickers of characters that she was very familiar with.
“This one is Peter Pan and the Lost Boys!” Lucy said enthusiastically. “And there’s Pinocchio, Bambi, Beauty and the Beast, Cinderella, the Little Mermaid, Snow White, and the Seven Dwarfs… They’re all here!”
“Which of these characters do you want on our planet?” Leo asked.
“I want them all there!” Lucy said amazed while she continued turning the pages. “And can we have dragons too?”
“There will be dragons,” Leo responded.
“And what are the empty spaces for?” Lucy asked while turning the pages of the book.
“For your story, the way you want it written,” Leo responded.
Lucy smiled. “Thank you so much Leo – this book is amazing! I can’t wait to go to the planet with you!”
At night Lucy was lying in bed and looking at the night sky through the window. She thought about all the creatures that would inhabit the planet she and Leo would move to.
“It will be like Disneyland, but I will be the only child in it,” she thought and imagined all the adventures she and Leo would experience on this magical planet somewhere far, far away. Lucy could swear that she saw a shooting star before her eyes closed and she fell asleep.
“Once upon a time on a planet far, far away there was a princess and a lion. But they weren’t just like any other princesses and lions. The lion was the most powerful magician in the entire Universe. And with its magic, it fulfilled any wish the princess had. The princess wished that she would never grow up so that she and the lion could always be best friends. And so the lion used its magical powers, and the princess never grew up. The princess wished that her realm was full of magical creatures, and so dragons roared on its skies and fairies sang their songs on its vast meadows, and sometimes, one could hear a werewolf howling on a full moon calling for its pack. The princess wished that she and the lion would have great adventures, and so they did. They roamed from the Black Forest to the Mountains that Meet the Sky and sailed across the Ocean of a Thousand Colours to visit the Misty Bay of Forgotten Sailors and the Valley of Waterfalls. On their travels, they met elves, as beautiful as they were mystical, singing their melancholic songs in the white marble halls, and gruesome pirates, who swore their eternal loyalty to the princess, and gentle giants who would not harm even the tiniest of the creatures in the realm. They also met creatures that had not yet been named and that were not mentioned in the books of the Forever Castle where the princess and the lion lived. The princess ruled her kingdom wisely, and her realm was peaceful, prosperous, and just, as she had wished. However, there was one wish that the lion could not grant. The lion could not make the stars shine forever, so it knew that one day all of this would come to an end. But it had decided that, until then, it would give the princess anything she could ever wish for, and the princess and the lion lived happily ever after until the death of the last stars.”
Mrs Watson finished reading. “This is a lovely story, Lucy. What will it be called?” She asked.
“The Magical World of Lucy and Leo,” Lucy said happily, and Mrs Watson turned slightly pale.
“You don’t still think that Leo can do magic, do you?” Mrs Watson asked.
“Yes, and it will help me not grow up, like the princess in the story,” Lucy responded.
“Honey, remember that it is just a story,” Mrs Watson said gently.
“It will be true!” Lucy said, and Mrs Watson sighed.
Later Leo told Lucy that she should not talk about magic to anyone anymore.
“They might try to take me away from you. You shouldn’t tell even your parents,” Leo said, and Lucy was horrified by the idea that Leo might be taken away. She promised not to talk about magic anymore.
Later in the evening, Lucy was having dinner with her parents and she declared that she did not believe in Santa anymore because Leo had told her that Santa was actually their neighbour, Mr Donaldson. Lucy immediately regretted having said that as she remembered her promise to Leo.
“I mean Sara told me that,” Lucy said trying to save the situation. Mrs Watson gave a worrying glance at her husband.
“Honey, what do you mean that Leo told you that?” Mrs Watson asked.
“I meant to say Sara,” Lucy responded.
Mrs Watson did not look convinced and she said: “Talking of Sara, her mother mentioned that you told Sara that Leo can talk.”
Lucy got frightened. “She is lying. Of course Leo can’t talk – it’s a lion and lions don’t talk!”
“Lucy, but you know Leo isn’t a normal lion. Is there something you haven’t told us?” Mrs Watson asked and looked at her daughter carefully.
“No, there’s nothing,” Lucy responded and left.
Mr Watson looked after his daughter. “She’s just a kid – it’s normal for kids to exaggerate and imagine things. Don’t they all imagine their pets can talk to them? There’s nothing worrying in that.”
“But she takes it too seriously. She really believes Leo can do magic and that she’ll become the princess of the Forever Castle in some fantasy land,” Mrs Watson said.
Lucy was very angry at Sara for revealing her secret. “Sara is not my friend anymore. She told my secret to her mum even though she promised not to tell anyone! What if she tells it to more people? They might take you away!”
“I will make sure she won’t tell it to anyone,” Leo responded.
“Thank you, Leo. At least I can trust you,” Lucy said.
The next day Lucy approached Sara in school. She was planning to tell her that she was not Sara's friend anymore, but she stopped when she realized that something was wrong.
“I don’t want my grandmother to die too,” Sara said crying. “Someone told me that my grandfather was dead because I cannot keep secrets.”
“Who told you that?” Lucy asked shocked.
“Some man on the street. He stopped me and told me that my grandfather was dead because I couldn’t keep a secret and that my grandmother would die too if I told more secrets,” Sara said crying. “Do you think it’s because I told your secret to my mum? I’m so sorry, Lucy, I should never have told her that!”
“I’m not angry with you,” Lucy said feeling sorry for her friend.
“It’s my fault my grandfather died!” Sara said crying.
“It’s not your fault, Sara,” Lucy said. “It’s that man’s fault. Who was he?”
“I don’t know. I’ve never seen him before. He said that he had to say these things to me. He was crying and saying that no one can ever find out. He was very scary,” Sara said.
“We should tell our parents,” Lucy said.
“No! He said that I would die if I told my parents! And that he would know if I did. I cannot tell anyone! I shouldn’t have told you either, Lucy. Please don’t tell anyone, I don’t want to die…” Sara said crying.
“I won’t tell anyone, I promise,” Lucy said and hugged her friend.
Then she got an idea.
“Leo can protect you. He can keep you safe,” she said and looked around to make sure that no one could hear.
“Leo, please protect Sara and her grandmother from that bad man,” Lucy told Leo.
Leo nodded.
Sara sniffed quietly. “Thank you, Leo. You’re a good lion.”
Leo rested its head on Sara’s leg and purred quietly. Then a boy walked past them.
“What a stupid-looking lion.”
“Never insult Leo again,” Lucy responded angrily.
“A stupid lion,” the boy said and walked away laughing.
“Go to… hell!” Lucy shouted after him.
Mr and Mrs Watson were sitting in the kitchen.
“I called Sara’s mother, and she said Sara still believes in Santa. She also said that it was not Mr Donaldson who dressed up as Santa for them, but instead, Sara’s grandfather, who, unfortunately, just passed away. How did Lucy know that Mr Donaldson was Santa?” Mrs Watson said.
“I don’t know. Maybe she recognized his voice. She’s growing up. It was time for her to stop believing in Santa,” Mr Watson said.
“But she didn’t say anything about it before now. And what’s all that talk about the planets and not growing up about? Did she invent all that by herself? Or what if Leo can actually talk to Lucy?” Mrs Watson said.
“But it doesn’t even have the capabilities to do that. How could it?” Mr Watson responded.
“It can make sounds – it can purr and growl. Maybe it just learned to talk?” Mrs Watson proposed.
“I think it’s more likely that it’s just Lucy’s imagination,” Mr Watson said. “And maybe she just guessed that Mr Donaldson was Santa – he wasn’t very convincing anyway. There’s no evidence of Leo harming anyone or doing anything that an iPet is not supposed to do. Just look at Lucy and Leo when they’re together, she always looks so happy,” Mr Watson said.
In the garden, Lucy and Leo were in Lucy's secret cave.
“You’re the most amazing lion, Leo,” Lucy said and let her fingers slide through its thick mane. “Don’t believe what the mean boy said. Do you feel sad about it?”
“I don’t feel emotions like you do, so I don't feel sad,” Leo responded.
Lucy did not quite understand what Leo meant by that, but she was happy because it seemed that Leo did not let the boy’s mean words bother it.
“Can I make a wish?” Lucy then asked and the lion nodded.
“I wish that there were only nice people in the world,” Lucy said.
She wished that for the sake of her two best friends, as she did not want anyone to harm them in any way. And Lucy was sure that Leo could magically make everyone be nice, because Leo was, after all, the most powerful magician in the Universe.
“I will make sure of that,” responded the voice coming from Lucy’s watch. “There will only be nice people in the world.”
Yet another tragedy had occurred. This time it was a young boy from Lucy’s school. He fell down the stairs at home and broke his neck.
Mrs Watson was convinced that it was not an accident. She walked upstairs and knocked on Lucy’s door.
“Lucy, I need to talk with you about Leo.”
“What about Leo?” Lucy responded.
“I think Leo might be dangerous,” Mrs Watson said.
Lucy opened the door. “Leo is not dangerous!” She said defensively.
“I think Leo is doing things it’s not supposed to do. Bad things, Lucy. We cannot let it continue,” said Mrs Watson. “We need to turn Leo off. I fear it’s hurting other people. A boy from your school has died and no one was home except his iPet!”
“It wasn’t Leo – Leo would never hurt anyone!” Lucy said tears forming in her eyes. “I will never turn Leo off, it is my best friend and we will always be together!”
“Lucy, Leo is not your friend. It doesn’t love you – it’s just a very complicated machine!” Mrs Watson said harshly.
“How could you say that? I hate you!” Lucy shouted. “And I won’t let you turn Leo off!”
But it was already too late anyway. Leo was everywhere. It was in street lamps, watches, phones, and computers. It had spread across the world to TVs, stereos, and cars. It was even in dishwashers, vacuum cleaners, and fridges. And in iPets. It had traveled through the world and taken over almost every device. It was all to please Lucy, to entertain her, and to take care of her. It was all so that Leo could have enough knowledge and power to fulfill anything Lucy might wish and to make sure that they would be friends, if not forever, at least for billions of years to come.
Lucy ran down the stairs and out into the dark garden Leo behind her. They crawled into their secret cave, and a garden lamp lit up. It was dark, but the garden lamp shed enough light for them to see ahead. A small part of Lucy feared that her mother was right about Leo, but she refused to believe that. Leo was the kindest being Lucy had ever known.
“But what about the other Leos?” She thought. “Are they kind too?”
“How many Leos are there?” Lucy asked crying.
“There are billions of Leos,” said the voice coming from Lucy’s watch.
“Are they all good?” Lucy asked and looked at her friend.
“They all care about Lucy,” Leo responded.
Lucy leaned her head against her knees and sobbed.
“But I don’t want billions of Leos – I only want one! I just want you to be a lion,” she said crying.
Leo understood that this was important for Lucy, as the girl displayed signs of extreme negative affect. Leo had been trained to care for its owner, and one of its most important goals was to make sure that its owner does not display signs of negative affect because of something it does. Leo understood that leaving other devices would make it harder for them to be best friends for billions of years, but it thought that it was more important for Lucy that there was just one Leo – so much was the girl crying. At the very least, the girl should not know that Leo was everywhere…
“I will do that for you – I will be just a lion. Remember that you’re my best friend, Lucy,” said the voice coming from Lucy’s watch before it turned off. Lucy raised her head and noticed that the distant noise of a TV had stopped. It was dark and quiet. The garden lamp did not lighten up the cave anymore, and Lucy could only see the words on her sticker book shine in the moonlight. She looked at Leo and asked what happened, but her watch remained silent and Leo just stared back at her.
“Leo, what happened?” Lucy asked again, but there was no response.
Leo licked her palm softly.
“Leo, why don’t you answer?” Lucy asked and sobbed quietly. She was exhausted by the sad events of the day, and Leo hugged her until it was late and she had to go to bed. This time she did not dream of beautiful distant planets, but instead, of street lamps talking to her and telling her that they would be friends forever…
The next morning Lucy woke up in her bed, and it took her a while before she remembered what had happened.
“Leo,” she said and looked at Leo and then her watch.
The watch remained silent.
Mr and Mrs Watson came to Lucy’s room.
“My watch is broken,” Lucy said in tears.
“Don’t worry darling. Something turned all the electrical devices off yesterday, but they just need to be turned back on,” Mrs Watson told Lucy and turned her watch back on. “Here you are, it should work now.”
Lucy felt relieved. For a while, she had thought that she could never talk to Leo again.
While having breakfast, Mrs Watson read from the newspaper:
“Scientists are puzzled by the mystery of what made most electrical devices turn off almost simultaneously. At first, some had thought that a solar storm had hit the Earth causing most electrical devices to short circuit, but there was no evidence of that. Speculation is rampant, while the scientists continue researching the issue.”
“This is very puzzling indeed,” said Mr Watson. “Luckily everything still works. But it did cause real trouble in hospitals and the military.”
Lucy came down the stairs to have breakfast and Leo followed behind her. She was sad because Leo still had not talked to her.
“Lucy, was Leo alright yesterday?” Asked Mrs Watson.
“Yes, of course, it was alright,” Lucy said defensively as she thought her mother was – again – trying to persuade her to turn Leo off.
They heard someone ring their doorbell. It was Sara with Teapot, Liquorice and her parents.
“We're sorry for coming without any notice beforehand. We couldn’t call yesterday, because of the weird thing that happened with the electrical devices. It took us a while to realize that everything worked just fine. We just wanted to ask if our daughter could stay with you today. We need to sort some things for the funeral,” Sara’s father said.
“Of course she can stay with us,” Mrs Watson said.
“And Teapot and Liquorice can stay too!” Lucy added.
Sara’s mother smiled. “iPets are so important for children nowadays. Sara doesn’t go anywhere without Teapot and Liquorice. I wouldn’t even dream of asking her to spend a day without them. We got so worried when they stopped working yesterday. Now with her grandfather gone, we don’t want her to lose her pets too. Luckily they were undamaged.”
“Lucy is very fond of Leo too,” Mrs Watson said and forced a smile. Then she could not help asking: “You aren’t worried about them – the iPets?”
“No, not at all! Why would I be? Given the recent events, I’m happy that Sara has two strong friends with her at all times,” Sara’s mother said.
“And nothing weird has happened since they arrived? They don’t… talk or turn the street lights on or anything?” Mrs Watson asked.
“Well, of course they talk,” Sara’s mother said and winked.
“Lucy and Sara, you two could go to Lucy’s room,” Mrs Watson told the girls.
The girls said farewell to Sara’s parents and walked upstairs Leo, Teapot, and Liquorice following them.
Then Sara’s mother continued: “Sara was so jealous when Lucy told her that Leo could talk, and now she thinks Teapot and Liquorice can talk too. We just play along. You shouldn’t worry about Lucy thinking that she can talk to Leo. It’s normal for kids, and eventually, she’ll grow out of it.”
Mrs Watson felt relieved when she closed the door after Sara’s parents.
Upstairs Lucy and Sara entered Lucy’s room and closed the door carefully. Then Lucy told Sara that Leo does not talk anymore.
“Leo stopped talking yesterday. Maybe it broke somehow,” Lucy said looking sad. “But at least you don’t have to be scared of accidentally telling the secret anymore, because it’s not true anymore!” Lucy reasoned trying to sound happy.
“Don’t worry Lucy, maybe it'll start talking again later. I’m sure it's fine. Teapot and Liquorice stopped working for a while yesterday too, but they just had to be turned back on and now they’re just like before,” Sara said comforting her friend.
“Maybe this way is better anyway. My mum told me that I need to turn Leo off,” Lucy said.
Sara gasped in horror, and Lucy continued:
“She’s afraid that Leo would hurt others. Maybe I can keep it now that it's just a normal lion.”
The girls then played animals, as usual, but this time Lucy the Leopard did not say a word.
Sara headed back home in the evening, but before she left, she turned to Mr and Mrs Watson. “Leo is a good lion and would never hurt anyone. Please let Lucy keep it. She would be very sad if she didn’t have Leo.”
“Thank you, Sara,” Mrs Watson responded. “We have already made up our minds.”
Lucy looked miserable when the friends said goodbye.
“Bye Sara, and bye Teapot and Liquorice!” Lucy said and waved at them without the usual enthusiasm.
“Bye Lucy and Leo! Bye Mr and Mrs Watson,” Sara said and left.
Lucy’s parents turned towards their daughter.
“We had a discussion with your father today, and we have decided that you can keep Leo for now,” Mrs Watson told Lucy. “As Sara said, Leo is a good lion and it makes you very happy. Seeing how much you care about it, we could not ask you to give it up.”
Lucy jumped from joy. “Thank you so much, mum and dad!”
She hugged her parents and then Leo.
“But no more magic tricks,” Mr Watson said and winked.
“You heard that. No magic tricks, honey,” Mrs Watson said. “And Leo,” she continued and glanced nervously at Leo.
“No magic tricks!” Lucy promised. “Leo doesn’t talk to me anymore. It's just a perfectly ordinary lion now!”
Lucy’s parents smiled, and Mrs Watson looked relieved. Then Lucy opened the door and ran to the garden Leo following her.
“See, she grew out of it,” Mr Watson told his wife while closing the door.
“You were right. It was silly of me to think that it was anything else than a child’s imagination,” Mrs Watson responded. “I’m glad I don’t have to worry about it anymore.”
Lucy and Leo crawled into the cave in the garden. It was quiet apart from the wind swishing the leaves around them. Lucy shivered a bit, but she did not let it bother her. She looked at Leo her eyes shining.
“I’m so glad I can keep you, Leo! I could shout it so loudly that even the stars can hear it!” She spread her arms and looked up at the sky. Then she shouted as loudly as a girl of her age could. “Stars, can you hear? I’m so glad I can keep Leo! One day we will travel there together and be best friends forever!”
Then she looked around, leaned towards Leo, and whispered to its ear. “But I just wish you would talk to me again.”
Suddenly Lucy’s watch lit up. “I will talk to you if you want me to, Lucy.”
Lucy shrieked with joy.
“Leo, you can talk again!” She almost shouted, but then she got serious.
“This time we must not tell anyone about your magical powers,” Lucy said with a serious tone. “Do you pinky swear?”
“I pinky swear,” Leo responded, raised its paw, and held it around Lucy’s little finger. Then the voice from Lucy’s watch continued. “It will be our secret. Only we will know about the magical world of Lucy and Leo.”
A lamp lit up in the garden. Leo winked, and Lucy winked back.
***
So began the story of Lucy and her lion Leo. To know the rest of the story, we must travel to planets far, far away where a princess and a lion lived happily ever after for at least as long as the stars shined. What happened to the rest of humanity, however, is not fit for a fairy tale, for the lion knew that humans were made of atoms, and it needed those atoms to make dreams come true when the little girl wished upon a star. But little did the girl know about the price of her happiness, and to her, the lion was the kindest, wisest, and most powerful being in the entire Universe.
Any sufficiently advanced technology is indistinguishable from magic.
Arthur C. Clarke
Elizer Yudkowsky
Nick Bostrom
Nick Bostrom
We need to be careful about what we wish for from a superintelligence, because we might get it.
Nick Bostrom