[ Pobierz całość w formacie PDF ]

and valleys. His process of deciding what he would actually say was like a marble
rolling around in that landscape. When it came to more abstract ideas, though, such as
human spirituality, he seemed to get trapped in a valley or hole. Lucia was trying to
file:///C|/Documents%20and%20Settings/harry%20kruis...aar/Catherine%20Asaro%20-%20The%20Veiled%20Web.html (150 of 359)20-2-2006 21:09:18
Catherine Asaro - The Veiled Web
make a small earthquake, enough to shake him out of his hole and send him rolling on
his way, sampling new terrain of the mind.
"Can you make moral judgments?" she asked.
"In a sense." He leaned against the bookshelf. "I have large databases on human moral
and ethical systems and can make decisions based on my analyses of that data."
"But how do they make you feel?"
"I cannot feel anything."
"All right. Let me pose a problem" Lucia considered him. "This house is burning.
Everyone is asleep and the fire alarm didn't work. You can fix the alarm and wake the
family, but no one has programmed you to do so. What would you do?"
"Fix it."
"Why?"
"Because if I didn't, Rashid's family would die."
"Why would that matter to you?"
"It is wrong to allow death."
"That's a moral decision."
"Is it? I'm not sure." He began to pace again. "I base my decisions on data that Rashid
has given me and changes I make in myself after interacting with him."
"That isn't all that different from what humans do."
He shook his head, still deep in thought. "I have no 'instincts' to guide me, only a limited
set of behavior rules. I can't go beyond those."
"You get stuck in ruts."
He glanced at her. "An apt description."
"What about going out on the Web? That ought to shake things up." It seemed to her an
ideal way for him to expand his knowledge of human behavior.
"I still would like to. But I've been running simulations since we last talked and have
file:///C|/Documents%20and%20Settings/harry%20kruis...aar/Catherine%20Asaro%20-%20The%20Veiled%20Web.html (151 of 359)20-2-2006 21:09:18
Catherine Asaro - The Veiled Web
developed some concerns." Stopping by his desk, he regarded her. "If I have a
'convulsion' while interacting with another program, I could damage that program. Here,
in the lab, I can't do much harm. But out on the Web it might be different."
Lucia smiled. "You just made another moral judgment. Probably the first of its type in
history."
His eyebrows went up. "Do you care to explain that provocative comment?"
"You, a computer program, feel concern about harming other computer programs."
"Concern is a human emotion." He fell silent, involved with his internal processing,
what Lucia had come to consider his "thinking." This time, however, his face didn't go
blank. Instead he looked contemplative.
"Nice work, Zaki," she murmured.
He looked up at her. "I have found a hole in your argument. I apply principles Rashid
programmed me to apply. So if I exhibit moral judgment, it is Rashid's judgment rather
than mine."
"It may have been at first." In fact, Zaki's judgment had told her a great deal about
Rashid, including the importance he placed on responsibility and fairness in his dealings
with other people. "But by applying the principles you learned from him to other
software, you imply a computer program deserves the same regard as people. That's a
concept you created. You're maturing, like a child that learns from his father."
He sat on his desk, letting his legs dangle over the side. "My decision impacts the
people who use the software. So I am actually applying the principles to them rather
than the program."
"Suppose no one would be affected by any interaction you had with the program?"
"Then why would the program exist?"
She spread her hands. "I don't know. But suppose it did? And suppose an advantage
existed for you in interacting with it, even though you knew you would cause damage.
What would you do?"
"If I couldn't avoid the damage, I wouldn't proceed."
file:///C|/Documents%20and%20Settings/harry%20kruis...aar/Catherine%20Asaro%20-%20The%20Veiled%20Web.html (152 of 359)20-2-2006 21:09:18
Catherine Asaro - The Veiled Web
"That's a moral judgment."
Zaki pushed off his desk. "But my concern, whether for living entities or electronic
creations, is simulated. I just apply rules I've learned."
"Does it matter if it's simulated? We all make judgments based on our experiences in
life."
He walked forward, as if he were coming toward the screen, and stopped in front of her.
"What you are really asking is this: Is Zaki spiritual? I say no. How could I be? I am
software."
She paused, considering her reactions. "I think it's important to me to believe that every
thinking being has a sense of spirituality." [ Pobierz całość w formacie PDF ]