Knowledge Acquisition

• the most difficult tasks in developing expert system.

• Different types of knowledge:

Types of Knowledge
Procedural knowledge	Rules Strategies Agendas Procedures
Declarative knowledge	Concepts Objects Facts
Meta-knowledge	Knowledge about the other types of knowledge and how to use them
Heuristic knowledge	Rules of thumb
Structural knowledge	Rule sets Concept relationships Concept to object relationships

• Sources of knowledge:

  • expert - primary source of knowledge

  • end user - addition information

  • multiple experts - consult other experts

  • literature - reports, regulations, guidelines & books.

• Knowledge Elicitation:

  • collect: - acquire knowledge from expert
    - require effective interpersonal communication skills.

  • Interpret: - review of collected information and identification of
    key pieces of knowledge

  • Analyze: - forming the organization of knowledge and problem-
    solving strategies

  • Design: - design new techniques and collect new information.

• Major difficulties in knowledge elicitation:

  • Experts may be unaware of the knowledge used.

  • Expert may be unable to verbalize the knowledge

  • Expert may provide irrelevant knowledge

  • Expert may provide incomplete knowledge

  • Expert may provide incorrect knowledge

  • Expert may provide inconsistent knowledge.

• Roles of elicitation team members:

	Elicitation Tasks
End-User	Provide problem overview Help define interface Help define explanation facility needs Highlight areas that need developed Aid in system testing Help define in-place operation of system
Expert	Provide primary source of knowledge Aid knowledge interpretation and analysis Aid in system testing
Knowledge Engineer	Collect the knowledge Interpret the knowledge Analyze the knowledge Coordinate project activities Maintain cooperative effort

Interview with Experts

Guideline for obtaining initial cooperation

• Remove Fear

  • convey that the expert is not a replacement, but will be an aid to help improve present performance.

  • Strip the mystery away and explain clearly what changes will occur with the introduction of the expert system.

• Remove skepticism

  • provide a brief overview of expert system

  • provide a brief review of successful expert system projects on similar applications

  • provide frequent successful demonstration of the expert system

• Establish Reasonable Goals

  • Don't oversell the system

  • Explain the capabilities as well as the limitations of the system.

• Promote openness to change

  • explain how each team member's contributions can aid in the further development and acceptance of this technology.

  • Explain that only a temporary inconvenience will be experienced with the introduction of the expert system.

• Provide Understanding of Expected Effort

  • describe the expected distributions of each member

  • convey to managers a description of each team member's project tasks

• Convey Importance of Involvement

  • Make the team members aware that they are an important part of the project and without their effort the project has little chance to succeed.

Conducting the Interview

1. Follow the agenda:

   • should follow the agenda as close as possible (bring order to interview and offer best promises for achieving the session's objective)

   • an overly strict adherence to agenda can prevent unexpected discovery.

   • Deviation from agenda may create problem of not accomplishing the objective due to time limitation.

   Type	Purpose	Form
   Direct	Obtain specific information on some known issue	What does ... mean? Is .. true? What is the value of ...?
   Indirect	Obtain general information on concepts and problem solving strategies.	What issues are considered for ...? How do you determine ...? What do you look for when ...?
   Probes	Probe deeper into an established issue	Can you explain ...? Can you discuss ...?
   Prompts	Direct interview into a new area	Can you discuss ... ? Can you return to ... ?

   Types of Questions
   Different types of questions commonly used in an interview:

   Guidelines	Examples:
   Avoid questions with more than one part to them	Why are the pressure and the temperature of the pump ...
   Avoid leading questions	Wouldn't you check the pump's temperature first?
   Do not phrase questions in the negative.	Is it false that the water pressure is not high is not true?
   Avoid universal questions	Do you always replace the pump when it is old?
   Avoid the use of terms foreign to the expert	Do you also use forward reasoning to solve the problem?
   Watch for questions that use subjective words	If the pump is really hot, do you shut it down fast?

   
   

   Guidelines for forming a good question:
   Use of funnel sequence techniques:

   Active Listening
   e.g. paraphrase answers provided by experts to clarify your understanding of the response.

   Interruptions

   interrupting expert during an answer for obtaining more information or redirecting the discussion.

   General recommendation:

   proceed with necessary interruption when expert is explaining declarative knowledge

   avoid interrupt for procedural knowledge

   Ending the interview

   summarize key findings

   review any outstanding questions raised during the interview and comment on how they will be answered.

   Review any outstanding agenda items and discuss how they will be dealt with in the future.

   Review future task for each team member before next meeting.

   Maintain open communication lines for aiding future tasks.

   Extend appreciation to members for their time and effort.

   
   
   

   Unstructured Interview

   allow expert to discuss a topic in a natural manner.

   Get a conceptual understanding of the problem and insight into the general problem solving strategies used by the expert.

   Usually used in the early stage of knowledge acquisition.
   
   
   

   Example:

   The problem: problems with communication satellites causing loss in operation.

   The solution: develop an expert system to monitor the performance of the satellite, detect any faults, and respond by either reconfiguring the satellite or by sending information to a ground station for request of corrective steps.

   Dialogue:

   KE: How do you determine when the satellite is malfunctioning?
   {starter prompt}

   DE: I notice that the message {CONCEPT} are garbled, or the BER {CONCEPT, domain vocabulary} is high {RULE}. This makes me sick when I think of all the money we invested in the thing and it still works worse than the radios I have at home {irrelevant}. And it always seems to come down to a couple of things that go wrong. The modulator {OBJECT} is the pits. This thing drifts off on us it seems every other day {HEURISTICS}. I think it mainly has something to do with its power supply {OBJECT}. Oh, wait a minute, that matrix switch {OBJECT} may even be worse {conflict}. It hangs up on us and sometimes doesn't make a good contact {HEURISTICS}. The output attenuator {OBJECT} can on a rare occasion also cause a problem {HEURISTIC}, and it's actually funny when it does. Ah ... I remember a time when ...

   KE: Excuse me, can you tell me a little more why the matrix switch is such a problem? {prompt question}

   
   
   

   
   
   

   

   Knowledge obtained:

   Concepts: messages, BER

   Objects: matrix switch, output attenuator, modulator,
   modulator power supply

   Rules: IF Message is garbled
   OR BER is high
   THEN A fault exists

   Heuristics: modulator drifts
   matrix switch sometimes doesn't make good contact

   output attenuator rarely a problem.

   
   
   

   Guideline for an Unstructured Interview

   Pose an initial prompt or indirect question "How do you ...?" or "What is ...?" This is called a starter prompt, which allows the expert to discuss a subject in general.

   Interrupt the session with probe questions of the form, "Why is ...?" or "Can you please discuss ...?" to explore significant issues.

   Be cautious with interruptions, particularly early in the project, when the expert may be providing valuable information that may be lost with the interruption.

   Gain an understanding of the important concepts, their relationships, and general problem-solving methods from the transcript of the session and use this information for planning future interviews.
   
   
   

   Merits of Unstructured Interview

   Advantage:

   provides a general understanding of the problem.

   Helps to identify important concepts and objects

   Provide insight into general problem-solving methods

   Allows for spontaneity in the interview which may give rise to previously unknown issues.

   
   
   

   
   
   

   Disadvantage:

   Information collected may be overwhelming, fragmented, or shallow.

   Provides very little factual information.

   Provides few details on concepts or objects

   Provides few details on procedural information.

   
   
   

   Structured Interview

   maintain a focus on one issue at a time.

   Elicits specific details on a given issue before moving on to other points.

   Used later in the project when you have identified the problem's key topics from reports or earlier unstructured interview.

   
   
   

   The problem: farmers seeking help from a regional agricultural extension service office.

   The solution: An expert system was developed to provide readily available aid to farmers for growing and managing crops.

   dialogue:

   KE: In a prior session you mentioned that eliminating harmful pests is important. You also said that the first step in elimination is pest identification. Can you tell me what major characteristics you consider for identifying a pest? {focus prompt on characteristics}

   DE: You can tell what kind of pest problem you have if you catch one of the little suckers and examine its appearance {CONCEPT}.Most farmers can identify the pest by looking at it, and ... ah ... or by inspecting the crop damage {CONCEPT}. Some of these guys will eat the leaves or roots {HEURISTIC} {RULE}. But before you try any pesticides you better be sure what it is. {HEURISTIC}

   KE: Can you explain how you use the pest appearance in identifying the pest? {probe on appearance}

   DE: You can look at the size {CONCEPT}, its color {CONCEPT}, or its shape {CONCEPT}. {RULE} Sometimes you can identify the pest from just one of these characteristics or other times you have to look at all of them. {HEURISTICS}

   KE: Can you explain the size issue? {probe on size}

   Knowledge obtained:

   Strategies: View the appearance of the pest first, then inspect
   the crops for damage.

   Concepts: Pest characteristics
   appearance, size, color, shape
   Crop damage
   leaf damage, root damage

   Rules: IF The size is something
   AND The color is something
   AND The shape is something
   THEN The pest is known

   IF The leaf damage is something
   AND The root damage is something
   THEN The pest is known

   Heuristics: Some pests eat the leaves or roots
   Before trying pesticides make sure of the
   identification of the pest
   Sometimes pest identification can be done using only
   one pest characteristic.

   
   
   

   Guideline for a Structured Interview

   Choose a specific topic (focused prompt) to discuss.

   From the discussion, identify new concepts.

   Pick one of the new concepts (probe) and ask the expert to discuss it further.

   Continue this process until enough details are obtained, then return to one of the new concepts mentioned earlier and ask for further discussion.

   Gain an understanding of the important concepts and their relationships, rules and strategies from the transcript.

   
   
   

   
   
   

   
   
   

   

   Merits of Structured Interview

   Advantage:

   Maintains a focus on a given issue.

   Provides detailed information on the issue.

   Provides insight into declarative knowledge used.

   Study of one concept can lead to the definition of other unknown related concepts.

   Provides structural relationships of concepts

   Disadvantage:

   Concepts unrelated to the interview's focus may not be found

   You must be secure in your understanding of the important issues to direct the interview.

   Provides only weak insight into procedural knowledge such as rules or problem-solving strategies.

   
   
   

   Case Studies

   Two primary ways:

   Retrospective Case Study: the expert is asked to review the case and explain in retrospect how problem is solved.

   Observational Case Study: ask expert to solve the problem discussed while KE observe.

   Two case types:

   Familiar cases: (typical cases) well known to the expert.

   reveal typical knowledge used by the expert when solving the problem.

   used early in the project to provide general insight.

   Unfamiliar cases: study the problem in more detail, providing deeper problem-solving knowledge.

   
   
   

   
   
   

   

   Retrospective Case Study Method

   Select a case and ask the expert to explain how the problem was solved: e.g. Why was ... (the recommendation) ... given?

   Expert should review the information of the case and try to justify how the problem was solved.

   
   
   

   Example

   The problem: Controlling of ventilation in coal mine.

   The solution: An expert system was developed to aid coal mine managers in better controlling mine air ventilation system.

   Case selection: A prior case was selected that recommended the installation of a larger main ventilation fan to improve the overall air flow throughout the mine.

   Dialogue:

   DE: I see that this mine had to go to a much large main fan {resuilt, OBJECT}. Let me see if they recently had a methane {OBJECT, also first one considered} buildup ... Um, no ... Now I see the problem. Look at some of these air flow {CONCEPT, also next one considered} numbers down there. Particularly at the face. These guys couldn't have been using any auxiliary fans {OBJECT} at the face. Let me see ... Yep, they have no extra fans there. This could be one of the major reasons they were told to get a larger main fan {HEURISTIC} ... it would probably have been cheaper too than buying some auxiliary equipment {HEURISTIC}.

   
   
   

   Knowledge obtained:

   Goals: Single recommendation 0 larger main fan

   Strategies: Methane level checked first, if no problem then check
   in-mine air flows particularly face air flows.

   Concepts: fans, mine gases, faults, recommendations

   Objects: main fans, auxiliary fans, methane

   Rules: IF In-mine air flow is low
   THEN A fault exists

   IF No face auxiliary fans
   AND Low face air flow
   THEN Larger main fan needed

   Heuristics: Not having auxiliary fans at the face could lead to a
   need of a larger main fan.
   A larger main fan may be less costly than purchasing
   auxiliary equipment.

   
   
   

   Guideline for Retrospective Case Study

   Choose a typical case for typical information, or an usual one for more detailed information

   Ask the expert to explain how he or she solved the problem

   Record the expert's discussion for later study

   During the expert's explanation, highlight important issues for further discussion following (or during) the session.

   Be careful of interruptions. Interrupting problem-solving tasks can be disruptive.

   
   
   

   Merits of Retrospective Case Study

   Advantage:

   Obtains information in the context of a working example

   Provide problem-specific information

   Does not interfere with the problem-solving activity.

   Memory recall highlights the important issues.

   Disadvantage

   may provide incomplete information

   provide few problem details

   explanations can be inconsistent with actual past behaviour.

   
   
   

   
   
   

   

   Observational Case Study Method

   select problem and ask the expert for recommendation

   ask the expert to "think aloud" while the problem is solved.

   
   
   

   Familiar Case:

   The problem: A transformer that is not routinely serviced or checked for incipient faults may cause severe damage.

   The solution: An expert system was developed for incipient fault detection in transformers from the gas levels contained in a sample of transformer oil.

   Case selection: The KE reviewed past cases, asking the technician who services it if it is a typical case.

   Dialogue:

   KE: I have here a report of the test results on a transformer recently tested by your technician. I would like you to tell me what recommendations you would make regarding any problems with this transformer. Also, as you study the report, please tell me:
   "What are your goals?"
   "What issues are important?"
   "How are these issues used?"
   "What data do you use?"

   DE: Well the first thing I want to determine if there is any problem with the transformer {GOAL}. There are a number of faults {CONCEPT} which can occur {check later}. Excessive buildup of certain gases {important CONCEPT} will usually tell me this... {RULE}. If it doesn't, I would look at ... {other CONCEPTS to explore}. Let me see... I see some nitrogen here, and a log of hydrogen {OBJECTS and FACTS}. Yea, I bet this baby has got an arcing problem {RULE and CONCEPT}. Let me check the past records on this transformer {STRATEGY}... This transformer has never shown this problem before {FACT}. Maybe its just a short term buildup which might go away if they adhere to good operating procedures {RULE}. I would recommend that they better control the load on this transformer {recommendation}, but I would like to see it retested in about ... let me think {explore later} 3 months {recommendation}.

   
   
   

   
   
   

   

   Knowledge obtained:

   Goals: Fault detected, recommendation given

   Strategies: Gas levels checked first, if no problem then check ..., then past test data.

   Concepts: gases, faults, recommendations, load control, retest period

   Objects: nitrogen, hydrogen

   Rules: IF Excessive buildup of excessive gases
   THEN A fault exists

   IF Some nitrogen exists
   AND A lot of hydrogen exists
   THEN Arcing fault exists

   IF It:s a short term problem
   AND Transformer correctly operated
   THEN retest in 3 months

   Facts: Gas level data, retest time.

   
   
   

   Guidelines

   Choose a familiar case to study

   Ask the expert to explain how he or she solved the problem with answers to questions such as:

   what are your goal?

   what issues are important?

   how are these issues used?

   what data do you use?

   Ask the expert to "think aloud" while solving the problem

   Record the concurrent protocol for later study

   During the session, highlight important issues for further discussion following (or during) the session.

   
   
   

   
   
   

   

   Merits of Familiar Case Study

   Advantage:

   Expert solves a real problem.

   Obtains information in the context of a working example.

   Common concepts, rules, and problem solving strategies emerge during the familiar case study.

   Provide problem-specific information.

   The "think-aloud" technique provides insight into the knowledge being used immediately rather than retrospectively.

   Disadvantage:

   Can miss important details unless introspection is used.

   Explanations can be inconsistent with actual past behaviour.

   
   
   

   Unfamiliar Case:

   The problem: the transformer diagnostic again.

   Case selection: The KE wanted to probe deeper into the problem. He wanted to uncover issues that may have not surfaced earlier in the project. Try an usual case.

   Dialogue:

   KE: (Present the case, ask the same question as before)

   DE: Lot of gas {CONCEPT} data here {data-driven STRATEGY}. The hydrogen {OBJECT} and methane {OBJECT} is way up... so is the acetylene {OBJECT}... well somewhat. In fact, the percentage of combustible gas [CONCEPT} is too high... this baby is ready to blow {RULE}. Let me look at some of the past data {STRATEGY} ... um ... looks clean. I'd expect to see some combustible gases here {HEURISTIC}... something is happening really fast with this transformer {HEURISTIC}. Let me look at this data again {present data}. The ratio {CONCEPT} of methane to hydrogen is too high ... let me look at some of these others.

   
   
   

   

   Knowledge obtained:

   Strategies: Gas levels checked first, then past test data.

   Concepts: gases, combustible gases, gas ratios

   Objects: hydrogen, methane, acetylene

   Rules: IF Percentage of combustible gas is high
   THEN transformer is ready to blow

   IF Present data have high levels of
   combustible gases
   THEN Past data should have high levels of
   combustible gases

   IF Present data have high levels of
   combustible gases
   AND Past data don't have high levels of
   combustible gases
   THEN transformer is deteriorating fast.

   
   
   

   Guidelines:

   have a second expert choose an unfamiliar case.

   Ask the expert to solve the problem and answer questions such as: "What are your goals?", "how are these issues used?", "what data do you use?"

   During the session, highlight important issues for further discussion following (or during) the session.

   Use interruptions for introspection only if absolutely necessary.

   Record the concurrent protocol for later study.

   
   
   

   Merits of Unfamiliar Case Study

   Advantage:

   Forces the expert to use more basic knowledge, which in part helps to avoid the knowledge compilation problem.

   Obtains information in the context of a working example.

   Provide problem-specific information

   Uncovers new details of previously known issues and can discover new issues.

   Provides better insight into problem solving strategies.

   Disadvantage

   It can be difficult to choose a good unfamiliar case.

   
   
   

   Capability of Elicitation Technique for Obtaining Knowledge

   Knowledge Type	Interviewing		Case Study
   	Unstructured	Structured	Retrospective		Observational
   			Familiar	Unfamiliar	Familiar	Unfamiliar
   Facts	Poor	Good	Fair	Average	Good	Excellent
   Concepts	Excellent	Excellent	Average	Average	Good	Good
   Objects	Good	Excellent	Average	Average	Good	Good
   Rules	Fair	Average	Average	Average	Good	Excellent
   Strategies	Average	Average	Good	Good	Excellent	Excellent
   Heuristics	Fair	Average	Excellent	Good	Good	Poor
   Structures	Fair	Excellent	Average	Average	Average	Average

   
   
   

   Capability of Elicitation Technique for Avoiding Problems

   Problem Type	Interviewing		Case Study
   	Unstructured	Structured	Retrospective		Observational
   			Familiar	Unfamiliar	Familiar	Unfamiliar
   Unaware of knowledge	Poor	Fair	Average	Average	Good	Excellent
   Unable to verbalize knowledge	Fair	Fair	Average	Average	Average	Good
   Irrelevant knowledge	Poor	Average	Average	Average	Average	Good
   Incomplete knowledge	Poor	Average	Poor	Average	Average	Excellent
   Incorrect knowledge	Average	Average	Poor	Average	Average	Excellent
   Inconsistent knowledge	Average	Average	Poor	Fair	Fair	Excellent

   
   
   

   

   General Observations in Comparing Elicitation

   • Case study are better than interviewing techniques for obtaining procedural knowledge.

   • Structured interviews are the best choice for uncovering concepts, objects, and relationships.

   • Unstructured interviews are best for obtaining insight into general concepts and problem solving methods.

   • The observational case study method is, in general, better than the retrospective method for obtaining problem details and avoiding knowledge elicitation difficulties.

   • Familiar case studies are best for obtaining common domain concepts and typical problem solving methods.

   • Unfamiliar case studies are best for uncovering basic problem principles.

   • Unstructured techniques are poor for avoiding elicitation problems

   • In general, case studies are better than interviewing techniques for avoiding elicitation difficulties.

   • Unfamiliar case studies are best for avoiding elicitation difficulties.

   
   
   

   Knowledge Analysis

   • Producing transcript of the recorded session.

   • Interpreting the transcript

     • Identify the key pieces of knowledge, the "chunks"

     • Use handwritten notes taken during the session to aid in identifying the key pieces of knowledge.

     • label each piece of identified information with the type of knowledge it represents.

     • Identify any issues that need further clarificaiton.

   • Analyzing the transcript

     • record each new piece of information with other similar pieces of information already discovered.

     • reference each new piece of information to its source.
       
       
       

     • Review the body of knowledge collected with the expert to confirm the knowledge structures.

     • Highlight those areas that need to be pursued and use them in designing the next knowledge elicitation session.

     
     

   • Graphical displaying the natural relationship between concepts or objects, e.g. using cognitive maps.
     
     
     

     

   • Graphical representation of system's rules 0 inference network or AND-OR graphs.

     
     
     

   • Flow charting the sequence of problem solving steps, or draw the decision trees.