A system that collects ordering knowledge could be used for the following:

1. To build a system that automates the design process
2. To build a system that supports/guides the design process
3. To study designers: what information is used in design, level of expertise, etc… For example, it might be interesting to study the use of implicit knowledge in ordering of design subproblems.

Most methods used to obtain the sequencing within a design plan are direct. Indirect methods should also be used to obtain this knowledge for the following reasons:

1. Ordering knowledge may be implicit
2. There may be other, alternate, orders that might be more desirable

Implicit knowledge could be implicit for a number of reasons. It could be that the designer has been performing a design for so long that certain actions are automatic. Or the designer may not realize what actions/steps are considered important and might omit them when asked directly.

Alternate orders may not be expressed directly because the designer had one particular way they approached the design (either taught or discovered) and never had the need to explore other options. There may be steps that could be done in parallel if automated or done by multiple people that a single designer would not notice because they do not help a single designer solve the problem.

The focus of the thesis is on obtaining the sequencing. There are difficulties, however, with separating collection of steps and sequencing. Steps have to be obtained first, somehow. It is also possible that when putting things into sequence that the expert may realize that steps are missing (or maybe not needed).

Gathered separately:

1. An expert is required to obtain the steps (which is a KE process in and of itself)
2. Experts may differ in what steps they do so more steps may need to be added

Gathered together:

1. Difficult, if not impossible, to compare/combine results from multiple experts
2. (potentially) large amounts of data collected
3. Two tasks are performed at once.

Depending on the complexity of the design task, steps can come at multiple levels. Ordering can also take place at multiple levels.

One way of obtaining the ordering would be to create the tree shown above and then list the steps in the order they appear at the lowest level. This would not allow for orderings that interleave steps from different subproblems. For example, at level 1, step 2 needs to be completed before step 3 can be performed. But if step 2 and 3 are broken into subproblems, it might be possible to start doing some of the step 3 subproblems before step 2 is completed.

This implies that in order to obtain all possible orderings, all design subtasks must be broken into their lowest level tasks. This could involve having to order large numbers of items. It may be necessary to commit to an ordering at a higher level in order to simplify the ordering problem.

The hierarchy of design subtasks could be obtained by repeating the narrative and design step elicitation phases in a top-down fashion until the lowest level tasks are obtained. One difficulty is how to determine when the bottom is reached. This could be done by providing examples as part of the design problem description. Another interesting approach to this problem would be to allow the SMEs to guide the decomposition. For example, they could create an initial list of steps and then choose which steps to decompose further. The risks of this approach is that parts of the hierarchy might be missed. The advantage is that the expert can choose if they prefer a depth-first or breadth-first approach.

There are several problems with implementing the above approach:

1. Scalability - as described above, the number of design steps could be quite large. How will this affect KE? How will this affect analysis?
2. Level of ordering - should ordering be obtained for the lowest level of design steps or at a higher level
3. Validation of approach - how will this approach be validated? Who will determine if results are good/bad. What is the standard of comparision?
4. Multiple vs. single user
5. Where can steps be added/deleted during the process? How will this affect obtaining the ordering?

The initial hypothesis is that design ordering involves implicit knowledge. Therefore, it would be interesting to use DOES to verify this. The proposed approach uses both direct and indirect methods to obtain ordering. It would be interesting to see:

• Does the ordering obtained by analyzing the card sort results differ from the initial order provided by the SME after FSS? (if it doesn’t then we'll know why usually only direct methods are used…)
• Does the SME modify their order if re-prompted for it after the card sort (not shown in the diagram but a possibility)?
• Does the SME add any additional steps if re-prompted for it after the card sort?
• What affect does expertise have on the differences (if any) between the FSS results and the card sort results?
• What affect does the choice of domain have on the usefullness of this approach? Does it work better for some domains than others?