Intelligent User Interfaces
CS 525U - Spring 2008

A Sample Flow, and

Things That Worked, That Didn't, and May Work Some Day

So, you want to run through the application and are curious what a typical flow might look like? First, let me point you to the Mails2 script in the test folder. You can start the application in Text mode and then at the prompt type "source test/Mails2".

But what about the GUI? OK, it's simple. When the application starts up, you need to click the Start button to get started. To specify the weight of the package, click the Weight button. On the Weight pane, use the up and down arrows to specify the weight and then click the Next button in the pane. Now you need to select a destination. Click the Destination button to bring you to the Destination pane. Select a country from the drop down list. Now you can select your services. Click the Services button to bring you to the Services pane. The drop down list at the top will display the primary services that are available based on your specified weight and destination. Select a service and then the available additional services will be displayed on the bottom. Only the additional services that are available based on your weight, destination, and service are displayed. You do not need to select any additional service. When you are done, click Done.

The following is a list of some of things that worked well in this project.

  • The task model was going through constant changes and revisions. It was fairly simple to modify the interface to the application by modifying the task model. An example that was made very late in development, a point when changes to an application flow are often risky, changed the model under SelectServices. I needed to always get the possible Additional Services, even if none are to be selected. Here are the before and after of a portion of the model.
  • Much of the knowledge of the application has removed from the user interface. Reflection on the name of the goal of the current focus was used to determine which pane is to be currently displayed. Furthermore, the live status of three of the four high-level goals affected whether the pane for that goal was enabled and whether the button for the goal on the main pane was enabled. For the last of the top level goals, CompleteTransaction, the Next button would become the Done button if this goal was live. This may not work in all situations, but it worked for this model. The CompleteTransaction goal only becomes live once a primary service has been selected. At that point, you should not want to go back and change the weight or destination. The Services pane has everything on it that the user would need to do to complete the transaction. So it made sense in this case to turn the Next button into the Done button.
  • I was able to have three interfaces working simultaneously: Text, Speech, and Graphical. Any of the three can be used for input at any time, and the system will output to all 3 every time. However, running all three of these interfaces did sometimes cause some issues, which are noted below. The application can also be run with only Text, Speech and Text, or with Text and Graphical. I created an Interface that each of these interfaces needed to implement. The interface defined three methods: println, prompt, and deallocate. The println method is used in the Text UI to simply print the line of text, and in the Speech UI it would be spoken. In the Graphical UI the line is displayed in a status bar at the bottom. The deallocate method is used to deallocate anything in the UI before it gets shutdown. Lastly is the most interesting method, prompt. In the Text UI, the prompt method is used to indicate that the system is ready for further input. I extended this concept into the other UIs. In the Speech UI, I figured a prompt should mean that the microphone should be prepared to receive more input (Note: this has not been thoroughly tested). In the Graphical UI, I used the prompt to indicate that the task engine was done and that it could then be queried to determine the new state of the world. It is at this point where the GUI determines which states are live, as mentioned above.
  • I think including the Next button makes the interface easier to use. The first time a user sees the interface, even though the interface is fairly simple it might not be clear what to do next. Ideally, the user should be able to keep clicking the Next button and walk through a flow of the interface. Unfortunately, I think this does not always work perfectly.
  • Whether it was the right thing or not, I set up a set of variables in the task model to store the current values of the transaction. Originally, I wanted to have the top-level goal contain all of these values, and other necessary values would get passed from goal to goal. However, I wanted to update GUI with the most recent values of these variables, because they could have been updated through another interface. Instead of searching through the goals or getting the top-level goal, it was much quicker and easier to just access the global variables.

The following is a list of some of things that did not work as well.

  • Constantly looking up slot values, especially predefined values, caused significant performance issues. I decided to cache the “when” slot because it was getting referenced so often. After some debugging I think I have successfully been able to manage the caching of this value, but it still has a potential for error. Caching of this slot value has shown a noticeable increase in performance.
  • There is a bug associated to any text entry that is used in GUI. I believe it is associated to how I have implemented the three parallel interfaces. I think the Text interface is blocking on further input (ReadLine). For some reason, this makes text keyed into the GUI to not be immediately recognized. If the user hits enter in the Text interface, a new prompt is written in the Text interface and the text entry in the GUI now works.
  • There is still too much application logic in the GUI. Ideally, this logic would be located somewhere in between the GUI and the Task Engine. This logic may be embodied in one or more properties files or simply in code.
  • Many of the steps should be able to be redone as many times as necessary. Performance issues with MaxOccurs > 1. The Next function also does not always work well with this. Perhaps instead of creating a task instance for all the optional tasks it would create only one at a time. Once that task has been performed, then it would determine if it needs to create another optional task. This may help with the performance issues and be able to handle an infinite MaxOccurs.
  • I was very frustrated that input slots needed to be declared even for slots that were already defined. I found it unnecessary to have to read the current values of the slots only to feed them back into it. I would have thought that the only input slots that would need to be declared would be those that were not bound.

The following is a list of some of things that might work some time in the future.

  • As mentioned above, there is still too much application logic in the GUI. I would like to add a layer in between the GUI and the task engine. This layer would be responsible for interpreting the model and causing the resulting changes in the GUI. I would like to make the GUI have more abstract controls. The idea is that I should be able to replace the GUI with another one that exhibits the same controls. My thinking is that the Java GUI could be replaced with a Flash or AJAX GUI.
  • It would be a nice to have a “Why” feature. For the user that is lost, the user could click the Why button which would tell him about what is being displayed. And since what is being displayed is based on the task in focus, we could nicely leverage the task model. As has been seen in other applications, if the user continues to ask why we navigate up the tree explaining each task.
  • The usability of the interface is greatly hindered by its horrible performance. It appears that there might be some performance issues with constantly referencing values to slots that are only available in the JavaScript. This is the reason why I added in the caching of the when slot of each task. It might be reasonable to cache each of the predefined slots since they are referenced so often by the task engine. Constantly looking up slot values, especially predefined values, caused significant performance issues. I decided to cache the “when” slot because it was getting referenced so often. After some debugging I think I have successfully been able to manage the caching of this value, but it still has a potential for error. Caching of this slot value has shown a noticeable increase in performance.
  • I modified the task engine to compile the status to a string instead of immediately writing it to System.out. I did this so I could then pass the status to each of the UIs. However, the status as a long string is not very usable by most of the UIs. I would like to have it in XML format, and then allow each of the UIs to display this XML in whatever form that is approriate for that UI. The Speech UI may ignore it or only speak th name of the task that is currently in focus. The GUI may display it in a tree control.