Review Comment:
This paper is based on previous research that models human navigation behavior in information space through decentralized search. It extend this approach by introducing external ontologies (in this case biomedical ontologies) as the background knowledge. A small case study based on human subjects testing is conducted to compare the similarity in navigation behavior between simulated navigation in a Wikipedia network and observed user navigation. The statistical analysis is thoroughly conducted along several evaluation metrics, and the ontology based navigation simulation outperforms random navigation, i.e. behaves more similar to human navigation behavior in information system.
Nevertheless, I feel that there are some serious conceptual issues in the design of the study, and that there is some limitation to the usability and relevance of the results.
My major concern relates to the user task in the study. The scenario assumes that a user cannot exactly remember the target term (e.g. a disease), and would therefore start the navigation at a more general Wikipedia article. However, as the first attempt of the study showed, with a search term not understood (i.e. their semantics and position in an ontology unknown), local information provided through text information on links of a Web site is not helpful for navigating towards the target site. If, however, a user remembers or understands a target term, then I would assume that the user simply searches for this term (either in Wikipedia or a search engine), so get pointed towards that Web site directly, and would not go through the hassle of hopping from page to page. Thus, the task to navigate towards a single target page is hardly used in practice, at least when the exact search term, like in the conducted study (Pneumonia, etc.), is known. So this task poses a Catch-22 problem: A user would have to navigate first to an unknown target to learn about it (i.e. its name). This is however, only possible, if he/she already knows where in the ontology this term is located (or at least how it is spelled), which would make the navigation unnecessary. Yes, one could probably simulate different user groups in medical information systems, as the authors state, but would anyone apply such a search in real world situations? This needs to be clarified.
The navigation algorithm is (too) simple.
a) For example, humans learn to some degree when they navigate through navigation networks (i.e., they update their ontology/mental map), but this component is apparently not present in the simulation model, at least I could not see it. Yes, there is backtracking, but, new links are not being added or removed between objects in the ontology, if corresponding information is discovered during the navigation event. If there is a good reason for not including a learning component this should be explained.
b) Also, right now the decision rule is deterministic. Human search and decision behavior has a stochastic component. This should be addressed.
Also, the model is currently limited to one application ontology. This may work if navigating towards a target with a simple concept, like a disease, but not for information spaces (e.g. the Web) in general, e.g. when the target node (i.e. a Web site) needs to satisfy several constraints (e.g. through attribute values). An example could be a search for a ski resort within 200km from Graz which has at least 20 km of ski slopes and can be reached by public transportation. Navigation towards such a Web site, if possible, would involve several ontologies. If this cannot be handled with the provided approach, this is limiting. It has not been explained, which kind of relations can be found in the sample ontologies (ICD-10, etc.), and whether all kinds can be used in the navigation algorithm. It looks like ICD-10 is strictly hierarchical with IS-A relationships, but oftentimes there may also be properties of objects involved, which could be used as background knowledge. Please clarify. In fig. 1 I see only nouns, but there may be more involved than this.
Some minor comments:
- fig. 5 right column: Where do the curves for random walks come from (dashed lines)? I would assume that they only occur in a simulation. A user would not purposely perform a random walk. Please clarify.
- You bring up a lot of advantages of the ontology-based approach on page 8, but do not implement, illustrate, or test any of them, e.g. how to extract different types of relations from an ontology for background knowledge, and how that would be used to simulate navigation behavior. More detail or an example would strengthen the paper.
- http://wikipediamaze.com/ did not work when I tested it (server error)
- Please check the grammar, e.g. “This paper extends this application by a using ontologies…”
All in all this paper is a logical extension of previous work, but I cannot yet see its relevance towards a better understanding of human navigation in information spaces.
|