FRBR-ML: A FRBR-based Framework for Semantic Interoperability
Review 1 by Kate Byrne
In my opinion the paper has been greatly improved and I congratulate the authors. It now seems entirely suitable for publication.
This is a revised resubmission, after an "accept with major revisions".
Review 1 by Kate Byrne
This is a very detailed paper with some interesting results, but I felt that the content could have been expressed more succinctly, in fewer than the 24 pages used. More tailoring to the likely interests of readers would be welcome, and some restructuring would make the ideas easier to follow.
The research concerns bibliographic data, in MARC format, and the experimental results deal with a use case in which MARC records are "round-tripped", ie converted to a format based on FRBR, cleaned and enhanced with new information, and then converted back to MARC. My concerns with the paper are that the target audience is a little unclear and that the quite complex structure of the paper is in many places back to front: unfamiliar ideas are mentioned in early sections and not explained until further on. This makes the paper quite hard going to read, until one reaches the final two sections when things become much clearer. I think the authors could do themselves a service by restructuring the content.
Regarding the audience for the paper: as it stands, this work is clearly most likely to be of interest to librarians managing bibliographic catalogues where enhancing data quality is an important issue. But the emphasis on formal, set-theoretic notation may be unappealing to this target readership. Also, it's not clear that a technique that seems heavily based on straight sting "string equivalence" matching will advance the present state of the art. If a wider semantic web research audience is intended, it is a pity that the "semantic interoperability" promised in the title becomes rather lost amongst the details of the methodology used.
There are a number of places where terminology and concepts that will only be accessible to specialists could usefully be explained further - or rather, where the position of the explanation could be brought forward to earlier in the paper. In several places practical examples would help to illustrate meaning. The MARC format is introduced in section 2.2 and a sample record would help readers unfamiliar with its layout; perhaps just a reference forwards to Figure 3. The term "added entries" is used several times from section 2.3 onwards, but not explained until section 4 (page 9). Section 3.1 is particularly involved: a dense comparison of MARC with FRBR that would be much easier to follow through a worked example. The abbreviation "LOD" (not spelled out) is introduced ahead of its explanatory gloss further on in the text (section 6.2). The intricate descriptions in sections 4, 5 and 6 become much clearer when one reaches section 8, so some re-ordering would help.
Section 2 gives useful background about bibliographic formats, though I'd have expected some reference to W3C initiatives and the OMG (Open Metadata Group) work on ISBD XML schemas, and IFLA work on FRBR definition using SKOS. (I'm not a specialist in the bibliographic field so my grasp of which developments are significant may be flawed.) The diagram in section 2 (Fig 1) is very helpful, though I wondered why the arrows were double-headed rather than directed, eg "is created by" is not a symmetric relation (nor of course is its inverse). The entity terms introduced here - Work, Expression, Manifestation etc - should be italicised or given a special font throughout the text to avoid confusion.
I felt the information given in sections 2 and 3 could be abbreviated (likewise section 5), especially if the paper is intended to appeal beyond the library world. It was a bit dispiriting to reach page 8 before getting to a heading (section 4) of "Preliminaries". It's a small point, but one or two sections could do with more expressive headings, to help navigation; I found myself referring back and forth a lot, and smiling wryly to find what I needed at one point in the rather complicated section entitled "Simplicity and Understandability".
Personally I didn't find the formal notation introduced in section 4 helpful - it slowed me down rather than the opposite. At best it just restates what can be expressed more simply in natural language, and in some places it adds confusion. For example, I was unclear whether Cdiamond and Ddiamond (I only have ASCII characters available here) are genuinely subsets of C and D as stated on page 10, or whether the sub/super-set arrangement can be either way round as stated earlier. In each case (MARC, FRBRizer, FRBR-ML) the relationships between R, C, D, S etc are identical, so it didn't seem to me that useful information was being summarised, which is the value of formal notation. The interesting mappings, that are the core of the paper, are *between* these separate formats, ie r to rdiamond, C to Cdiamond etc, but these mappings are not given. The "map_d()" function (as opposed to "map_s()") is defined as taking only a MARC datafield tag label (eg "100" or "240") as argument, and then shown taking a tag-value pair (page 12). It is shown as evaluating to "mu", which is not defined in Table 1. To take just one example amongst many that seemed obfuscatory rather than helpful, the statement "In FRBR-ML, a FRBR entity f* in F* is related to another one with a relationship l* in L* such that l*: f* x f*." could be expressed as "FRBR entities can be linked by relationships". The link at footnote 8 on page 12 - to the FRBR-ML schema - seems to be broken.
The distinction between the "hierarchical" and "reference" methods in section 5 does not seem important enough to justify the effort around it. The algorithm for deciding which format to use is nicely set out but is this step really necessary? Why not simply use the "referencing" method throughout and drop the complication of "hybrid representation"?
The most important part of the paper is introduced in section 6, namely the attempt to enhance or correct input records by matching entity mentions against other records in the input dataset or other library catalogues, or against external sources such as DBpedia and Freebase. This final step is the contribution of this work, to my mind. The steps are to find entities in the input data, categorise them (as "person", say) and then attempt to deduce a relation to another entity in the input. The use case describes transforming an unspecified relation between "Hans-Joachim Maass" and a bibliographic record into a more specific relation: an "is realised by" connection from the Expression entity. In fact Hans-Joachim is the translator, but this relation cannot be explicitly expressed in the original MARC, nor in FRBR-ML. However FRBR-ML is able to insert the missing relation with Expression, and to specify which translator goes with which instance of Expression.
A set of metrics is given in section 7. The "completeness" measure calculates how different the input and output fields are - so one could trivially get a perfect score by a null transformation of the input. Section 7.1 uses MARC control fields as an example but it's not clear why these would be changed, and the paper does not describe control field mapping. Once again the formal notation fails to convey additional information; for example, the comp_f() function defined on page 18 shows that input and output lines of a MARC record (tags, subcodes and values) are going to be compared, but doesn't tell us how - I assume a string comparison, perhaps implemented through the hashing mentioned earlier in the paper. The "redundancy" measure considers the amount of duplication present (exact string equivalence again) - but surely from a practical point of view such duplication should simply be eliminated. Why bother with an intermediate step of measuring it? The third metric is "extension" and, as specified in the expressions in section 7.3, it is simply a measure of *how much* has been added - so one could trivially get an arbitrarily high score by adding characters in the output. Surely the only important evaluation measure is on the *semantic* content - whether what has been added is correct.
Section 8 is the best part of the paper, where a good deal of the earlier material falls into place, and where a genuinely useful evaluation is described. It's only on page 20 that we reach commentary on whether the enrichment process has produced better bibliographic records or not. Since this evaluation, by 8 human judges, seems the core result I would have welcomed more detail on the process. Measuring precision is feasible using experts (ie is this data correct?) but measuring recall (is any information missing?) is notoriously difficult in open-ended tasks of this nature, and the precise method used would be of interest. The remark "...presented the top three candidate matches...including a manual search on the knowledge bases for the entry value when needed..." is particularly intriguing. I would have preferred numbers in Figure 9 instead of a bar chart, but the scores seem remarkably high for what is a difficult knowledge enrichment problem. I would encourage the authors to restructure the paper around these results and drop some of the methodological details (that I've spent far too long on myself, in this review). Right at the end of section 9 there is a hint that the authors plan to move on to a wider interpretation of co-reference resolution and grounding against external authorities, using pattern matching that goes beyond exact equality, which indeed seems the obvious next step.
I noted a number of minor typographical and similar errors, and can supply a list if required.
Review 2 by Sarantos Kapidakis
This is a solid paper, that presents how to handle FRBR and MARC, in order to add semantic information to the FRBRized data. The FRBRized data, with the extended information, can also be converted back to MARC (for the legacy systems), or be used in other XML based formats (for more modern applications).
The described system can communicate with other sources of additional information, and is using well trimmed heuristics to extract the appropriate information when it is needed.
The authors provide a formal model,they explain their data layout,
they make an experimental evaluation on collections of the Norwegian National Library, and they also estimate evaluation metrics.
Review 3 by Ray Larson
Bibliographic Records specifications. Overall, this is a very well-written and informative paper about the issues and potential for transforming conventional MARC bibliographic data to FRBR-compliance XML using the
FRBR-ML framework. I found a few minor errors in grammar or spelling,
but otherwise most of my comments are concerning the structure of
the FRBR-ML records themselves.
First the grammatical issues - In section 2.1 "that constitutes a …"
should be "that constitute a …"
later in the same section "use of punctuations and separators characters"
should be "use of punctuation and separator characters"
and "that are users need to search" should be something like "that are needed by users for searching"
My major issue with the paper is more of a question about whether the
"hierarchical Method" described sort of turns the whole notion of FRBR
upside down. One of the characteristics of FRBR that was most revolutionary was the adoption of the Work instead of the manifestation as the primary unit of organization. But in the hierarchical method you have made
the manifestation the top-level element with the work nested within. This is naturally a better fit to the philosophy of the source MARC data, but
I would argue that it sacrifices the benefits of the FRBR stucture
(such as grouping all expressions and manifestations of a work under
that work, instead of vice-versa). Although the paper points out that
there are drawbacks to this method (and to the referencing method as
well) I think that could be made a little more forcefully.
But, overall this is an interesting and useful paper about FRBR and its instantiation in FRBR-ML.