Personal Learning Environments on the Social Semantic Web
Review 1 by Eleni Kaldoudi
I would advise that the paper is accepted for publication as is.
The revised paper addresses all my earlier comments in a very satisfactory way(and as far as I can tell, it addresses other reviewers' comments as well), it is much improved, it reads really well and I think that it will be a good contribution to the journal and the specific scientific field.
Review 2 by Ivana Marenzi
The authors have improved the paper according to the given suggestions.
Review 3 by Fridolin Wild
I agree with the changes applied. Congratulations, good work.
The reviews below are form a previous version of the manuscript.
Review 1 by Eleni Kaldoudi
The paper presents a very comprehensive analysis of current trends in personal learning environments and describes the architecture, the implementation and preliminary evaluation of a novel social semantic web based tooly (DEPTHS) that creates an interactive personal learning environment for the domain of software design patterns education.
The paper includes a comprehensive review on current state in personal learning environments (PLEs) and in social semantic web applications as applied to education. This review is thorough, well organized and especially well balanced to report on principles, functionalities and technology issues of reviewed systems, tools and applications. The authors make a worth to note effort to present their survey in a critical and informative way, and the section on identified "Principles" for the design of PLEs is indeed novel, interesting and most useful. Finally, the paper presents the authors novel work towards accommodating all these principles in the design of the DEPTHS personal learning system. This is a very interesting architecture merging social and semantic web principles and technologies. It is worth noting that the implementation described builds on a rich variety of available open source software, well exploiting strengths and advantages of each one of them. A case study in a real learning environment is also presented as a preliminary evaluation and proof-of-concept.
Overall the paper is well written, without grammar of syntax errors, while the reference list is extensive and comprehensive, and the figures & tables adequate and useful.
I would recommend that the paper is accepted for publication as is. A few minor comments and suggestions (mostly typos) for the authors to check at their discretion are the following:
page 1, 2nd paragraph of "Introduction": The phrase "Some even consider that further development of social networking tools and Web in general will lead to serious disruptions in education
[Christensen et al, 2008]" might be rephrased so as not to give the impression that the cited book indeed proposes that web technologies will lead to disruptions. Rather that these technologies comprise much needed disruptive innovations. (The cited book uses the provocative word "disruption" in a positive sense, and it actually proposes that such "disruption" is needed for innovation and reform in a current (USA) school system that only poorly addresses student's individual learning styles and interests. Moreover, it proposed that major advancements that will make this disruptive innovation work are the emerging ability for user-generated content and user networks.)
abstract, line 9: "illustrate them thorough the design" "should change to "illustrate them through the design"
page 2, paragraph 2: refs. [Muirhead, 2004] and [Caring, 2006] cited in the text do not appear in the reference list
page 3, section 2.2, 1st paragraph: in the phrase "the following3" the note number "3" should be formatted as superscript. The same with note numbers for the rest of the document.
page 3, last paragraph: ref [Wild et al, 2009] cited in the text, doest not appear in the reference list
page 4, section 2.3, bullet 2: "an an excellent bases" should change to "as an excellent basis"
page 5, section 2.3.2, paragraph 1: probably no need to repeat the ref for Linked Data principles
page 5, section 2.3.2, paragraph 1, line 10: "cannot be consider" should change to "cannot be considered"
page 9 & reference list: ref [Carmagnola et al, 2009] should become [Carmagnola et al, 2010]
page 13: although very clearly explained in the manuscript that PBL stands for Project-Based Learning, still I feel that a distinction should be made between Project-Based Learning and Problem-Based Learning (conventionally referred to as PBL for the last half century) – probably a quick reference/discussion about both?
page 14, section 4.2: learning context: would you consider (in general) the educational objectives and expected learning outcomes as factors influencing and defining context (along with these already suggested)?
page 15: it would read more clearly if terms such as "Repository of interaction data" etc are indicated with some different formatting (e.g. in italics, in quotes, all words in 1st letter Caps, etc)
page 21: The results of the evaluation on a group of 13 students probably should be presented in a more qualitative way, as the size of the group is rather small to deduce meaningful statistics
reference list: 2 different references are cited with the same code name [Jovanovic et al, 2007]
reference list: references that appear in the reference list, but are not cited in the text: [Bratsas et al, 2011], [Palmer et al, 2009], and [Torniai et al, 2008]
Review 2 by Matthew Rowe
This paper presents a comprehensive review of current approaches and tools to current Personal Learning Environments (PLEs) together with a list of well-motivated principles that such tools should adhere to. The authors go on to present their own system named 'DEPTHS' that was designed according to such principles and detail an evaluation of said tool within a learning environment setting.
Overall I felt that the paper was well written and comprehensive, particularly the review of related work and the set of principles - which were well described and motivated. However the latter part of the paper, that presents the author's tool/approach lacked in detail and described components in too brief a manner. As a result I was left wanting more details of the intricacies of the approach and how the authors dealt with data portability and integration - two of their principles. Additionally, I was expecting to see how the tool measured up against existing approaches, thus demonstrating the benefits/shortfalls of using the Social Semantic Web in a learning setting.
Comments per section:
Title: should be 'on the Social Semantic Web'
-Paragraph 3: the author's are describing the issue of data portability, they should refer to the work being performed within the Data Portability movement  to address the challenges of moving/exporting data from Social Web data silos.
-'Accordingly learning environment' -> 'Accordingly the learning environment'
-'Flicker' -> 'Flickr'
-'by allowing user to' -> 'by allowing users to'
-'quantity' -> 'quantities'
-§3.2: the authors state that the notion of data openness is entering the Social Web. I am not sure that this is the case, if anything the issue of openness is getting worse. Social Web platforms want to hold on to their data, in particular large ID providers such as Facebook, so they do not heed advice from organisations such as W3C, they simply do their own thing. I think that this statement needs to be toned down or simply removed.
-§3.1/3.2: again, related to the above point, data integration, although supported somewhat via semantic web technologies, is still an issue on Social Web platforms, given the provision of social data using bespoke schemas for each individual platform.
-The architectural description of DEPTHS omits several key details that I would like to see introduced into the paper, and thus make it easier for the reader to see the value and novelty of the approach, these are:
--How does the data mapping module function?
--How does identity management function?
--How does semantic annotation and indexing function?
-Also, the authors state that linked data is used. I conjecture that this is not the case, as it seems to me that RDF triples are produced and stored in a single repository. For linked data to be linked data links between resources must be traversed and related/associated information returned. At present there is no evidence for this.
-In the section describing how interactivity is improved, the authors state that content comes from social networks, while in the previous section the example social networks that are used are not described. This should be included.
-After reading the introduction of the paper I expected the evaluation section to demonstrate the comparative utility of the presented approach against a baseline system/method that does not utilise Semantic Web technologies or the Social Web. Although the questionnaires and their results provide some interesting insights, without a clear demonstration of the benefits of the approach I am left wondering how the participants would have fared without the system, or with another similar approach. The authors either need to perform another evaluation or tone down the claim in the abstract and introduction of improvement with their approach.
Review 3 by Fridolin Wild
The submitted article proposal investigates how social semantic web technologies and linked data can be used in personal learning environments (PLEs).
Following an introduction (1), the article therefore breaks the analysis of the state of the art down into three segments informed by literature (2.1-4) and an additional survey of existing tools (2.5). From this, a set of seven key requirements for the creation of PLEs with social semantic techniques and technologies are elaborated in Section 3. A case study on a new system (called DEPTHS) follows in Section 4, which consists of pedagogical approach (4.1), architecture (4.2), context-aware services (4.3), implementation of the architecture (4.4), and a review of how DEPTHS serves the requirements elaborated above (Section 5/5.1). Open challenges (5.2) and claims on interactivity improvements (5.3) follow, trailed by evaluation results of a small-scale trial with 13 participants (6). The article is wrapped up by a conclusion (7).
For me, the submission looks like its actually two articles combined into one, which makes it very difficult to review as a whole. The article – for me – has two parts: Part one is a survey of the state of the art of social semantic web technology in this application area of personal learning environments, which culminates in the elaboration of seven key requirements. This part has about 16 pages. The second part is the description of a novel system DEPTHS in which social semantic web technologies are applied in a new application area – learning technology – and which describes an implementation that brings these new technologies to the very popular Moodle environment. This part has about 11 pages.
Both parts fit the scope of the special issue very well and are novel and original. However, both parts have several shortcomings in my view that should be mentioned here and that need revision to make it an acceptable contribution.
The 'survey' part:
(1) Many urls in the introduction and in the subsequent footnotes make the text more difficult to read (and the references more difficult to find as they are not listed in the literature list, where they would be expected)
(2) Although the literature covered is exhaustive, you seemed to have missed an important one: how do the principles presented relate to the dimensions we worked out in
Sire, Bogdanov, Gillet, Palmer, Wild (2011): Introducing
qualitative dimensions to analyse the usefulness of Web 2.0
platforms as PLEs, In: International Journal of Technology
Enhanced Learning, Volume 3, Number 1, pp. 40 – 60.
This has been a proposal for judging the quality of web 2.0 platforms with respect to their usefulness as PLEs, which is arguably related.
(3) How you get from section 2 (literature and tools survey) to 3 (key requirements) is not expressed in the paper. Maybe you can put more explicit links to the state of the art into the description of the 'principles'? Or you might want to put some more state-of-the-art like paragraphs from the principles up into the section on the state of the art?
Maybe you could move the 'tool survey' part (the one about existing tools) into a separate section after the presentation of the principles, so as to illustrate the support for the principles with the help of the examples of these existing tools?
The 'system' part:
(1) the acronym 'DPs' is not defined when it first appears (only a few pages later)
(2) how the learning context is identified (e.g. assume: user is pursuing a specific 'task') and how this is then used to identify resources from one of the mentioned pattern repositories is not clear to me. Maybe you could make an example?
(3) Figure 1 is too small to read
(4) a system snapshot is missing
(5) do the pattern repositories provide linked data or are they always annotated (automatically)?
(6) Can you be more explicit how you link DEPTHS to the idea of PLEs, particularly how to interoperate it with other e.g. social software tools? I am not sure that DEPTHS is really a PLE. Isn't it rather an LMS with personalisation? You can't use ANY tool with it, can you?
(7) You could shorten the overall implementation description further, particularly references to existing modules in Moodle are likely to outdate quicker than the article is published.
(8) Table 2 does not follow the ordering of the principles in the first part of the article. And it uses different headings for the principles, which makes it very difficult to relate. If you follow my proposal to split the article in two, then I would anyway recommend to drastically shorten this section (together with the subsequent challenges) into a 'conclusion' or 'summary'.
(9) about the resource crawling: are the annotations provided as linked data again?
(10) The section on improving interactivity is full of unsupported claims and value statements (e.g. 'fosters extensive social interactions'). This is not appropriate.
(11) The evaluation study presented is not very convincing. It is very small (13 participants) and the evaluation was conducted in a course taught by the authors. A methodology part is missing. It is not clear how the mapping was done from the Likert-scale to the percentages (e.g. 84.62%). Several of the measured constructs can – in my view – not be measured with the help of introspection of the participants: participants cannot judge very well if they have learnt 'more'.
Either you extend this evaluation with the follow up trials you are likely to have conducted already. Or you leave them out completely. I am unsure whether an evaluation is a must for a SWJ article on systems, clarification by the editors would be helpful (I'd think it depends on the type of contribution made)!
(12) If I follow the discussion about system-for-paper versus paper-for-system (see comments for reviewers of the journal), I have the feeling that his part of the article proposal does not exactly know where it stands: is it documenting the social semantic add-ons to a highly successful system (Moodle)? If so, how successful are the add-ons already? Or is it documenting a new design pattern? If so, are there further supporting cases (rule of three) or why is this the only irrefutable solution? Also it would be more than good to have access to the tool.
Either way, this second part of the contribution needs in my view much more work than the first part to elaborate the essence of the contribution out of what now looks more like a documentation of what has been done.
Some further comments relating to both parts:
The introduction (and the latter relating to) the interactivity triangle seems a bit unmotivated and the literature referenced in 2.4 is not summarised. Also later, when you refer back to it, this could be improved.
The language needs a native speaker check. Particularly the use of articles (or rather: non-use thereof) is worth having checked! The first part has much more articles than the second (system description) part ;)
A minor one (but still): On page 4 you mention about the MUPPLE system I have created with colleagues, that the tools do not communicate or exchange data. This is not completely correct: we had foreseen the use of feeds and connect statements to set-up feed channels between tools (see e.g. Moedritscher & Wild (2009): Sharing Good Practice through Mash-Up Personal Learning Environments, ICWL'2009, Springer). Arguably, this is not interoperability as we would love to have it, particularly not from a semantic web perspective. Still: tools communicated (and with mediating services for providing the required FeedBack API it would even scale – at least somehow-ish).
Review 4 by Ivana Marenzi
This paper proposes a set of principles for designing PLEs based on both Social Semantic Web (SSW) paradigm and Linked Data paradigm, and presents the evaluation results of a specific PLE (DEPHTS)
The concept of interactive PLEs built on Social Semantic Web paradigm and Linked Data principles is original, as well as the integration of students´ interaction data and advanced forms of adaptation and personalization in e-learning.
Principle 4 (Context awareness) providing pedagogical support and proactive advises to learners is very interesting and appealing.
-Significance of the results (-)
The results described in section 6 refer to an evaluation carried out in February 2009. As it was already in the ISWC paper (2009), the discussion presents the results of the final questionnaire about the perceived usefulness of the system by 13 students, but specific examples of the support given by the system during the evaluation are missing.
-quality of writing
A clear structure of the contents and a good organization of the flow are given: references to the previous and following sections guide the reader through the paper.
The paper is well written and it presents a good introductory overview on current e-learning practices. It clearly discusses the motivation of using social networking tools and social software in education and it provides examples.
The English is plain and clear and the authors give clear definitions of specific terms such as Semantic Web – Social Web.
STATE OF THE ART
This section gives a good overview of the relevant research areas with a good balance between theory, methodology and practice. The reported references are supported by a few examples of systems for each category.
Table 1 in the Appendix also gives a summary of the State of the Practice (list of the tools and services that can be used in practice for educational purposes)
The section is well organized and the contents are relevant. In the discussion about openness and ubiquitous data access I missed a mention about privacy preservation and data ownership, which are briefly cited in the conclusion.
CASE STUDY (development of the DEPTHS PLE)
The authors give example scenarios at the beginning of the case study but they do not include pragmatic examples of the system in use.
The analysis mentions the challenges and difficulties faced during the implementation, but it is not clear how these affect the system in reality. What is possible and what is limited by these difficulties?
The analysis gives the results of the questionnaire but does not describe the activities on which the evaluation was based. The employment of context-aware educational services is very interesting and appealing, but no evidence of actual proactive advices is given.
The evaluation should give more pragmatic examples of the tasks and activities supported by the system to make clear to the reader what the system actually does when running: better describe the activities done by the students (what problems they had to solve and how they interacted), what resourced they found and what recommendations they got from the PLE.
References are a lot and recent. Please uniform the format as regards the page numbers.
I would suggest to add the following references as regards examples of PLEs.
1) Ivana Marenzi , Rita Kupetz , Wolfgang Nejdl , Sergej Zerr (2010). Supporting Active Learning in CLIL through Collaborative Search. In X. Luo et al. (Eds.): ICWL 2010, LNCS 6483, pp. 200-209. Springer, Heidelberg.
2) Bogdanov, Evgeny ; El Helou, Sandy; Gillet, Deni; Salzmann, Christophe et al. (2010). Graaasp: a web 2.0 research platform for contextual recommendation with aggregated data. Presented at: Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, Atlanta, Georgia, USA, 10-15 04 2010. New York, USA: ACM Press.
Fig 1 is not 100% clear (some parts are too small or bad resolution)
Please format the label of Table 2 correctly.