Today, I have sent the peer review instructions to the solution submitters that uploaded their contribution before the original deadline.
Here is a copy of the instructions:
Dear [*FIRST-NAME*],
thanks for your submission to TTC 2011.
As indicated in the call for solutions, the first phase of the solution reviewing consists of a peer review. The idea is that this kick-starts the knowledge transfer that we aim at with the TTC workshop.
We have various requests for the two week deadline extension but since you are one of the submitters of the original deadline, we want to reward you by sticking to the original, more relaxed, schedule.
Please find below the proposed assignment for the public peer review. Please use the TTC fora for discussion on the submissions and don't hesitate to challenge each other's statements from the papers. Keep in mind also that you should be able to access each other's software in SHARE.
ASSIGNMENT for HELLO WORLD:
* GRETL (submitter Tassilo Horn) => reviewed by Kevin Lano,Amir Hossein Ghamarian
* UML-RSDS (submitters Kevin Lano, Shekoufeh Kolahdouz Rahimi) => reviewed by Steffen Mazanek, Amir Hossein Ghamarian, Pieter Van Gorp
* GROOVE (solution authors Amir Hossein Ghamarian, Maarten De Mol, Arend Rensink and Eduardo Zambon) => reviewed by Steffen Mazanek, Kevin Lano, Pieter Van Gorp
* MDELab (solution authors Stephan Hildebrandt, Sebastian Wätzoldt, and Holger Giese) => reviewed by Tassilo Horn, Maarten De Mol (?), Pieter Van Gorp
* Henshin (solution authors Stefan Jurack and Johannes Tietje) => reviewed by Arend Rensink (?), Sebastian Wätzold (?) and Pieter Van Gorp
(Also: Pieter Van Gorp will review more Hello World submissions, as they come in after the extended deadline.)
ASSIGNMENT for PROGRAM UNDERSTANDING
* GRETL (submitter Tassilo Horn) => reviewed by Stephan Hildebrandt, Stefan Jurack
* MDELab (submitters Stephan Hildebrandt, Sebastian Wätzoldt and Holger Giese) => reviewed by Edgar Jakumeit, Tassilo Horn
* Henshin (submitters Stefan Jurack and Johannes Tietje) => reviewed by Edgar Jakumeit and Stephan Hildebrandt
* GrGen.NET (submitters Edgar Jakumeit and Sebastian Buchwald) => reviewed by Tassilo Horn and Stefan Jurack
ASSIGNMENT for COMPILER OPTIMIZATION
* GROOVE (submitters Arend Rensink and Eduardo Zambon) => reviewed by Tassilo Horn, Sebastian Buchwald
* GRETL (submitter Tassilo Horn) => reviewed by Arend Rensink, Sebastian Buchwald
Please submit your final reviews no later than May 17th to the online forum
(http://planet-research20.org/ttc2011/index.php?
option=com_community&view=groups&Itemid=141). We will then consider this input with the organizing committee and (if necessary) the steering committee and send out the notifications on May 19th. We may contact you again if we decide to use another review platform than the TTC forum, and for now please focus on the review content rather than the form.
Please use the following reviewing criteria:
1) *** SUBMISSION TITLE:
2) *** REVIEWER NAME:
3) *** OVERALL EVALUATION:
--- 3 (strong accept)
--- 2 (accept)
--- 1 (weak accept)
--- 0 (borderline paper)
--- -1 (weak reject)
--- -2 (reject)
--- -3 (strong reject)
4) *** REVIEWER'S CONFIDENCE:
--- 4 (expert)
--- 3 (high)
--- 2 (medium)
--- 1 (low)
--- 0 (null)
5) *** SOLUTION REPRODUCIBILITY: from 1 (lowest)
to 4 (highest)
--- 4 (Paper (with code listing) + SHARE demo available)
--- 3 (Paper (without code listing) + SHARE demo available)
--- 2 (Paper (with code listing), no SHARE demo)
--- 1 (Paper (without code listing), no SHARE demo)
6) *** PAPER READABILITY: from 1 (lowest) to 5 (highest)
--- 5 (excellent)
--- 4 (good)
--- 3 (fair)
--- 2 (poor)
--- 1 (very poor)
7) *** CASE SPECIFIC SCORE FOR CORE SOLUTION (E.G. CORRECTNESS, CONCISENESS,
CLARITY): from 1 (lowest) to 5 (highest)
--- 5 (excellent)
--- 4 (good)
--- 3 (fair)
--- 2 (poor)
--- 1 (very poor)
8) *** CASE SPECIFIC CORE FOR EXTRAS: from 1 (lowest) to 5 (highest)
--- 5 (excellent)
--- 4 (good)
--- 3 (fair)
--- 2 (poor)
--- 1 (very poor)
9) *** FAIR DESCRIPTION OF DISADVANTAGES OF SOLUTION: from 1 (lowest) to 5
(highest)
--- 5 (excellent)
--- 4 (good)
--- 3 (fair)
--- 2 (poor)
--- 1 (very poor)
10) *** OTHER REMARKS
--- Motivation of the above scores in free text...
Originally, for criterion (5) we had "appendix" instead of "code listing". This has triggered Tassilo Horn to me with the following good questions:
I have a question regarding the peer review reproducibility criteria:
That distinguishes between with and without appendix. What does that mean?
For example, Arend and Eduardo (GROOVE) don't have an appendix in their
paper, but a SHARE demo with a readme file. All in all, I was able to
run their solution without hassle, so I'd say that's a 4 (highest).
So I'm tempted to understand "appendix" as "has readme/screencast and
the full transformation code is somehow accessible", right?
Bye,
Tassilo
PS: Yes, that's a question that should have been asked in the forums.
However, it's not depending on the case and there's no general TTC
forum. So you might want to blog my question and your answer so that
the others can read it.
Here is my reply (the blog post has also been updated in the meanwhile):
With regards to "5) *** SOLUTION REPRODUCIBILITY", please do not focus on whether or not there is actually an appendix. Instead, check whether there is a (complete) "code listing". This makes a difference, as many submissions turn out to have the listing in the body of the paper, which is clearly as reproducible as having a separate appendix.
. The ZIP archive is available at http://is.ieis.tu...