[Ngasp-help] Gunnar Raetsch's comments

[Tristan Fiedler]

---------------------------- Original Message ----------------------------
Subject: Re: nGASP manuscript
From:    "Gunnar Raetsch" <Gunnar.Raetsch at tuebingen.mpg.de>
Date:    Sat, July 26, 2008 12:30 am
To:      "Dr. Tristan J. Fiedler" <fiedler at fit.edu>
Cc:      "Alexander Zien" <Alexander.Zien at tuebingen.mpg.de>
         "Gabriele Schweikert" <Gabriele.Schweikert at tuebingen.mpg.de>
--------------------------------------------------------------------------

Hi Tristan,

the worm genomics meeting in Cambridge kept me busy most of the day. I
finally came to read the manuscript.
I think it is already in pretty good shape. I have a few comments,
which you may consider before submitting the manuscript:

* I missed a more detailed discussion of the annotation dataset that
has been used for evaluation. There are a few issues:
	* For unconfirmed genes, how where they predicted? Did any of the
compared methods produce these predictions?
	* Also, for the EST-confirmed genes: these gene models where
generated using EST alignments, some manual curation and probably also
gene finder predictions. Except the manual curation, the input is the
same as for cat-3 gene finding. Who knows what the correct gene models
are...
    (or something in this direction, you probably have your own
thoughts on this).

* The evaluation for the combiners was done on only the 3' end of the
regions. We noticed there is a significant difference for in the
performance between the 5' and 3' ends (several percent in transcript
level). Hence, the performance on the 3' end and the performance on
the whole region are not directly comparable. I don't necessarily want
you to redo the evaluation, but at least this should be noted in the
main text and also that combiners had about 50% more data for training.

* Since the result of the paper is that combiners are the method of
choice, it would be very interesting to understand how important the
accuracy and the choice of the base-gene finders are. It would be
really great if you could show some results, indicating which
(sub-)set of (the three) gene finders lead to which performance.

* I really would like the evaluation separately for each category.
This would allow us the compare the contributions within the
categories more easily. It would be very nice to have, at least in the
supplement.

* One feature that came to our mind that you did not check whether it
leads to wrong predictions, is whether there is a gene on the opposite
strand. Many gene finders only predict genes on one strand. mGene, for
instance, does not.

* It would be a great service to the gene finding community and also
would facilitate reproducibility of this research, if you'd provide
the evaluation scripts which lead to exactly the numbers given in the
table in the supplement of this paper. (In any case I'd like to get
them to evaluate new predictions that we have in the same way.)

Finally, I'd like to ask you to reconsider the choice of the Journal.
I think it would have a good chance in PLoS Comp Bio or Genome
Biology, which I find considerably better suited for this work than
BMC Bioinformatics. Why not trying it?

Thank you and the others for writing the manuscript and organising
this competition!

All the best,

Gunnar


On 16.07.2008, at 19:26, Dr. Tristan J. Fiedler wrote:

Dear nGASP Participants,

We thank you again for your participation in nGASP.

The nGASP analysis team has now written up the results of nGASP
as a paper, which we plan to submit to BMC Bioinformatics.

As agreed, we are sending you a copy of the draft manuscript
for your perusal before submission.

We would be very grateful if you can let us know if you have any
major comments on the draft manuscript by Thursday 24th July.

Comments may be sent to ngasp-help at wormbase.org

Yours sincerely,

The nGASP analysis team.

<ngasp_16jul08b_alc.doc>

+-------------------------------------------------------------------+
Gunnar Rätsch                         http://www.fml.mpg.de/raetsch
Friedrich Miescher Laboratory       Gunnar.Raetsch at tuebingen.mpg.de
Max Planck Society                          Tel: (+49) 7071 601 820
Spemannstraße 39, 72076 Tübingen, Germany   Fax: (+49) 7071 601 801



On 16.07.2008, at 19:26, Dr. Tristan J. Fiedler wrote:

> Dear nGASP Participants,
>
> We thank you again for your participation in nGASP.
>
> The nGASP analysis team has now written up the results of nGASP
> as a paper, which we plan to submit to BMC Bioinformatics.
>
> As agreed, we are sending you a copy of the draft manuscript
> for your perusal before submission.
>
> We would be very grateful if you can let us know if you have any
> major comments on the draft manuscript by Thursday 24th July.
>
> Comments may be sent to ngasp-help at wormbase.org
>
> Yours sincerely,
>
> The nGASP analysis team.
>
> <ngasp_16jul08b_alc.doc>

+-------------------------------------------------------------------+
Gunnar Rätsch                         http://www.fml.mpg.de/raetsch
Friedrich Miescher Laboratory       Gunnar.Raetsch at tuebingen.mpg.de
Max Planck Society                          Tel: (+49) 7071 601 820
Spemannstraße 39, 72076 Tübingen, Germany   Fax: (+49) 7071 601 801




+-------------------------------------------------------------------+
Gunnar Rätsch                         http://www.fml.mpg.de/raetsch
Friedrich Miescher Laboratory       Gunnar.Raetsch at tuebingen.mpg.de
Max Planck Society                          Tel: (+49) 7071 601 820
Spemannstraße 39, 72076 Tübingen, Germany   Fax: (+49) 7071 601 801




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://brie4.cshl.edu/pipermail/ngasp-help/attachments/20080726/80c04638/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 185 bytes
Desc: not available
URL: <http://brie4.cshl.edu/pipermail/ngasp-help/attachments/20080726/80c04638/attachment.sig>