From: Ben Santer
To: Carl Mears
Subject: Re: Our d3* test
Date: Mon, 02 Jun 2008 09:32:01 -0700
Reply-to: santer1@llnl.gov
Cc: Steven Sherwood , "Thorne, Peter" , Leopold Haimberger , Karl Taylor , Tom Wigley , John Lanzante , "'Susan Solomon'" , Melissa Free , peter gleckler , "'Philip D. Jones'" , Thomas R Karl , Steve Klein , carl mears , Doug Nychka , Gavin Schmidt , Frank Wentz
Dear Carl,
This issue is now covered in the version of the manuscript that I sent
out on Friday. The d2* and d3* statistics have been removed. The new d1*
statistic DOES involve the standard error of the model average trend in
the denominator (together with the adjusted standard error of the
observed trend; see equation 12 in revised manuscript). The slight irony
here is that the new d1* statistic essentially reduces to the old d1*
statistic, since the adjusted standard error of the observed trend is
substantially larger than the standard error of the model average trend...
With best regards,
Ben
Carl Mears wrote:
> Hi
>
> I think I agree (partly, anyway) with Steve S.
>
> I think that d3* partly double counts the uncertainty.
>
> Here is my thinking that leads me to this:
>
> Assume we have a "perfect model". A perfect model means in this context
> 1. Correct sensitivities to all forcing terms
> 2. Forcing terms are all correct
> 3. Spatial temporal structure of internal variability is correct.
>
> In other words, the model output has exactly the correct "underlying"
> trend, but
> different realizations of internal variability and this variability has
> the right
> structure.
>
> We now run the model a bunch of times and compute the trend in each case.
> The spread in the trends is completely due to internal variability.
>
> We compare this to the "perfect" real world trend, which also has
> uncertainty due
> to internal variability (but nothing else).
>
> To me either one of the following is fair:
>
> 1. We test whether the observed trend is inside the distribution of
> model trends. The uncertainty in the
> observed trend is already taken care of by the spread in modeled trends,
> since the representation of
> internal uncertainty is accurate.
>
> 2. We test whether the observed trend is equal to the mean model trend,
> within uncertainty. Uncertainty here is
> the uncertainty in the observed trend s{b{o}}, combined with the
> uncertainty in the mean model trend (SE{b{m}}.
>
> If we use d3*, I think we are doing both these at once, and thus double
> counting the internal variability
> uncertainty. Option 2 is what Steve S is advocating, and is close to
> d1*, since SE{b{m}} is so small.
> Option 1 is d2*.
>
> Of course the problem is that our models are not perfect, and a
> substantial portion of the spread in
> model trends is probably due to differences in sensitivity and forcing,
> and the representation
> of internal variability can be wrong. I don't know how to separate the
> model trend distribution into
> a "random" and "deterministic" part. I think d1* and d2* above get at
> the problem from 2 different angles,
> while d3* double counts the internal variability part of the
> uncertainty. So it is not surprising that we
> get some funny results for synthetic data, which only have this kind of
> uncertainty.
>
> Comments?
>
> -Carl
>
>
>
>
> On May 29, 2008, at 5:36 AM, Steven Sherwood wrote:
>
>>
>> On May 28, 2008, at 11:46 PM, Ben Santer wrote:
>>>
>>> Recall that our current version of d3* is defined as follows:
>>>
>>> d3* = ( b{o} - <**> ) / sqrt[ (s{****} ** 2) + ( s{b{o}} ** 2) ]
>>>
>>> where
>>>
>>> b{o} = Observed trend
>>> <****> = Model average trend
>>> s{****} = Inter-model standard deviation of ensemble-mean trends
>>> s{b{o}} = Standard error of the observed trend (adjusted for
>>> autocorrelation effects)
>>
>> Shouldn't the first term under sqrt be the standard deviation of the
>> estimate of <****> -- e.g., the standard error of **** -- rather
>> than the standard deviation of ****? d3* would I think then be
>> equivalent to a z-score, relevant to the null hypothesis that models
>> on average get the trend right. As written, I think the distribution
>> of d3* will have less than unity variance under this hypothesis.
>>
>> SS
>>
>>
>> -----
>> Steven Sherwood
>> Steven.Sherwood@yale.edu
>> Yale University ph: 203
>> 432-3167
>> P. O. Box 208109 fax: 203
>> 432-3134
>> New Haven, CT 06520-8109
>> http://www.geology.yale.edu/~sherwood
>>
>>
>>
>>
>>
>>
>
--
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel: (925) 422-2486
FAX: (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------
**