Archive for e-values

on stopping rules

Posted in Books, Statistics with tags , , , , , , , , , , , , , , , , , on August 3, 2025 by xi'an

The workshop in Chennai and its focus on sequential procedures made me realise (among other things) I had never read Cornfield’s 1966 TAS paper on sequential testing and the likelihood principle:

“By sequential analysis I mean any form of analysis in which the conclusion depends not only on the data, but also on the stopping rule.”

Written with little maths and formalism, this paper argues that keeping a fixed critical level amounts to keeping a fixed amount of evidence. Hence constituting an early critique of p-values even though not expressed in such terms. The part of the paper related with the likelihood principle does not address testing or evidence in a Bayesian way. As a side (late awakening) remark, iid observations in sequential settings are not longer independent, conditional on the stopping rule realisation N=n, since they are constrained by the fact that the stopping rule realisation is n and not n-1, n-2, …  For a short while, I thought it was in turn impacting the distribution of any “sufficient” statistic one may propose, with a normalising constant that depends on the unknown parameter and hence cannot be neglected. Over all those years, I had never though of the modification of sufficiency characteristics in such contexts. But in fine the pair made of the value of the stopping rule and of the unsequential sufficient statistics proves enough. And the normalisation constant is the probability that the stopping rule.. stops!, which is equal to one! For the same short while, I was then wondering that the stopping rule principle!

“my second line of argument that there is a reasonable alternative explication of the idea of inference and one which leads to the rejection of sequential analysis. This explication is provided by the likelihood principle—which states that all observations leading to the same likelihood function should lead to the same conclusion.”

I thus went back to the fundamentals (!), namely [freely available] Bernardo’s and Smith’s Section 5.1.4 (reproduced in EJ’s Stopping rule appendix, also citing Cornfield at length), where the likelihood is properly defined by the joint density of the stopping rule τ and the attached sample at their realised values. And failing in the end (and a discussion with Judith)nto spot a missing normalisation constant.

e-values in Chennai

Posted in Books, pictures, Running, Statistics, Travel, University life with tags , , , , , , , , , , , , , , , , , , , on July 23, 2025 by xi'an

To recap, I thus attended the BIRS-CMI workshop 25w5482 at the Chennai Mathematical Institute, Navalur, Tamil Nadu, in early July, for being intrigued by the developments around the concept. And enjoyed the week, from partaking in the company of friendly and enthusiastic academics to the exposure of new views and concepts, mostly remote from mine’s. Recall that an e-value attached to an hypothesis H described as a collection of distributions is a non-negative random variable E with expectation less than 1 for E~Q and all Q ∈ H. When a stopping rule is involved, the e-value is extended into an e-process. (Beyond Aaditya Ramdas’ E-book, Ruodu Wang also wrote a “tiny” review.) Aaditya Ramdas recalled in his introduction of the workshop that e-values are fundamentally equivalent to p-values and confidence intervals. And that a confidence sequence is a sequence of confidence intervals that contains the true value for all time steps t’s with a probability of at least 1-α.

The talks reflected a general belief in α levels and in Neyman-Pearsonian likelihood ratio optimality in simple vs simple settings, considering extension for sequential analysis settings, anytime inference, universality under general alternatives, and connections with FDRs, incl. Benjamini & Hochberg solution, but pointed out a lack of middle ground between frequentists and Bayesians.

“e-values have a clear interpretation in terms of betting and are closely related to likelihood ratios and other Bayes factor. At the same time, evalues do not require prior distributions conditional on the null and alternative hypotheses”

Although David R. Bickel attempted a Bayesian version, using a marginal likelihood ratio within betting settings, that is an incoming American Statistician paper. I may have being missing some aspects due to a lack of sleep the night before (!), but I find the attempt resulting in a fairly unusual vision of Bayesian testing as either not depending on any parameter or on the opposite using a family of priors. I did not understand either the “criticism” that the predictive depends on the prior and felt that this representation was bending in a rather onsiderable way the Bayesian perspective towards achieving a certain degree of agreement with p– and e-value notions, to conclude that the Bayes factor is an e-value. (As an aside, this may be the first paper that cited our critical review of Aitkin! Similarly, Shubhada Agrawal mentioned Roger Farrell in his talk, with whom we wrote a complete class Annals paper in the late 1980’s.) Nikos Ignatiadis also explored Empirical Bayes e-values, while Ben Chugg gave a presentation (constrained) admissibility, albeit under type-I error constraints that makes Bayes infeasible and using Neyman-Pearsonian loss functions. On the last day, Peter Grünwald tried for some BFF cohesion with openings on e-posteriors, treating hypothesis testing losses symmetrically, defining it as an inverse of e-values but incorporating pseudo-posteriors of many flavours like confidence, inferential, and fiducial distributions. He also mentioned a Savage-Dickey version while using an arbitrary prior, which is also an e-value, but with upper & lower meanings, again with measure issues

Given the hosting of the workshop in the Chennai Mathematical Institute, which is quite far from the centre of town (much closer to Mahabalipuram!), I did not visit Chennai but enjoyed the South Indian cuisine (albeit missing some fierceness in the spices!) and local fruits from street stands, if being sorry I could not find cocoa pods from nearby Kerala.

off to Chennai

Posted in Statistics, Travel, University life with tags , , , , , , , , , , , , , , , , on June 29, 2025 by xi'an

Today, I am off to Chennai (aka Madras) for a… Banff workshop! As one of its several branches and associated institutions (incl. Oaxaca), BIRS has the Chennai Mathematical Institute hosting the workshop “Game-theoretic statistical inference: Optional Sampling, Universal Inference, and Multiple Testing based on e-values”, which I attend in an attempt to better ascertain e-values. (Obviously, George Pérec could not have been invited!)

Nice meeting!

Posted in pictures, R, Running, Statistics, Travel, University life with tags , , , , , , , , , , , , , , , , , , , , , , , on December 18, 2024 by xi'an

The ICSDS 2024 meeting in Nice is quite impressive and not primarily because it is in Nice under a beautiful December sun. As other (numerous) IMS meetings I attended (since the initial one in Uppsala in 1990!), the program is of high quality and along topics that are currently moving fast or emerging. From the sessions I attended, e-values are strongly represented, although it remains unclear to me why they should constitute a major departure from p-values, as they stick to hypothesis testing, Type I error, power, and the whole paraphernalia of Neyman-Pearson formalism. If I manage to attend a BIRS workshop on the subject next Summer, I may manage to get a better e-derstanding!The MCMC (only!) session included a presentation by Guanyang Wang that generalised different approximate MCMC schemes into a unified one. And one by Filippo Ascolani on Gibbs beating the competition! I also attended the Bayesian prediction session, where my friends Sonia Petrone and Chris Holmes have presentations on their respective Series B papers. I discussed both on the ‘Og, on 15 March 2023 and 07 November 2022, respectively. This time, I found that both talks had a Bayesian bootstrap flavour, which is not surprising when considering the non-parametric nature of the approach. And they left me wondering at it being protected from overfitting.
My only plenary session was Cynthia Dwork’s on outcome indistinguishability, which, while related to the privacy topics I was topic, remained somewhat obscure as to its purpose. Meaning I have to get through the paper to get a more holistic perspective.
Of course, Nice in Winter is a very nice place, with the waterfront available for running an uninterrupted 15km as we found out with Jérémie Houssineau (at a brisk 4’09” pace I had not planned before starting!) and the sea all for myself (for a dozen minutes before losing digits!). Unfortunately I had to skip the final day due to examinations of the Paris Dauphine MASH master. And miss Stan receiving a student award. But I am looking forward the next iterations of ICSDS. (Not including Copenhagen, Madrid and many many other places in 2025, since ICSDS seemed a most common name for conferences, some presumably predatory! The true location is Sevilla, to keep up with the Mediterranean theme of ICSDS!)

full Bayesian significance test

Posted in Books, Statistics with tags , , , , , , , , , , on December 18, 2014 by xi'an

Among the many comments (thanks!) I received when posting our Testing via mixture estimation paper came the suggestion to relate this approach to the notion of full Bayesian significance test (FBST) developed by (Julio, not Hal) Stern and Pereira, from São Paulo, Brazil. I thus had a look at this alternative and read the Bayesian Analysis paper they published in 2008, as well as a paper recently published in Logic Journal of IGPL. (I could not find what the IGPL stands for.) The central notion in these papers is the e-value, which provides the posterior probability that the posterior density is larger than the largest posterior density over the null set. This definition bothers me, first because the null set has a measure equal to zero under an absolutely continuous prior (BA, p.82). Hence the posterior density is defined in an arbitrary manner over the null set and the maximum is itself arbitrary. (An issue that invalidates my 1993 version of the Lindley-Jeffreys paradox!) And second because it considers the posterior probability of an event that does not exist a priori, being conditional on the data. This sounds in fact quite similar to Statistical Inference, Murray Aitkin’s (2009) book using a posterior distribution of the likelihood function. With the same drawback of using the data twice. And the other issues discussed in our commentary of the book. (As a side-much-on-the-side remark, the authors incidentally  forgot me when citing our 1992 Annals of Statistics paper about decision theory on accuracy estimators..!)