Expert Insight: Dr Brad Jones (JMP) on Definitive Screening Designs

We recently welcomed Dr. Brad Jones, award-winning statistician and Principal Research Fellow in the JMP division of SAS, to Cambridge for a free half-day seminar titled How Definitive Screening Designs are Revolutionising Experimental Design. As co-creator of the Definitive Screening Design (DSD) alongside Dr Christopher J Nachtsheim, Dr Jones discussed the many ways DSDs can help scientists, engineers and statisticians make the most of their data and how the proper implementation of Design of Experiments (DoE) can lead to new, helpful and efficient understanding. In this special blog, he shares some insights into the origins and merits of Definitive Screening Designs.

Brad Jones on… the need for Definitive Screening Designs:

In the 1960s, when George Box created the regular factorial design family, experiments were done at two levels. My experience is that engineers are very uncomfortable with that, because they don’t think the world is linear. When you think about the way physics works, you can’t fit a curve with two lines – there are an infinite number of curves that go through any two points. Therefore, having three levels on a design is something that is really potentially useful… and that’s what Definitive Screening Designs do.

Brad Jones on… the genesis of Definitive Screening Designs:

The Definitive Screening Design is a special screening case of something Chris Nachtsheim and I called ‘Alias Optimal designs’. What we were thinking about back in 2009 was we have these D-optimal designs that maximise our capability estimating a model we’re interested in, but the Plackett-Burman happens to be globally D-optimal, but has this other thing that we don’t like which is correlations between main effects and two factor interactions (2FI). Can we give up a little bit of estimation capability for our main effects in return for something? Is it possible to give up a little bit of precision in the estimates of the main effects in return for the ability to do other stuff – like not have main effects correlate with two factor interactions and maybe we can estimate quadratics. That’s how Definitive Screening Designs were discovered.

Brad Jones on… using the Definitive Screening Design to think differently:

The first idea that people use when trying to analysis DSDs was to use stepwise regression. There are a lot of other things you can do. In JMP, you could use generalised regression tools in JMP Pro, you could use model averaging. All of those methods are basically treating the data as if you were analysing observationally. In such cases, you don’t know what the provenance of the data is.

In Design of Experiments you know a huge amount about the structure of your design in advance, so you can use your knowledge of the structure to inform your data analysis. So instead of using an off the shelf method, you can create an analytical approach which is built around the design. In the last 20 - 30 years, there’s been a lot of emphasis on model-orientated design. What I’m proposing is design-orientated models; how can we make a modelling approach that takes advantage of the special structure of the design space.

Brad Jones on… the ultimate screening design wish list:

The thing you want in a design of experiments is to have orthogonality main effects, orthogonality between the main effects and two factor interactions, and orthogonality between pairs of two factor interactions… but you can’t have all of those things. Something has got to give! For example, I’ve been doing some work in my house with a contractor – he said you can have things fast, you can things cheap, or you can have things well done. Pick two! This is what I wish for when I’m doing a screening experiment – admittedly, I came up with list after I’d discovered Definitive Screening Designs!

• Small number of runs – order of the number of factors
• Orthogonal main effects
• Main effects uncorrelated with 2FIs
• 2FIs not confounded with each other
• Estimable quadratic effects / 3 level design
• Good predictive models

Brad Jones on… comparisons with other designs:

If you were to compare a Plackett-Burman design, an Alias Optimal design and a Definitive Screening Design side by side in JMP 13, the Plackett-Burman design does the best job at estimating the main effects, but suppose it were true that factors A, B & C had 2FIs … then the story changes! If we were to look at is the fraction design space plot, you might begin to see that the Pluckett-Burman design isn’t doing as well as you’d predict, whilst the Definitive Screening Design has become more efficient at estimating the main effects. However, the DSD isn’t as efficient as the Alias Optimal Design at estimating the 2FIs because the Alias Optimal design has + and – mirrors. However, neither the Alias Optimal design nor the Plackett-Burman design can fit quadratics, whilst the DSD can estimate all of them! So that’s two additional things DSDs can do that other attempts at screening can’t handle; it has main effects that are uncorrelated with 2FIs AND it can fit all the quadratic terms of the factors.