Insider Intel on what Reviewers are Saying about Intervention Optimization Proposals Guided by the Multiphase Optimization Strategy

Lizbeth ‘Libby’ Benson, PhD; Danielle M. Ostendorf, PhD; Michele L. Patel, PhD - Intervention Optimization SIG

Many academics can relate to the experience of pouring their heart and soul into a grant proposal only to have reviewers completely misunderstand key details pertaining to the importance of the research and the approach. This is especially true when leveraging paradigms, frameworks, and trial designs that are from the status quo. In intervention science, the classical treatment package research paradigm involves leveraging a randomized controlled trial (RCT) to test whether an intervention package performs better than a suitable control. Intervention optimization is an alternative paradigm. The Multiphase Optimization Strategy (MOST) is a framework within the intervention optimization paradigm, which can be used to systematically guide intervention optimization with the goals of strategically balancing effectiveness, affordability, scalability and efficiency.¹ Rather than immediately evaluating an intervention package against a suitable control using an RCT, the MOST framework specifies activities across three phases, Preparation, Optimization and Evaluation, with the potential for work to progress non-linearly across phases.  

Knowing how to strategize intervention optimization grant proposals can take years of trial and error. To speed up this process for those at any career stage who plan on submitting intervention optimization-related proposals, we collected intel from 16 of our SBM Intervention Optimization (IO) SIG members pertaining to what reviewers have said about their past intervention optimization proposals and their top tips for effectively responding to them. Primarily, respondents indicated they submitted their optimization grant as an early-career scientist (58.8%, n=10), with 3 others as trainee scientists (17.6%), 1 as a mid-career scientist (5.9%), and 3 as senior scientists (17.6%). The majority of submissions were to NIH across a range of institutes including NIDDK, NHLBI, NCI, NIDA, NICHD, NIAAA. Diverse funding mechanisms were also listed including career development awards (K01, K99/R00, K23), R01, R03, R61/R33, R34, R21, P50, P01 and U01. Topic areas included cancer prevention and treatment, obesity, weight loss, and physical activity, smoking cessation, alcohol use, autism, HPV vaccination, and methods development.

Common Reviewer Critiques 

Respondents selected applicable themes of reviewer comments, shown in Table 1, and also provided explanatory text elaborating on the feedback, which we summarize below. 

Table 1.

Reviewer Comment Themes Endorsed by Respondents	Endorsements
Trial design was confusing (factorial, SMART, MRT)    Sample size or power concerns    Concerns about feasibility/acceptability    Innovation was not sufficiently justified    Questions about MOST (Preparation vs. Optimization vs. Evaluation)    Questions about clinical significance    Questions about efficiency/resource use    Other: concerns about contamination between conditions	15 (33.3%)   11 (24.4%)   7 (15.6%)      3  (6.7%)       1 (2.2%)         1 (2.2%)    1 (2.2%)    1 (2.2%)

Questions about the MOST framework, Significance and/or Innovation  

Several reviewers questioned the need for MOST Preparation Phase research prior to Optimization and/or Evaluation Phase research, especially when at least some feasibility/acceptability evidence already existed. In response, investigators emphasized that “failure to optimize has led to equivocal results and interventions that cannot be disseminated” and by “clearly justifying which feasibility/acceptability evidence is still needed to inform a future optimization trial.” In response to another reviewer's question about why distal health outcomes are not the focus of optimization studies, one set of investigators provided evidence showing how targeting proximal outcomes will impact distal outcomes.  

Reviewers also noted concerns about significance, clarity, and clinical impact of optimization focused research questions, and one reviewer noted a concern about the complexity of the conceptual framework. One reviewer also wanted a clearer explanation of why testing different versions of the intervention components was important and more discussion of the minimum clinically meaningful differences that would justify selection of one intervention package over another. In their response, the investigators indicated they would emphasize that “instead of testing whole programs as fixed packages, MOST helps researchers identify which specific components work best and how they can be combined in the most efficient way.” 

Confusion Regarding Trial Design 

Some reviewers also had general confusion with trial designs. For example, one reviewer wanted a clearer rationale for why the proposed trial was the most appropriate mechanism to address the scientific question, and another reviewer wanted more details on why an evaluation RCT was not being proposed. As noted in the sample size or power concerns section, the most common confusion pertaining to factorial designs pertained to whether the approach involved testing differences between cells or differences between levels of components.  

One reviewer thought MOST was the same thing as a factorial study design and thus had some confusion about a proposed SMART that only had two levels of one component in the stage 1 randomization. Another reviewer was familiar with a “prototypical” SMART design where only non-responders are re-randomized but had confusion about a proposed SMART that involved re-randomization of both responders and non-responders (which is called a nonrestricted SMART). This same reviewer also asked for more rationale for the definition of responders and non-responders (in addition to the definition itself). Pertaining to a micro-randomized trial (MRT), one reviewer expressed confusion about within-person randomizations during the optimization phase, rather than examining efficacy of an already tailored intervention. 

Sample Size or Power Concerns 

The majority of comments noted by reviewers pertained to sample size or power concerns. The most common confusion for factorial designs pertained to whether the approach involved testing differences between experimental conditions. For example, “one reviewer stated that my (fully powered) factorial trial was underpowered to assess main effects and interaction effects given the small sample size in each of the 8 conditions; they referred to it as an 8-way factorial trial.” To address this concern, our respondent emphasized that the statistician and methodologist on the team were both well-versed in factorial designs for intervention optimization and that they had confirmed the trial was fully powered. The respondent also provided further details pertaining to how the conditions in the factorial are not directly compared to one another and gave an example of how conditions of the same factor level are collapsed to assess main effects, using the full sample size to compare component levels instead of experimental conditions. Notably, this approach proved effective because the re-submission was funded! Another common concern pertained to power for detecting interactions.

Additional Concerns  

There were also several reviewer comments pertaining to the feasibility of carrying out the trial including feasibility of re-randomization and the appropriateness of a high number of feasibility/acceptability outcomes in Preparation Phase research. The primary concern about intervention components pertained to identifying a clear rationale for all factors and each of their levels, particularly for factors that have two active levels (as opposed to on vs. off levels of a factor).

Praise from Reviewers and Funding Wins

Reviewers also identified several strengths, such as:  

“The proposed study design is robust and well-structured, with a clear plan for iterative evaluation and refinement” 

“The MOST *design is efficient and appropriate, a good fit for the study goals” 

“The approach has several strengths, including an adequately powered study, a rigorous analytic plan, an efficient MOST *design to optimize intervention components and strong preliminary data supporting the feasibility of the proposed methods.” 

*We should note here that MOST is a framework (rather than a design) for optimizing interventions that can be used to guide the trial design and there are several trial designs that fit within the Optimization Phase of the MOST framework (e.g., factorial, SMART, MRT, etc.).  

Encouragingly, half of our respondents had their intervention optimization proposals funded, with another 38% of respondents still in the review process.

Advice for Future Investigators

Respondents also provided their top advice for future investigators submitting intervention optimization-related work:  

1. Tables and figures are worth a thousand words. A key source of advice from respondents was to use tables and figures:  

“A figure of the conceptual model that depicts the experimental components, any core components, putative mediators, and outcomes of interest is highly recommended. Spend time crafting this figure in a thoughtful and clear manner, and make it span the width of the grant page (don't shrink it)” 

“Reviewers love a clear and thoughtful conceptual model.” 

“[It is] important to use tables to describe factorial design.” 

2. Spend time making the MOST framework and the research approach clear and accessible to reviewers. Respondents suggested to explain the paradigm shift from classical approaches to MOST, explain why optimization is important (and what can happen without optimization), and describe how the proposed research fits into a border intervention optimization pipeline including the immediate and long-term next steps (e.g., funding mechanisms). Respondents also suggested operationally defining everything, making it clear what the experimental components are and the rationale for each, indicating what evidence is lacking in the field that requires an optimization trial, showing the trajectory of your research program, and repeating the same terms (especially those specific to MOST that reviewers may be less familiar with) throughout the grant proposal to make it easier to follow.

3. Seek help. Respondents suggested finding senior mentors to include as collaborators on the proposal who have had success with getting optimization grants funded and to make use of the many MOST trainings and workshops, including adding these to training plans for career development award applications. Two key sources for identifying trainings and code libraries relevant to intervention optimization science are the Center for Advancement and Dissemination of Intervention Optimization (CADIO) directed by Dr. Linda Collins and the Data Science for Dynamic Intervention Decision-Making Center (d3center) directed by Dr. Inbal Nahum-Shani.

Conclusion:

Altogether, we hope the collective knowledge from several IO SIG members will speed up the process for getting your intervention optimization grants from submitted to funded! Thanks very much to all of the SBM IO SIG members who shared details with us related to their intervention optimization grant proposals! 

Reference:

1. Collins, L. M., Nahum-Shani, I., Guastaferro, K., Strayhorn, J. C., Vanness, D. J., & Murphy, S. A. (2024). Intervention optimization: a paradigm shift and its potential implications for clinical psychology. Annual Review of Clinical Psychology, 20(1), 21-47.