zahrcast
sia · methods

how Sia works, in detail

This page describes how Sia turns the data so far into forecasts: the simulated trials, the two submodels, borrowing from historical trials, how a new indication is set up, and validation. The manuscript gives the full specification.

forecasts

forecasts as simulated trials

Each forecast comes from simulating every patient forward from the data so far, thousands of times, which gives thousands of simulated completed trials. A statement such as “a 70% probability the benefit exceeds 2 months” means that the benefit exceeds 2 months in 70% of the simulated trials. The result is the full set of simulated outcomes. The set is wide for two reasons: patients differ from each other, and the data so far is consistent with more than one set of parameter values.

Formally, the simulated trials are draws from the posterior predictive distribution.1

Most interim forecasting methods work on the endpoint directly. They estimate a hazard from the events observed so far2 and extrapolate the survival curve.3 Sia instead models each patient's disease burden over time and derives progression and death from it. A patient with four recorded assessments adds little to an event count, but a lot to an estimate of their burden trajectory.

submodels

disease burden, then events

The two submodels are fitted jointly, as in joint models for longitudinal and time-to-event data.4–6 The burden trajectory provides covariates for the multistate submodel, and the event data also informs the trajectory. Progression and death are states in the multistate submodel, so progression-free and overall survival come from the same simulated trials and are consistent with each other.7 When an indication includes a state for leaving the trial, that state is forecast as well. The overview diagram shows how the pieces connect.

disease burden time on treatment baseline nadir 20% above nadir: progression on the target edge total burden shrinking part growing part

The mechanistic submodel, drawn for one illustrative patient. At the start of treatment, a share of the disease burden shrinks at rate and the rest grows at rate . Their sum falls to a nadir and then regrows. Each patient has their own , and , partially pooled across patients, arms and trials. An indication can also be set up so that regrowth slows over time. In the small-cell lung cancer and colorectal case studies, the burden is the sum of the longest diameters of the target lesions. The prostate case study uses PSA, and the pancreatic case study fits tumor size and CA19-9 to one burden. In the small-cell lung cancer case study, progression on the target edge of the multistate submodel is the time the forecast burden passes 20% above its nadir. RECIST 1.1 uses the same 20% threshold, together with an absolute increase of at least 5 mm.8

target non-target progression-free progressed left the trial deceased

The states of the multistate submodel and the transitions between them.9 A patient can progress in two ways. On the target edge, progression follows the RECIST rule applied to the forecast tumor burden of the target lesions. On the non-target edge, progression comes from a hazard that covers new and non-target lesions. Each indication sets which edges it uses. In the small-cell lung cancer case study, a progression can come from either edge. In the colorectal case study, both edges are hazards that follow the reader's recorded call, one for target lesions and one for non-target and new lesions, and a further state covers patients who stop treatment before progression. Each case study page shows its own states and transitions.

structure

pooling, declarations and checks

burden dynamics

Each patient's disease burden follows a bi-exponential curve of decline and regrowth. It extends the bi-exponential form of Stein and colleagues.10 The current burden and growth rate enter the multistate submodel as time-varying covariates.

hierarchy

Parameters are partially pooled across patients, arms and trials.1 The amount borrowed from a historical trial is estimated from the data. When the trials disagree, Sia borrows less and the intervals are wider. Related methods include power priors, commensurate priors and robust meta-analytic-predictive priors.11–13

declarations

Each indication is described by a declaration: its states and transitions, its priors and how its data is read. Moving to a new indication or biomarker usually means writing a new declaration and new data handling rather than changing Sia's core code. Some indications also need new code, for example for several biomarker channels.

validation

Forecasts are checked by leave-future-out validation.14 Sia is fitted at an earlier cut-off and scored against the data that arrived after it. The test suite also evaluates the log density at fixed parameter values and compares the result with recorded baselines. The tests flag any code change that alters it. Each case study's diagnostics page also compares survival replicated from each fit with the observed data, and shows a PIT histogram per endpoint and arm.

tools

open tools

Sia is written in Julia.15 Its statistical code is written in Turing.jl16 and fitted with the No-U-Turn sampler, a form of Hamiltonian Monte Carlo.17 These are open-source tools with public documentation, so a client's statistician can check how the fitting works.

back to the platformcase studies
contact

if you have a trial in flight

email about zahrcast
references
  1. Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB. Bayesian Data Analysis. 3rd ed. Boca Raton: Chapman & Hall/CRC; 2013. doi:10.1201/b16018.
  2. Bagiella E, Heitjan DF. Predicting analysis times in randomized clinical trials. Statistics in Medicine. 2001;20(14):2055–2063. doi:10.1002/sim.843.
  3. Latimer NR. Survival analysis for economic evaluations alongside clinical trials—extrapolation with patient-level data: inconsistencies, limitations, and a practical guide. Medical Decision Making. 2013;33(6):743–754. doi:10.1177/0272989X12472398.
  4. Wulfsohn MS, Tsiatis AA. A joint model for survival and longitudinal data measured with error. Biometrics. 1997;53(1):330–339. doi:10.2307/2533118.
  5. Henderson R, Diggle P, Dobson A. Joint modelling of longitudinal measurements and event time data. Biostatistics. 2000;1(4):465–480. doi:10.1093/biostatistics/1.4.465.
  6. Rizopoulos D. Joint Models for Longitudinal and Time-to-Event Data: With Applications in R. Boca Raton: Chapman & Hall/CRC; 2012. doi:10.1201/b12208.
  7. Meller M, Beyersmann J, Rufibach K. Joint modeling of progression-free and overall survival and computation of correlation measures. Statistics in Medicine. 2019;38(22):4270–4289. doi:10.1002/sim.8295.
  8. Eisenhauer EA, Therasse P, Bogaerts J, et al. New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). European Journal of Cancer. 2009;45(2):228–247. doi:10.1016/j.ejca.2008.10.026.
  9. Putter H, Fiocco M, Geskus RB. Tutorial in biostatistics: competing risks and multi-state models. Statistics in Medicine. 2007;26(11):2389–2430. doi:10.1002/sim.2712.
  10. Stein WD, Figg WD, Dahut W, et al. Tumor growth rates derived from data for patients in a clinical trial correlate strongly with patient survival: a novel strategy for evaluation of clinical trial data. The Oncologist. 2008;13(10):1046–1054. doi:10.1634/theoncologist.2008-0075.
  11. Ibrahim JG, Chen MH. Power prior distributions for regression models. Statistical Science. 2000;15(1):46–60. doi:10.1214/ss/1009212673.
  12. Hobbs BP, Carlin BP, Mandrekar SJ, Sargent DJ. Hierarchical commensurate and power prior models for adaptive incorporation of historical information in clinical trials. Biometrics. 2011;67(3):1047–1056. doi:10.1111/j.1541-0420.2011.01564.x.
  13. Schmidli H, Gsteiger S, Roychoudhury S, et al. Robust meta-analytic-predictive priors in clinical trials with historical control information. Biometrics. 2014;70(4):1023–1032. doi:10.1111/biom.12242.
  14. Bürkner PC, Gabry J, Vehtari A. Approximate leave-future-out cross-validation for Bayesian time series models. Journal of Statistical Computation and Simulation. 2020;90(14):2499–2523. doi:10.1080/00949655.2020.1783262.
  15. Bezanson J, Edelman A, Karpinski S, Shah VB. Julia: a fresh approach to numerical computing. SIAM Review. 2017;59(1):65–98. doi:10.1137/141000671.
  16. Ge H, Xu K, Ghahramani Z. Turing: a language for flexible probabilistic inference. In: Proceedings of the 21st International Conference on Artificial Intelligence and Statistics. PMLR 84; 2018:1682–1690. proceedings.mlr.press/v84/ge18b.html.
  17. Hoffman MD, Gelman A. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research. 2014;15(47):1593–1623. jmlr.org/papers/v15/hoffman14a.html.