zahrcast
zahrcast
For every patient, the model simulates the patient's course many times, once for each posterior draw. Each simulation ends in an event week, or it reaches the end of the simulated follow-up without an event. The event is progression or death for progression-free survival, and death for overall survival. These simulated weeks are the patient's replicates. On the diagnostics pages each patient has about 2,000 of them, one per posterior draw. The example below uses 20 so that each one is visible.
The patient's observed week goes on the same axis, and the PIT counts the share of replicates that came earlier.
An example patient with 20 replicate event weeks. Two of them reach the end of the simulated follow-up, week 60, without an event. Move the observed week, or mark the patient as censored, to see how the PIT changes.
Time is counted in whole weeks, so some replicates land in exactly the observed week. They are shown in the example as dots of the second color. A tied replicate is neither earlier nor later than the observed event, so the PIT is a range:
The histogram needs one number per patient, so the PIT is a point picked at random inside that range. Each patient has a fixed random number u between 0 and 1, so the histogram is the same on every page load:
PIT = low + u × (high − low)
Step 5 explains why the point is random and not the midpoint of the range.
At an early data cut most patients are still event-free. For them, the event is only known to come after the observed week. Tick “censored” in the example to see this case.
Every replicate at or before that week is earlier than the real event. A replicate after it may be earlier or later, because the real event time is unknown. The PIT range therefore runs from the share of replicates at or before the censoring week up to 1, and the PIT is a point in that range picked with the same u.
event at week y: range from (share before y) to (share at or before y) censored at week y: range from (share at or before y) to 1
Two special cases:
Suppose the model is exactly right. Then the patient's real event time is one more draw from the same distribution as the replicates. Among the real time and the 20 replicates, the real time is equally likely to take any rank, from first to last. Its rank sets its PIT, so the PIT is equally likely to take any value from 0 to 1.
This holds for each patient. With one PIT per patient, a calibrated model gives a flat histogram. With 10 bins and n patients, each bin expects about n/10 patients.
The random point inside the range of a tie or a censored patient keeps this true. Always taking the low end of the range would move the PIT values toward 0, and always taking the midpoint would move them toward 0.5. A uniform random point inside the range keeps the histogram flat when the model is right. For censored patients this also requires that censoring is unrelated to when the event would have happened, which is the usual assumption in survival analysis.
The five histograms below are made-up illustrations for 100 patients, not data. The dashed line is the flat count of 10 per bin.
The shaded band on the diagnostics pages shows how far a bin's count can move from n/10 by chance. Under a calibrated model a bin's count is Binomial(n, 0.1), and the band runs from its 10th to its 90th percentile. Each bin falls outside the band about 20% of the time by chance, so one or two of ten bins outside it is expected. A steady slope or a pile-up at one end is the pattern the band helps to separate from chance.
The caption under each histogram gives the share of PIT values from 0.1 to 0.9. A calibrated model puts about 80% of patients there. A share well below 80% means more outcomes in the tails than predicted, as in the U shape. A share well above 80% means the predictions are wider than the outcomes.
The PIT in steps 1 to 6 is in sample. Each fit used the outcomes of the patients it predicts, so a patient's replicates already reflect that patient's own course. The leave-one-out PIT is the out-of-sample version. It asks where the observed outcome falls among the outcomes predicted by a fit that never saw the patient.
Refitting the model once per patient would take hundreds or thousands of fits. The existing posterior draws are reweighted instead. A draw under which all of the patient's data were likely gets a smaller weight, and a draw under which they were unlikely gets a larger one. The weighted draws then approximate a fit to every other patient. Pareto-smoothed importance sampling (PSIS) makes these weights stable by smoothing the largest of them.
The replicates are those with effects redrawn from the population. A replicate built on the patient's own estimated effects would still carry the patient's data through those effects. The PIT is then computed as in steps 2 to 4, with each replicate counting by its weight:
PIT = (weight of replicates with an event before the observed week) ÷ (total weight)
Ties and censored patients are handled as before, with weights in place of counts.
The weights can fail for a patient whose data pull the fit strongly, such as a patient unlike the others. A few draws then carry almost all of the weight. PSIS gives each patient a number, Pareto , that measures this. At most 0.5 is good, 0.5 to 0.7 is acceptable, and above 0.7 the weights, and that patient's leave-one-out PIT, are unreliable. The weights cover all of a patient's data at once, so a patient has one for both endpoints.
The leave-one-out PIT removes one patient at a time, but the fit still saw every later outcome of the other patients. The leave-future-out PIT scores each fit only on outcomes that came after its data. It needs a model refitted at many dates, so it is shown only for case studies where those fits were run.
Sia is refitted once a month, each time using only the data available at that date. Each fit is scored on a window that runs from its date to the date of the next fit. The last fit is scored up to the final data cut. A patient enters a window for an endpoint if the patient was in that fit, had not had the event at its date, and was followed past that date by the end of the window.
For each of these patients, the fit simulates replicates that start from the patient's record at its date, so every replicated event comes after that date. The observed week is the patient's status at the end of the window: the event week, or the last week the patient was known to be event-free. The PIT is then computed from the replicates exactly as in steps 2 to 4, including the random point for ties and for patients without the event.
The histograms pool all windows. A patient who has not had the event is scored again in each later window, so one patient can give several PIT values. These are patient-windows, and the histograms count them in place of patients. Values from the same patient are not independent. The shaded band assumes independent values, so it is narrower than the range the counts can take by chance. Single windows are not shown, because a late window can hold only one or two patients.
Each leave-future-out histogram also gives the C-index pooled over the windows. It measures how well the fits rank patients by risk. For each patient in a window, the forecast risk is the share of replicates with an event by the end of the window. Within a window and arm, a pair of patients is comparable when one had the event before the other was last seen. The pair is concordant when the patient with the earlier event had the higher forecast risk. Pairs with equal forecast risk count as half concordant. The C-index is the share of comparable pairs that are concordant, over all windows. A random ranking gives 0.5 and a perfect ranking gives 1.
| choice | what the pages show | reason |
|---|---|---|
| histograms | one per endpoint and arm, for the selected fit | Calibration can differ by arm and by endpoint. |
| patients | the target trial's patients, one PIT each | These are the patients the forecasts are about. |
| observed week | each patient's status at the data cut: the event week, or the last week the patient was known to be event-free | At a data cut only the follow-up up to that date is known, and the fit used only that data. |
| replicates | about 2,000 per patient, one per posterior draw, from the patient's baseline covariates and burden; with effects redrawn from the population, the patient-level effects are drawn again from the fitted population distribution | A check on single patients needs predictions for single patients. |
| bars | split into patients with an event and censored patients, for arms with at least 20 patients | At early cuts most patients are censored, and the split shows which group fills a bin. Smaller arms show only the total, so that no bar describes one patient. |
| left out | patients censored at week 0 | Their PIT range is 0 to 1. |
| in or out of sample | a control switches the histograms between the in-sample PIT, the leave-one-out PIT, which is available for the full data fit, and, where the monthly fits were run, the leave-future-out PIT, which is then shown first | The leave-one-out weights are computed once, for the fit to all the data. The leave-future-out PIT pools many fits and does not depend on the fit selected. |
| Pareto | the number of patients in each class, in the caption under each leave-one-out histogram | The counts describe how reliable a histogram is without giving any one patient's value. |
| leave-future-out values | pooled over all windows, in the histogram and the caption: patient-windows, events, mean PIT and C-index | A late window can hold one or two patients, so a value for one window could describe a single patient. |
The PIT histograms for each fit are on the diagnostics page of each case study: small-cell lung cancer, colorectal cancer, prostate cancer, pancreatic cancer.