Impute Non-Detect Values in Laboratory Data — impute

This function imputes non-detect (censored) values in environmental laboratory analytical data using survival models with automatic distribution selection. It validates data quality requirements and fits multiple distributions to select the best model based on AIC. Each imputed value is guaranteed to be below its respective detection limit and above the specified minimum value.

Usage

impute_nondetect(
  dt,
  value_col = "value",
  cens_col = "censored",
  parameter_col = NULL,
  unit_col = NULL,
  dist = c("gaussian", "lognormal", "weibull", "exponential", "logistic", "loglogistic"),
  min_observations = 25,
  max_censored_pct = 75,
  min_value = 0,
  zero_value = 1e-05,
  control = survival::survreg.control(),
  verbose = FALSE
)

Arguments

dt: A data.frame or data.table containing laboratory analytical data
value_col: Character string specifying the column name containing values
cens_col: Character string specifying the column name containing censoring indicators (0 = non-detect/censored, 1 = detected/observed)
parameter_col: Character string specifying the column name containing parameter names (optional, for validation)
unit_col: Character string specifying the column name containing units (optional, for validation)
dist: Character vector of distributions to test. Options include: "gaussian", "lognormal", "weibull", "exponential", "logistic", "loglogistic"
min_observations: Minimum number of observations required for modeling (default: 25)
max_censored_pct: Maximum percentage of censored values allowed (default: 75)
min_value: Minimum allowable value for imputed concentrations (default: 0, use 1e-10 for strictly positive)
zero_value: Value that replaces exact zeros when fitting the distributions defined only for positive values ("lognormal", "weibull", "exponential", "loglogistic"), which otherwise cannot be fitted (default: 1e-5). The replacement is used for model fitting only; the zeros are kept in [value_col]_final. Use NULL to disable, in which case those distributions are skipped whenever the data contain a zero.
control: A survreg.control object used to control the fitting algorithm, e.g. maximum number of iterations and convergence tolerance. Defaults to survival::survreg.control(). Increase maxiter (e.g. survreg.control(maxiter = 200)) when convergence warnings are raised for complex datasets.
verbose: Logical indicating whether to display progress messages and distribution fitting information (default: FALSE)

Value

A data.table with additional columns:

[value_col]_imputed: Imputed values for non-detect observations
[value_col]_final: Final values combining original detected and imputed non-detect values

The returned object also has attributes containing model information:

best_model: The fitted survival model object
best_distribution: Name of the best-fitting distribution
detection_limits: Vector of all detection limits found in the data
max_detection_limit: The highest detection limit (for reference)
parameter: Parameter name (if parameter_col provided)
unit: Unit of measurement (if unit_col provided)
aic: AIC value of the best model
sample_size: Total number of observations
censored_pct: Percentage of censored observations
best_converged: FALSE if no distribution converged and the selected model is the best non-converged fit
fits: A data.table with one row per distribution tried: AIC, convergence, and any error or warning message
n_zero_replaced: Number of exact zeros replaced by zero_value when fitting the positive distributions

Details

The function performs several validation checks: 1. Ensures sufficient sample size (>= min_observations) 2. Checks that censoring percentage is reasonable (<= max_censored_pct) 3. Validates that only one parameter and unit are present (if columns provided) 4. Tests multiple distributions and selects the best based on AIC among those that converged (if none converged, the best non-converged fit is used and a warning is raised) 5. Generates random imputed values below each observation's detection limit and above min_value

The imputed values are random draws from R's random number generator; call set.seed before impute_nondetect() to make them reproducible.

For non-detect observations (censored = 0), the value in value_col is treated as the detection limit for that specific analysis, allowing for different detection limits across samples or analytical methods.

IMPORTANT: This function should be applied to data containing only ONE parameter at a time. Different environmental parameters have different distributions and should not be modeled together.

When verbose = FALSE, the function operates silently except for critical errors, making it suitable for batch processing of multiple parameters.

Examples

# Load example data
data(multi_censored_data)

# Basic imputation with default settings
set.seed(123)
result <- impute_nondetect(
  dt = multi_censored_data,
  value_col = "value",
  cens_col = "censored",
  verbose = FALSE
)

# View imputed values for non-detects
head(result[censored == 0, .(value, value_imputed, value_final)])
#>    value value_imputed value_final
#>    <num>         <num>       <num>
#> 1:    25      8.487192    8.487192
#> 2:    15     10.300671   10.300671
#> 3:     5      4.937120    4.937120
#> 4:     5      4.798058    4.798058
#> 5:     5      4.408386    4.408386
#> 6:     5      4.284067    4.284067

# Check best distribution selected
attr(result, "best_distribution")
#> [1] "lognormal"

# Increase max iterations for difficult datasets
result <- impute_nondetect(
  dt        = multi_censored_data,
  value_col = "value",
  cens_col  = "censored",
  control   = survival::survreg.control(maxiter = 200)
)

# With parameter and unit validation
result <- impute_nondetect(
  dt = multi_censored_data,
  value_col = "value",
  cens_col = "censored",
  parameter_col = "parameter",
  unit_col = "unit"
)

# For strictly positive values (avoiding exactly zero)
result <- impute_nondetect(
  dt = multi_censored_data,
  value_col = "value",
  cens_col = "censored",
  min_value = 1e-10,
  verbose = FALSE
)