Skip to contents

Headless curation

Source: vignettes/headless-curation.Rmd
headless-curation.Rmd

Headless curation runs the whole pipeline from a script. It queries the CompTox dashboard, so the code in this vignette is shown but not executed.

One call 

library(concert)

out <- curate_headless(
  input_path = "data/inventory.xlsx",
  output_path = "output/inventory_curated.xlsx",
  tag_map = list(
    chemical_name = "Name",
    cas_number = "CASRN",
    result = "Result",
    unit = "Unit"
  )
)

tag_map keys are column names after janitor::clean_names(), so "CAS Number" becomes cas_number. Run suggest_column_tags() on the cleaned names if you are unsure (see vignette("cleaning-pipeline")).

The output workbook has one sheet per pipeline stage: cleaned data, resolution state, cleaning audit, consensus summary, and a Pipeline Config sheet that lets the app or parse_concert_export() reload the run.

The return value mirrors the workbook:

out$data          # one row per input row, with dtxsid, preferred name, and status
out$audit_trail   # every cleaning change

Set write_files = FALSE to get the same list without touching disk.

Useful options

Argument Use it when
header_row Frontmatter detection picked the wrong row.
value_corrections A prefix, suffix, or typo affects many rows. Applied before cleaning.
cleaning_steps A cleaning step mangles this dataset. list(chiral = FALSE) and so on.
postprocess_candidates You want the same enrichment, scoring, and auto-resolve pass as the app.
accept_suggestions Accept every candidate scored suggested after postprocessing.
review_picks You know the right DTXSID for specific names. Validated against CompTox.
row_flags Mark rows FOLLOW-UP, BAD, or VERIFIED with a reason.
multi_analyte_resolutions Split, keep, or rename cells like "Lead + Zinc".
harmonize, media, format Parse results, convert units, and write ToxVal parquet or CSV.
starts_with Allow a CompTox starts-with fallback for names nothing else matched.

Review inputs are content-keyed. A review_picks row matches on the cleaned name (and casrn when given), never on row position, so decisions survive re-runs after upstream corrections change the row order.

out <- curate_headless(
  input_path = "data/inventory.xlsx",
  output_path = "output/inventory_curated.xlsx",
  tag_map = list(chemical_name = "Name", cas_number = "CASRN"),
  postprocess_candidates = TRUE,
  accept_suggestions = TRUE,
  review_picks = data.frame(
    name = "TKN",
    dtxsid = "DTXSID0020000"
  ),
  row_flags = data.frame(
    name = "Total PFAS",
    flag = "FOLLOW-UP",
    reason = "Group total, no single DTXSID"
  ),
  value_corrections = data.frame(
    column = "chemical_name",
    pattern = "^Dissolved ",
    replacement = ""
  )
)

Stages

curate_headless() is a composition of six stage functions. Call them yourself when you want to stop partway, inspect state, or swap a step.

state <- stage_ingest(input_path = "data/inventory.xlsx", tag_map = tag_map)
state <- stage_clean(state)
state$cleaning_result$audit_trail

state <- stage_curate(state)
state$consensus_summary

state <- stage_review(state, accept_suggestions = TRUE)
state <- stage_harmonize(state, harmonize = TRUE)
stage_export(state, output_path = "output/inventory_curated.xlsx")

Each stage adds fields to the same state list. The header comment in R/curate_stages.R lists them.

Replay scripts

generate_concert_script() writes a standalone script that reproduces a run with one curate_headless() call. It embeds every decision plus compact snapshots of the reference lists, unit map, and media map, so the replay is pinned even after package defaults change.

script <- generate_concert_script(
  input_path = "data/inventory.xlsx",
  output_path = "output/inventory_curated.xlsx",
  tag_map = tag_map,
  header_row = state$detection$header_row,
  reference_lists = state$reference_lists,
  accept_suggestions = TRUE,
  review_picks = data.frame(name = "TKN", dtxsid = "DTXSID0020000")
)
writeLines(script, "output/replay.R")

The app’s export does the same from the Review Results tab, and the agent loop below writes one on every run. Commit replay.R with the data. It is the reproducibility record.

Agent loop

For unattended curation, the package ships a decide-run-inspect loop. An agent (or you) edits one file, decisions.R, and re-runs until no row is pending.

curate_decisions_template("data/inventory.xlsx", out_dir = "output/loop")

This writes output/loop/decisions.R with the detected header row, suggested tags, a pinned reference-list snapshot, and a commented schema for every other decision object. Then:

curate_iterate("output/loop/decisions.R")

Each run writes:

  • status.md: done or not, errors, unmatched decision keys, and a list of low-similarity matches that deserve a second look.
  • pending.csv: one row per unresolved data row with a pending_type (multi_analyte, verified_unresolved, candidate_validation, identity_scope, disagree, suggested, no_match), the suggested DTXSID or split, and the candidate list.
  • replay.R: the standalone reproduction script.
  • cache/: CompTox search results, so re-runs only re-query changed names.

Resolve pending rows by adding to the decision objects in decisions.R (review_picks, row_flags, multi_analyte_resolutions, value_corrections, cleaning_steps, reference-list overrides), then re-run. Root-cause fixes that clear many rows beat per-row picks.

A command-line runner and an agent skill document the same loop:

system.file("scripts/curate_loop.R", package = "concert")
system.file("skills/concert-curate/SKILL.md", package = "concert")
Rscript <runner> --template input.csv output/loop
Rscript <runner> output/loop/decisions.R   # exit 0 = done, exit 2 = rows pending

Review evidence and accepted identities

done and queue_complete describe queue disposition. A retained FOLLOW-UP can complete that queue while reconciliation_complete, identity_review_complete or review_complete remain false. Each iteration writes review_reconciliation.csv, candidate_review.csv, identifier_diagnostics.csv and identity_review.csv. Changed evidence can reopen candidate validation; repeating a VERIFIED flag cannot resolve a blank/tied identity. Reviewed WQX canonical names retain their separate name-only contract and never manufacture a DTXSID.

Historical flags have no decision snapshot. They report baseline_missing, regardless of the reason’s wording. To record evidence actually inspected at an explicit decision, use the shared portable object:

run <- curate_iterate("output/loop/decisions.R")
review_decision_evidence <- capture_review_state(
  run$state, name = "Reviewed analyte", disposition = "deferred",
  flag = "FOLLOW-UP", reason = "Specific source-scope question remains"
)

Save that object in decisions.R; generated replay and workbook Session State preserve its immutable revisions. acknowledge_review_evidence() binds an explicit acknowledgment to the decision revision, exact source/content scope and current evidence fingerprint. It resolves that reconciliation event, without clearing FOLLOW-UP or accepting an identity. A later evidence or scope change reopens work. Use validate_review_candidates() with an explicit configured validator to recheck saved candidates independently of name discovery. It returns structured outcomes and a versioned cache; outages are retryable, and no DSSTox download is triggered. Candidate ordering, duplicates and audit timestamps are not new evidence.

The DTXSID tag supplies attributed source evidence. Registry membership is validated independently of source-identity correspondence; successful source, resolver and PubChem agreement does not promote a source ID. Untagged ID columns warn, while deliberate metadata can be retained without queries:

curate_headless(
  input_path = "input.csv", output_path = "curated.xlsx",
  tag_map = list(chemical_name = "Name", source_dtxsid = "DTXSID")
)
# Alternatively omit source_dtxsid from tag_map and retain it only as metadata:
# ignored_identifier_cols = "source_dtxsid"

Ignored columns must have no chemical evidence role. Existing unignored Other roles retain their generic-search behavior. Raw input columns named dtxsid or dtxsid_* cannot bypass the generated-evidence registry.

Lookup consensus remains in Curated Data. identity_review_state() derives blockers and accepted_dtxsid; accepted_identity_view() retains eligible rows with all source lineage. The Accepted Identities sheet is additive. Registered mixtures can qualify through explicit correspondence review; aggregates may be kept unresolved without selecting a representative component. Split rows retain original name/CAS and part lineage, with repeated component CAS blocked until explicit mapping or correspondence review.

Structured acceptance is available through identity_decisions and apply_identity_decisions(). In Review Results, open Override or Compare Candidates and use Source identity correspondence. Select one source row when the table groups several rows. Choose its scope and remaining conflict, enter a selected ID, explicitly confirm correspondence, and supply a reason and evidence reference. Save source decision checks membership independently and records the scoped decision. Keep unresolved retains candidate evidence without granting acceptance. FOLLOW-UP/BAD flags stay in place; recording correspondence does not clear them. Reopen the review if evidence changes while the dialog is open. Each record requires a unique source/content selector plus original lineage, resolved scope/conflict, reason, evidence reference, selected ID, explicit correspondence = TRUE, and identity_evidence_fingerprint() from the inspected row. Registry membership is checked separately. Changed or ambiguous evidence fails closed. Name/CAS selectors may span many rows for flags and ordinary picks; they cannot express source-specific scope acceptance.

For safe ToxVal output, opt in to toxval_identity_mode = "accepted" in headless or iteration decisions when harmonize = TRUE. Every measurement row remains, with NA identifiers for blocked identities and no raw-ID fallback. Existing lookup defaults are retained as an explicit compatibility choice; this audit mode is not an accepted-identity filter. The standalone mapper supports the equivalent identity_mode = "accepted". Workbook import, GUI refresh and generated replay preserve the explicit mode; accepted-only refresh blanks IDs if current alignment cannot be verified.

A small development demo uses the actual Review Results module with synthetic rows and mocked registry responses. From the repository root:

source("inst/examples/identity-review-demo.R")
run_identity_review_demo()

Try a registered mixture, two grouped source rows, unavailable source membership, and a retained FOLLOW-UP. Only synthetic IDs DTXSID123, DTXSID456 and DTXSID789 pass the mocked membership service; source validation and correspondence gates still apply. Discovery is disabled. These fixtures do not validate real chemical identities or change pilot assignments.