Grant-coding walkthrough

This walkthrough classifies ten entirely synthetic grant abstracts into four topics. It demonstrates the complete pilot workflow without requiring you to design a project first. The example contains no real research data and is safe to inspect or modify. Copy the checked-in examples/grant_coding directory to practice this walkthrough before adapting your own project; kllm-batch init my-project scaffolds a bare single-task template instead (see Basic workflow).

The commands below assume you are in the root of the repository checkout created during Installation. If needed, clone and install it first:

$ git clone https://github.com/rs-kellogg/llm-batch_pipeline.git
$ cd llm-batch_pipeline
$ python -m pip install -e .

What the example does

The project combines five pieces:

data/input-data.csv ──┐
                  ├── prompts ──> provider request ──> validated results
context/codebook.csv ────────────┘                └── schema.json

  • data/input-data.csv contains ten grants identified by grant_id and illustrative reference labels for local evaluation.

  • project.yaml sends each grant’s title and abstract to the model while preserving year, investigator, and source-file metadata locally.

  • context/codebook.csv defines the financial, organizational, technical, and other labels.

  • prompts/ contains the coding instructions and inserts both the codebook and grant records into each request.

  • schema.json requires a primary label, optional secondary label, confidence score, and short justification for every grant.

The relevant input mapping in project.yaml is:

input:
  id_column: grant_id
  fields_sent:
    project_title: project_title
    abstract: abstract
  columns_preserved:
    - year
    - investigator
    - source_file

Only record_id, project_title, and abstract enter the model prompt; the preserved columns are joined back into results locally. With rows_per_request: 3, the complete ten-row example produces four requests. See Project settings for what each field controls.

The source column reference_primary_label is deliberately neither sent nor preserved in results. Instead it drives local evaluation:

evaluation:
  random_seed: 42
  gold_columns:
    primary_label: reference_primary_label

See Project settings for how gold_columns scoring works. The ten reference labels here are teaching examples based on the codebook, not an externally validated benchmark — replace them with your own reviewed labels before using agreement as research evidence.

1. Validate the project

Run the local validation before creating a run:

$ kllm-batch validate -c examples/grant_coding/project.yaml

See Basic workflow for what validation checks. This step is local, requires no API key, and creates no run artifacts.

2. Prepare a four-grant pilot

Prepare a deterministic sample with OpenAI:

$ kllm-batch prepare -c examples/grant_coding/project.yaml --provider openai --sample-size 4 --seed 42

See Basic workflow for what prepare does. The command prints a new directory under examples/grant_coding/runs/; use that complete path as RUN_DIR in later commands. Because this is a selected pilot, it defaults to synchronous execution.

3. Inspect before submitting

Open RUN_DIR/REVIEW.md first. Confirm:

  • four of ten rows were selected with seed 42;

  • the provider, model, and synchronous execution mode are correct;

  • the request count and estimated maximum cost are reasonable; and

  • the displayed next command points to this run.

Then inspect RUN_DIR/api_requests/segment_*.jsonl. These files contain the exact saved payloads, including the rendered system instructions, codebook, and grant records. If anything is wrong, edit the source project and prepare a new run—do not edit generated payloads.

Important

Everything through this inspection step is local and free. The next command contacts the selected provider and may incur model charges.

4. Submit the pilot

Set the OpenAI credential if it is not already present:

$ export OPENAI_API_KEY="your-key"

Submit the exact requests you reviewed:

$ kllm-batch submit RUN_DIR

See Basic workflow for what submit does for a synchronous pilot; it does not require a separate status or sync command here.

The equivalent Anthropic pilot and complete-batch workflow appears below.

5. Review the results

After a successful pilot, inspect:

Pilot outputs

Path

Contents

RUN_DIR/outputs/results.csv

Human-readable predictions, preserved metadata, and row provenance.

RUN_DIR/outputs/results.parquet

The same results in a type-stable format for analysis.

RUN_DIR/run_reports/audit.json

Expected, valid, missing, unexpected, and duplicate record counts.

RUN_DIR/run_reports/run_summary.json

evaluation_metrics.primary_label gives exact-match accuracy against the selected reference labels.

RUN_DIR/input_snapshot/gold_labels.parquet

Local reference labels selected for this run; its relative path is recorded as gold_labels_file in manifest.json.

RUN_DIR/run_reports/usage.json

Request-level and total token usage.

RUN_DIR/outputs/failures.jsonl

Created only if a request or result failed validation.

Check that each selected grant has one valid result, the label and justification are sensible, and the preserved metadata is present. The accuracy metric counts only valid results with matching reference IDs; inspect the audit report for missing or invalid results before interpreting it. If failures occur, follow Retry failed records and merge the results rather than editing raw responses.

6. Continue to the complete batch

Once the pilot is satisfactory, omit the sample options to prepare all ten grants. A complete run defaults to batch execution:

$ kllm-batch prepare -c examples/grant_coding/project.yaml --provider openai
$ kllm-batch submit RUN_DIR
$ kllm-batch sync RUN_DIR --watch

Use the new run directory printed by the second prepare command; do not reuse the pilot’s path. Review the complete run before submitting it, just as you reviewed the pilot.

7. Run with Anthropic

Set the Anthropic credential if it is not already present:

$ export ANTHROPIC_API_KEY="your-key"

Prepare and submit a separate deterministic pilot. The selected pilot runs synchronously, so submit processes its results and audit directly:

$ kllm-batch prepare -c examples/grant_coding/project.yaml --provider anthropic --sample-size 4 --seed 42
$ kllm-batch submit ANTHROPIC_PILOT_RUN_DIR

After reviewing the pilot, prepare and process the complete Anthropic batch:

$ kllm-batch prepare -c examples/grant_coding/project.yaml --provider anthropic
$ kllm-batch submit ANTHROPIC_RUN_DIR
$ kllm-batch sync ANTHROPIC_RUN_DIR --watch

Replace each placeholder with the complete run directory printed by its prepare command. As with OpenAI batch runs, kllm-batch status ANTHROPIC_RUN_DIR and kllm-batch audit ANTHROPIC_RUN_DIR are optional diagnostics. sync --watch already polls status, processes the responses, and runs the audit.

8. Compare providers

After both complete runs have been processed, compare their results locally:

$ kllm-batch compare OPENAI_RUN_DIR ANTHROPIC_RUN_DIR

The comparison reports agreement on categorical and string fields and exports disagreements for human review. It does not use a model to adjudicate differences. Use runs containing the same record IDs and output schema for the clearest interpretation.

Where to go next

  • Read Advanced workflows for retry/merge chains, provider comparisons, cancellation recovery, targeted reruns, and detailed configuration.

  • See the full example README for teaching fixtures and additional operational detail.