Build patches

Creator guide

Dataset format with families, deterministic splits, the fixed training configuration, evaluation and review.

View as Markdown

The creator studio turns a dataset of examples into a trained, evaluated and signed patch candidate, then into a published version once a reviewer approves it. Everything happens in the workspace under Studio. This page covers each step and the rules behind it.

1. Describe the failure

A new patch project needs:

  • a title (3 to 80 characters) and a slug (lowercase kebab-case, 3 to 60 characters),
  • the problem: the failure in at least 40 characters, with a concrete example,
  • a success criterion of at least 20 characters,
  • a category: structured output and tool calls, domain extraction, small code repair, classification and labelling, or formatting and style conventions,
  • the base model, chosen from the supported revisions,
  • visibility (private or public) and licensing intent (free or paid).

Pick a narrow, recurring failure with an answer you can check exactly. The grader compares outputs to your expected answers, so a task with many correct phrasings will score badly even when the model improves.

2. Prepare the dataset

Upload one UTF-8 JSON Lines file (.jsonl), one example per line:

json
{"user": "Ship to: Ada Byrne, 4 Quay St, Galway. Order 1182.", "assistant": "{\"name\":\"Ada Byrne\",\"city\":\"Galway\",\"order\":1182}", "family": "irish-addresses"}
{"system": "Reply with JSON only.", "user": "Deliver order 2210 to Lars Holm, Storgatan 9, Malmo.", "assistant": "{\"name\":\"Lars Holm\",\"city\":\"Malmo\",\"order\":2210}", "family": "swedish-addresses"}
Field Required Meaning
user Yes The input the model sees.
assistant Yes The exact answer you expect.
family Yes A label for the template or source the row came from. Rows in the same family never cross a split.
system No A system prompt. Rows without one use "You are a helpful assistant."

Any other field is rejected. user, assistant and family must be non-empty strings.

Families matter

Families keep near-duplicates apart. If two rows come from the same sentence template, the same customer, the same document or the same source page, give them the same family. Splitting by family means the held-out suite tests generalization to shapes the adapter never saw, not memory of a reworded training row.

Limits

Rule Limit
File size 20 MB
Row length 8,000 characters
Valid rows At least 60
Families At least 3
Duplicates A repeated user and assistant pair is an error

Upload checks

Before anything is stored, every line is parsed and scanned:

  • Rows that fail to parse or break a rule are listed by line number, up to 50, and nothing is stored.
  • Secrets block the upload: AWS access keys, private key blocks, sk- style API keys and JSON Web Tokens.
  • Likely personal data, such as email addresses, phone numbers, valid IBANs and card numbers that pass a Luhn check, needs an explicit acknowledgement before the upload continues.
  • Findings are shown masked, so the full value is never echoed back.

You also declare where the data comes from (written by your organization, generated by your organization, provided by customers with written consent, licensed from a third party, or a public dataset), give its license as one SPDX identifier such as CC-BY-4.0 or proprietary-owned, and attest that you have the rights to use it. Read the dataset policy.

3. Splits

The studio splits the dataset by family, deterministically: families are ordered by the sha256 of a seed and the family name, so the same file always splits the same way.

Split Target share Used for
Held-out 15% of rows The only result a version may cite. Never used for training or tuning.
Validation 15% of rows Development checks. Its cases may be shown publicly as examples.
Train The rest Gradient updates, and three worked examples for the prompt-only control.

Held-out fills first, then validation, each with whole families until it reaches its share. Every split keeps at least one family. Because families are indivisible, actual shares vary with family sizes, and the studio shows the real counts.

4. Training

Creators can't tune hyperparameters. Every studio run uses one bounded configuration, the same one used for the first-party patch:

Setting Value
Method Supervised fine-tuning with LoRA, loss on assistant tokens only
Rank, alpha, dropout 16, 32, 0.05
Target modules q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Learning rate, schedule 2e-4, cosine with 5% warmup
Epochs 2
Batch 1 per step with 8 gradient accumulation steps
Max sequence length 448 tokens
Precision float16 frozen base, float32 adapter

You choose a seed and a budget cap. A cap of 0 runs on operator hardware. A non-zero cap starts paid compute and needs org admin access, and is limited to $10,000. Hosted GPU workers aren't attached yet, so jobs run on operator hardware, and the train page warns you when no worker is running. Time estimates scale from one measured data point, about 14 seconds per step on an Apple M1 Pro, and are labelled as estimates.

Only one job per project can be queued or running at a time. Only the train split is read during training.

5. Evaluation

After training, the worker evaluates the adapter on the held-out and validation splits and on the shared regression suite, with the same harness as first-party patches: base, prompt-only control with three training examples, patched, and a rollback check with the adapter disabled. Your rows are graded by exact match after trimming whitespace, or by parsed JSON equality when the expected answer is JSON.

The packager sets the starting evidence state from the held-out numbers: at least 10 percentage points of gain and at most 2 introduced regressions is IMPROVED_ON_SCOPE, no gain or a regression drop beyond the allowance is DEGRADED, and anything in between is EXPERIMENTAL. See Evaluation methodology.

Studio packages are signed with the platform key under the role platform-build. That signature attests that Retoque's worker built these exact bytes, not that the publisher is verified. Until a version is reviewed, its LICENSE.txt says no license has been granted and the adapter must not be redistributed.

6. Submit for review

Submitting needs org admin access, because it makes declarations on behalf of your organization. You confirm the dataset rights, base model license compliance and that validation cases may be shown publicly, and you write any third-party attribution, or "None". Paid licensing is limited to verified publishers and needs a price in USD and a linked payout wallet.

The studio's QA checklist must pass first:

Check Required
A finished held-out evaluation exists Yes
Regression results are recorded Yes
Adapter, package and manifest digests are present Yes
Disabling the adapter reproduces the base outputs No, but reviewers will ask
Evidence is not DEGRADED No. Degraded versions can be submitted and are shown as they are

7. Review and publication

A reviewer or operator who did not submit the version decides. Self-approval is refused. Approval needs every item on this checklist:

  • dataset rights and base model license compliance verified,
  • held-out metrics reproduce from the evaluation report,
  • compatibility pinned to an exact base revision and digests,
  • regressions disclosed in the evidence,
  • the artifact signature is present and matches the manifest digest.

Only an approved version can be published. At first release both reviewer accounts belong to the founder, which is disclosed in the decision log and is not presented as independent review.

After publication

Published files never change. To fix something, train and submit a new version. Deprecating or withdrawing a version is done by an operator, with a recorded reason. Its evidence stays public either way. Earnings from paid licenses are recorded in the studio ledger from confirmed onchain purchases only, and are labelled as test activity on test networks.