# Creator guide

> Dataset format with families, deterministic splits, the fixed training configuration, evaluation and review.

Source: https://retoque.xyz/docs/creators
Markdown: https://retoque.xyz/docs/md/creators

The creator studio turns a dataset of examples into a trained, evaluated and signed patch candidate, then into a published version once a reviewer approves it. Everything happens in the workspace under [Studio](https://retoque.xyz/studio). This page covers each step and the rules behind it.

## 1. Describe the failure

A new patch project needs:

- a title (3 to 80 characters) and a slug (lowercase kebab-case, 3 to 60 characters),
- the problem: the failure in at least 40 characters, with a concrete example,
- a success criterion of at least 20 characters,
- a category: structured output and tool calls, domain extraction, small code repair, classification and labelling, or formatting and style conventions,
- the base model, chosen from the supported revisions,
- visibility (private or public) and licensing intent (free or paid).

Pick a narrow, recurring failure with an answer you can check exactly. The grader compares outputs to your expected answers, so a task with many correct phrasings will score badly even when the model improves.

## 2. Prepare the dataset

Upload one UTF-8 JSON Lines file (`.jsonl`), one example per line:

```json
{"user": "Ship to: Ada Byrne, 4 Quay St, Galway. Order 1182.", "assistant": "{\"name\":\"Ada Byrne\",\"city\":\"Galway\",\"order\":1182}", "family": "irish-addresses"}
{"system": "Reply with JSON only.", "user": "Deliver order 2210 to Lars Holm, Storgatan 9, Malmo.", "assistant": "{\"name\":\"Lars Holm\",\"city\":\"Malmo\",\"order\":2210}", "family": "swedish-addresses"}
```

| Field | Required | Meaning |
| --- | --- | --- |
| `user` | Yes | The input the model sees. |
| `assistant` | Yes | The exact answer you expect. |
| `family` | Yes | A label for the template or source the row came from. Rows in the same family never cross a split. |
| `system` | No | A system prompt. Rows without one use "You are a helpful assistant." |

Any other field is rejected. `user`, `assistant` and `family` must be non-empty strings.

### Families matter

Families keep near-duplicates apart. If two rows come from the same sentence template, the same customer, the same document or the same source page, give them the same family. Splitting by family means the held-out suite tests generalization to shapes the adapter never saw, not memory of a reworded training row.

### Limits

| Rule | Limit |
| --- | --- |
| File size | 20 MB |
| Row length | 8,000 characters |
| Valid rows | At least 60 |
| Families | At least 3 |
| Duplicates | A repeated `user` and `assistant` pair is an error |

### Upload checks

Before anything is stored, every line is parsed and scanned:

- Rows that fail to parse or break a rule are listed by line number, up to 50, and nothing is stored.
- Secrets block the upload: AWS access keys, private key blocks, `sk-` style API keys and JSON Web Tokens.
- Likely personal data, such as email addresses, phone numbers, valid IBANs and card numbers that pass a Luhn check, needs an explicit acknowledgement before the upload continues.
- Findings are shown masked, so the full value is never echoed back.

You also declare where the data comes from (written by your organization, generated by your organization, provided by customers with written consent, licensed from a third party, or a public dataset), give its license as one SPDX identifier such as `CC-BY-4.0` or `proprietary-owned`, and attest that you have the rights to use it. Read the [dataset policy](https://retoque.xyz/legal/datasets).

## 3. Splits

The studio splits the dataset by family, deterministically: families are ordered by the sha256 of a seed and the family name, so the same file always splits the same way.

| Split | Target share | Used for |
| --- | --- | --- |
| Held-out | 15% of rows | The only result a version may cite. Never used for training or tuning. |
| Validation | 15% of rows | Development checks. Its cases may be shown publicly as examples. |
| Train | The rest | Gradient updates, and three worked examples for the prompt-only control. |

Held-out fills first, then validation, each with whole families until it reaches its share. Every split keeps at least one family. Because families are indivisible, actual shares vary with family sizes, and the studio shows the real counts.

## 4. Training

Creators can't tune hyperparameters. Every studio run uses one bounded configuration, the same one used for the first-party patch:

| Setting | Value |
| --- | --- |
| Method | Supervised fine-tuning with LoRA, loss on assistant tokens only |
| Rank, alpha, dropout | 16, 32, 0.05 |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
| Learning rate, schedule | 2e-4, cosine with 5% warmup |
| Epochs | 2 |
| Batch | 1 per step with 8 gradient accumulation steps |
| Max sequence length | 448 tokens |
| Precision | float16 frozen base, float32 adapter |

You choose a seed and a budget cap. A cap of 0 runs on operator hardware. A non-zero cap starts paid compute and needs org admin access, and is limited to $10,000. Hosted GPU workers aren't attached yet, so jobs run on operator hardware, and the train page warns you when no worker is running. Time estimates scale from one measured data point, about 14 seconds per step on an Apple M1 Pro, and are labelled as estimates.

Only one job per project can be queued or running at a time. Only the train split is read during training.

## 5. Evaluation

After training, the worker evaluates the adapter on the held-out and validation splits and on the shared regression suite, with the same harness as first-party patches: base, prompt-only control with three training examples, patched, and a rollback check with the adapter disabled. Your rows are graded by exact match after trimming whitespace, or by parsed JSON equality when the expected answer is JSON.

The packager sets the starting evidence state from the held-out numbers: at least 10 percentage points of gain and at most 2 introduced regressions is `IMPROVED_ON_SCOPE`, no gain or a regression drop beyond the allowance is `DEGRADED`, and anything in between is `EXPERIMENTAL`. See [Evaluation methodology](https://retoque.xyz/docs/evaluation).

Studio packages are signed with the platform key under the role `platform-build`. That signature attests that Retoque's worker built these exact bytes, not that the publisher is verified. Until a version is reviewed, its `LICENSE.txt` says no license has been granted and the adapter must not be redistributed.

## 6. Submit for review

Submitting needs org admin access, because it makes declarations on behalf of your organization. You confirm the dataset rights, base model license compliance and that validation cases may be shown publicly, and you write any third-party attribution, or "None". Paid licensing is limited to verified publishers and needs a price in USD and a linked payout wallet.

The studio's QA checklist must pass first:

| Check | Required |
| --- | --- |
| A finished held-out evaluation exists | Yes |
| Regression results are recorded | Yes |
| Adapter, package and manifest digests are present | Yes |
| Disabling the adapter reproduces the base outputs | No, but reviewers will ask |
| Evidence is not `DEGRADED` | No. Degraded versions can be submitted and are shown as they are |

## 7. Review and publication

A reviewer or operator who did not submit the version decides. Self-approval is refused. Approval needs every item on this checklist:

- dataset rights and base model license compliance verified,
- held-out metrics reproduce from the evaluation report,
- compatibility pinned to an exact base revision and digests,
- regressions disclosed in the evidence,
- the artifact signature is present and matches the manifest digest.

Only an approved version can be published. At first release both reviewer accounts belong to the founder, which is disclosed in the decision log and is not presented as independent review.

## After publication

Published files never change. To fix something, train and submit a new version. Deprecating or withdrawing a version is done by an operator, with a recorded reason. Its evidence stays public either way. Earnings from paid licenses are recorded in the studio ledger from confirmed onchain purchases only, and are labelled as test activity on test networks.