---
title: A model from scratch
sidebar_label: 1. A model from scratch
description: "Write an embedding model's checkpoint by hand (config, tokenizer, random weights), load it through the registry, and compare two texts, all offline."
---

This series is for adding your own model, as opposed to running the catalog's. It walks the same ground the [first tutorial](../first-model/01-pick-a-model.md) covered from the consumer side, now from the author's side, in three parts with a running result each. The worked model is an embedding model on purpose: text goes in, one vector comes out, and there is no machinery beyond that idea. This part demystifies the checkpoint itself: a model is a directory of files, and you can write one by hand.

<!-- CERTIFICATION: the program compiles and runs against the pinned release (linux x86_64; the pooling helpers are
clika_modelverse/modules/pooling.h); the output block is that run's capture under the fixed seed (another platform's normal_distribution may vary the values). -->

The program below synthesizes a tiny embedding checkpoint (an 8-word vocabulary, hidden size 32, a gemma3-shaped text encoder), loads it through the registry, and compares three texts. Nothing downloads and the weights are random; random weights still embed, they are only bad at it, and that is enough to see every file a model needs and where each one enters. Same two-file CMake project as [part 4 of the first tutorial](../first-model/04-use-it-from-code.mdx); only `main.cpp` changes.

## The files an embedding model needs

- `config.json` names the architecture (`model_type`, `architectures`) and its geometry (layers, heads, hidden size). Identity resolution reads exactly this.
- `tokenizer.json` turns text into token ids; the toy one below is a word-level vocabulary of eight entries.
- The weights (`model.safetensors` here) are tensors under the names the architecture expects.
- An embedding checkpoint additionally ships its module chain, the sentence-transformers layout: `modules.json` lists the stages (transformer, pooling, dense heads, normalize), and each configurable stage carries its own small directory.

## The program

```cpp title="main.cpp"
#include <cstdio>
#include <filesystem>
#include <fstream>
#include <random>
#include <string>
#include <vector>

#include <ClikaRT/clika_rt.h>

#include "clika_modelverse/modules/pooling.h"
#include "clika_modelverse/registry/model_registry.h"

namespace fs = std::filesystem;
namespace mv = clika_modelverse;
using ClikaRT::DataType;
using ClikaRT::Tensor;

// The whole identity: model_type and architectures are what the registry
// matches; the rest is geometry the loader shapes the encoder from.
constexpr const char* kConfigJson = R"({
  "model_type": "gemma3_text",
  "architectures": ["Gemma3TextModel"],
  "hidden_size": 32,
  "num_hidden_layers": 4,
  "num_attention_heads": 4,
  "num_key_value_heads": 2,
  "head_dim": 8,
  "vocab_size": 8,
  "intermediate_size": 64,
  "max_position_embeddings": 128,
  "rms_norm_eps": 1e-6,
  "rope_theta": 1000000.0,
  "rope_local_base_freq": 10000.0,
  "sliding_window": 16,
  "sliding_window_pattern": 2,
  "query_pre_attn_scalar": 8,
  "use_bidirectional_attention": true
})";

// A word-level tokenizer: eight words, one id each.
constexpr const char* kTokenizerJson = R"({
  "version": "1.0",
  "pre_tokenizer": {"type": "WhitespaceSplit"},
  "model": {"type": "WordLevel",
            "vocab": {"a": 0, "b": 1, "c": 2, "d": 3, "e": 4, "f": 5, "g": 6, "[eos]": 7},
            "unk_token": "a"},
  "decoder": null,
  "added_tokens": [{"id": 7, "content": "[eos]", "special": true, "normalized": false}]
})";

// The module chain: transformer -> mean pooling -> two dense heads -> normalize.
constexpr const char* kModulesJson = R"([
  {"idx": 0, "name": "0", "path": "", "type": "sentence_transformers.models.Transformer"},
  {"idx": 1, "name": "1", "path": "1_Pooling", "type": "sentence_transformers.models.Pooling"},
  {"idx": 2, "name": "2", "path": "2_Dense", "type": "sentence_transformers.models.Dense"},
  {"idx": 3, "name": "3", "path": "3_Dense", "type": "sentence_transformers.models.Dense"},
  {"idx": 4, "name": "4", "path": "4_Normalize", "type": "sentence_transformers.models.Normalize"}
])";

constexpr const char* kPoolingJson = R"({
  "word_embedding_dimension": 32,
  "pooling_mode_cls_token": false,
  "pooling_mode_mean_tokens": true,
  "pooling_mode_max_tokens": false,
  "pooling_mode_mean_sqrt_len_tokens": false,
  "pooling_mode_weightedmean_tokens": false,
  "pooling_mode_lasttoken": false
})";

Tensor randn(std::mt19937& rng, std::vector<std::int64_t> shape) {
    std::int64_t n = 1;
    for (const std::int64_t d : shape) n *= d;
    std::normal_distribution<float> dist(0.0f, 0.2f);
    std::vector<float> host(static_cast<std::size_t>(n));
    for (float& v : host) v = dist(rng);
    return Tensor::from_data(host.data(), shape, DataType::Float32);
}

// One dense head: its own directory with a config and one weight.
void write_dense(std::mt19937& rng, const fs::path& dir, std::int64_t in,
                 std::int64_t out) {
    fs::create_directories(dir);
    std::ofstream(dir / "config.json")
        << R"({"in_features": )" << in << R"(, "out_features": )" << out
        << R"(, "bias": false, "activation_function": "torch.nn.modules.linear.Identity"})";
    ClikaRT::NamedTensors sd;
    sd.set("linear.weight", randn(rng, {out, in}));
    ClikaRT::io::save_safetensors(sd, (dir / "model.safetensors").string());
}

// Random weights under the names the gemma3-shaped encoder expects (bare
// keys, no prefix); the fixed seed keeps every run identical.
fs::path write_snapshot() {
    const fs::path dir = fs::temp_directory_path() / "my_first_model";
    fs::create_directories(dir);
    std::ofstream(dir / "config.json") << kConfigJson;
    std::ofstream(dir / "tokenizer.json") << kTokenizerJson;
    std::ofstream(dir / "modules.json") << kModulesJson;
    fs::create_directories(dir / "1_Pooling");
    std::ofstream(dir / "1_Pooling" / "config.json") << kPoolingJson;

    constexpr std::int64_t kVocab = 8, kHidden = 32, kLayers = 4, kHeads = 4;
    constexpr std::int64_t kKvHeads = 2, kHeadDim = 8, kFfn = 64;
    constexpr std::int64_t kDenseMid = 48, kDim = 16;
    std::mt19937          rng(20260831);
    ClikaRT::NamedTensors sd;
    const auto put = [&](const std::string& name, std::vector<std::int64_t> shape) {
        sd.set(name, randn(rng, std::move(shape)));
    };
    put("embed_tokens.weight", {kVocab, kHidden});
    put("norm.weight", {kHidden});
    for (std::int64_t l = 0; l < kLayers; ++l) {
        const std::string p = "layers." + std::to_string(l) + ".";
        put(p + "self_attn.q_proj.weight", {kHeads * kHeadDim, kHidden});
        put(p + "self_attn.k_proj.weight", {kKvHeads * kHeadDim, kHidden});
        put(p + "self_attn.v_proj.weight", {kKvHeads * kHeadDim, kHidden});
        put(p + "self_attn.o_proj.weight", {kHidden, kHeads * kHeadDim});
        put(p + "self_attn.q_norm.weight", {kHeadDim});
        put(p + "self_attn.k_norm.weight", {kHeadDim});
        put(p + "mlp.gate_proj.weight", {kFfn, kHidden});
        put(p + "mlp.up_proj.weight", {kFfn, kHidden});
        put(p + "mlp.down_proj.weight", {kHidden, kFfn});
        put(p + "input_layernorm.weight", {kHidden});
        put(p + "post_attention_layernorm.weight", {kHidden});
        put(p + "pre_feedforward_layernorm.weight", {kHidden});
        put(p + "post_feedforward_layernorm.weight", {kHidden});
    }
    ClikaRT::io::save_safetensors(sd, (dir / "model.safetensors").string());
    write_dense(rng, dir / "2_Dense", kHidden, kDenseMid);
    write_dense(rng, dir / "3_Dense", kDenseMid, kDim);
    return dir;
}

int main() {
    // 1. A checkpoint is a directory; this one is yours, written just now.
    const fs::path dir = write_snapshot();
    std::printf("wrote %s\n", dir.string().c_str());

    // 2. The registry reads config.json, matches architecture
    //    "Gemma3TextModel" to the gemma-embedding family, and returns the
    //    runnable model, exactly as for a fetched checkpoint.
    mv::EmbeddingModel model =
        mv::ModelRegistry::builtin().load_embedding(dir.string(), {});

    // 3. Embed three texts: one [3, 16] Float32 tensor, one row per text
    //    (the module chain pools, projects to 16 dims, and normalizes).
    const std::string texts[] = {"a b c", "a b d", "f g"};
    const Tensor rows = model.embed(texts);
    std::printf("embeddings: %s\n", rows.to_string().c_str());

    // 4. Compare them: the cosine matrix is unit rows times their own
    //    transpose. The explicit normalize keeps the demo self-contained
    //    (the chain's Normalize stage already produced unit rows).
    const Tensor unit = mv::modules::l2_normalize_rows(rows);
    const Tensor sim = ClikaRT::ops::matmul(unit, unit, {}, std::nullopt,
                                            false, /*transpose_b=*/true);
    const std::vector<float> s = ClikaRT::ops::reshape(sim, {9}).item_as_vec<float>();
    std::printf("close pair  (a b c ~ a b d): %.3f\n", s[1]);
    std::printf("far pair    (a b c ~ f g):   %.3f\n", s[2]);
    return 0;
}
```

```bash
cmake -S . -B build -DModelverse_DIR="$MODELVERSE_INSTALL_DIR/cmake"
cmake --build build
./build/hello
```

```text
[2026-09-22 00:30:20.206] [modelverse] [info] hub: '/tmp/my_first_model' already on disk; loading from /tmp/my_first_model
[2026-09-22 00:30:20.208] [gemma-embedding] [info] embedding encoder on CPU:-1 (dim 16)
wrote /tmp/my_first_model
embeddings: Tensor(shape=[3, 16], dtype=Float32, device=CPU, numel=48, data=[0.03671, -0.04182, -0.2858, 0.1808, 0.1262, 0.399, ...])
close pair  (a b c ~ a b d): 0.400
far pair    (a b c ~ f g):   -0.324
```

Random weights, so the numbers mean little; what matters is the shape of what happened. A directory you wrote from scratch went through the same registry, loader and embedding surface as a fetched checkpoint would, and three texts became three vectors you can compare, because a checkpoint is nothing more than these files.

## What the registry did with it

`config.json`'s `model_type` and `architectures` are the identity keys; the architecture `Gemma3TextModel` matched the built-in `gemma-embedding` family, and that family's loader shaped the encoder from the geometry fields and bound your `model.safetensors` names to it. The tokenizer file turned each text into ids before the encoder saw them, and `modules.json` told the loader what follows the encoder: mean pooling, the two dense projections, and the final normalize. Delete the directory and nothing else remembers it.

Your own architecture will not say `"architectures": ["Gemma3TextModel"]`, and then the match fails; that refusal, and fixing it by registering a family of your own, is [part 2](02-join-the-catalog.md).