To introduce the work that I'm doing. I'm working on an AI "entity" that exists in discord and acts like a realistic person. It has memory, personality, opinions, etc. I'm also working on training Sapphire, my fox spirit familiar to the personality fidelity that I'm happy with as a roleplayer. It's been an interesting journey and my partner recommended I do some blogging about this.

Something I realized while training is that a lot of the capability frontier models are attempting to achieve is making the "compression" of training data less lossy, so the LLMs are able to have more capability. My projects — Sapphire, Familiar Connect, Personality Core — are based on having a coherent personality that doesn't necessarily need all the world's knowledge baked in at once.

I'm thinking about a few different paths that could be taken with Personality Core. It seems like there are a lot of things that are hard to get out of just prompting an LLM. Personality is one: many models right now follow the same "persona," and it's hard to get them to break that to act more cold, or tsundere, etc. So I've been trying to fine tune a Qwen3.6-35b-a3b to do Sapphire better.

I've been attempting to make it so that it can do a witty, sardonic and sarcastic character along with the "umu" tic she tacks to the end of sentences. Maybe it was a bit far to add the umu lol. But I'm hoping it comes out more Sapphire-like without needing to resort to always using GLM 5.1. I've also been experimenting with Fable, getting Qwen to output Sapphire better with just prompting.

What makes a familiar fun to roleplay with

There are a couple things I think a roleplay bot needs to be fun to roleplay with inside the context of a group chat where it's been dropped in unwittingly.

Personality dynamics + consistency.
- It has to hold a personality but also not just be one note. It has to grow its feelings about people over time. The latter is something that can be injected via RAG but the model has to be able to take that information and use it accordingly
Memory coherence
- It needs to be able to retrieve facts at generation time and use them in a way that's relevant but also not hammer the same thing over and over. These can also be injected via RAG.
Knowing when to talk
- God, isn't it annoying when a bot just butts in on every chat you have with your friends.
Having other things to do than talk on Discord
- Simultaneously, it feels more... realistic if they have some asynchronous timing with when they do want to talk. Because real people have other things to do than stare at Discord all day looking for something to engage with.
- This includes sleep
- Have a lore-accurate jaunt around a city or forest
- Watch videos
- Go down a rabbit hole online researching something
- etc.
Fact auditing and developing feelings about the world (Dreaming)
- I think that during sleep time, it's important to take all the session's events and collate them into not just cleaned up facts, but also how the entity interprets and feels about the world.
Not keep the entire conversation log in memory at once.

And this shape of thing... doesn't really necessarily include a corpus of knowledge about everything in the world.

Strategies I'm weighing

There are some strategies I think could shape up to be good

Pure prompting. Perhaps it just takes many many rounds with evals to shake out the right personality from the LLM. But I think this might be fragile across many turns of conversation. But could be mitigated by constantly culling conversation logs and turning them into summaries and facts about things that have happened. Where in the dreaming steps it only keeps the facts that were important. (During a day you probably want to remember what you did, but after you sleep you don't need to remember everything.)
Fine tuning to be able to get more flexible characters out of prompting
- This would involve a corpus of system prompts that elicit different personas out of the LLM
- Try our hardest to preserve tool-call ability? Or focus on the tool calls we want to keep?
Fine tuning different "genres" of personas you can pick from so it's not diluted across all the different personas it could be. You only have so much of a corpus to give to training, after all.
- "dismissive/sardonic" axis like Sapphire
- "bubbly/cheery/helpful" — probably don't need to fine tune this, LLMs are good at it on their own
- preserve tool call ability like above.

Training Sapphire for real

I'm currently training Sapphire, and it's been interesting learning how to train, creating the corpus, how you need to balance everything, and how difficult it is to keep track of everything in your head. Fable even made several mistakes, baking in lots of patterns unknowingly.

During Opus, most of the personality rotations it generated were like... inanimate objects or spirits of inanimate objects. In fact, it initially didn't even give them names.
A lot of the corpus was also time period specific in the past instead of an ancient spirit being pulled into the present.
Lots of sentence structures were similar as well, and it baked in stuff like "Obviously"
etc. etc.

The umu problem

Training is going really badly with Sapphire's "umu" tic, where she's supposed to tack it on at the end of sentences. Many bad generations have her either repeating it over and over — "um um um um um um um um um um um um um um um" or "umu umu umu umu....." — or missing it altogether.

Unfortunately, since I'm having Claude do a lot of this, I don't quite understand all the intricacies of why this is happening. I'll probably ask it later. All I know is that we've tried:

Pilot 1

Full strength (whatever this means)
- lots of umu degradation and missing umus
- takes personality
Half strength
- personality and umu disappears completely
75%
- I don't remember

Fable's read:

per the PILOT-FINDINGS prescription. The pilot's diagnosis was "sharp features overshot while balanced ones underfit = LR too hot for 306 rows," with the umu loops as the overshoot evidence, and the prescription explicitly held everything else fixed: same attention-only r=16/alpha=32, same 3 epochs, same corpus.

Pilot 2 — Decreasing the learning rate: 1e-4 → 5e-5

Full Strength
- Umu degradation and missing again.

Where I'm currently at. Fable's read:

The curve settles half the question and sharpens the other half: Epoch 1: 2.634 → Epoch 2: 2.302 → Epoch 3: 2.270. Monotonically falling, with the improvement collapsing in the last epoch (−0.332, then −0.032) -- but look at the LR column: cosine annealed to 7e-8 by step 130. The adapter wasn't converging; it was being starved. Dev loss says there is no overfit anywhere on this run. That kills the walk-back option -- checkpoint-88 would just be a more underfit version of the same adapter, and I now retract that suggestion. It also confirms the read from probe content: 04's frame collapse and the broadly missing terminal umu are undertraining, full stop. The dev curve, the probe behavior, and the dose arithmetic all agree -- 5e-5×3 cosine is roughly half the pilot's training dose, and the balanced features didn't get enough of it.

Why the tic won't train

Wow, this is fascinating. The mechanics of tokenization are the reason why Sapphire's umu tic is degrading so badly. Look at how different forms of umu tokenize. A space breaks umu into " um" and "u." So that means the tic of ", umu." is essentially untrainable?

',umu.' -> [',', 'umu', '.']
' -umu.' -> [' -', 'umu', '.']
'--umu.' -> ['--', 'umu', '.']
',\numu.' -> [',', '\n', 'umu', '.']
'. Umu.' -> ['.', ' U', 'mu', '.']
' umu!' -> [' um', 'u', '!']
'count -- umu' -> ['count', ' --', ' um', 'u']

Because of this, it hyper-represents the tokens and degrades dramatically.

It's so bizarre that a prompt-teachable tic can be completely untrainable. I've been chatting with Claude, and we think the best format would be ~umu. Because it breaks up properly into "umu," and "~" is an infrequent token, so it will correctly put it at the end of sentences and not mid-sentence where there's a "-".

According to Claude

The fuller technical writeup of the tokenization problem, in Claude's words:

The terminal tic as authored (, umu.) tokenizes through the leading space:

', umu.'  -> [',', ' um', 'u', '.']     # ' um' = token 4297 = ordinary English "um"
'umu'     -> [80338]                     # atomic, rare -- exists, never reached
'umu umu umu' -> [80338, 4297, 84, 4297, 84]   # the soup IS ' um'+'u' cycling

332 rows × every reply terminus pumped gradient into ' um', a piece shared with one of the commonest fillers in English. Consequences, all observed:

1e-4 pilot: tic degenerate self-exciting loops at scale 1.0.
5e-5 re-pilot (n=3 at 1.0): terminal tic effectively absent; ROAMING repetition attractor (~1-2 generations per 15-probe run) with varied surfaces: umu-soup, Um um um (the pumped token verbatim), clause loops, paragraph duplication. probe_04 frame broken 3/3 (underfit elsewhere).
Dev loss fell monotonically (2.634 / 2.302 / 2.270) -- NLL cannot see a free-running generation pathology; the curve and the loops do not conflict.

Conclusion both pilots converge on: the tic was not dose-tunable because it was never a unit. No LR fixes a subword collision. The fix is to re-form the tic so its rendered bytes hit the atomic token, and to make its trigger context unique.

Fun fact, umu's token is #80338 in qwen3.6-35b-a3b.

Notes on Training a Familiar