Apple's model

Add Apple's on-device model to the model picker on macOS 27, through the fm serve command that ships with macOS.

By the end of this page the model button lists apple-on-device on macOS 27, and you chat with Apple's built-in model like any Ollama model. On Windows and on older macOS versions the option does not appear, and you can skip this page.

macOS 27 ships a small on-device model and a command called fm. fm serve starts a server that speaks the OpenAI Chat Completions format. The messages you send are the same as for Ollama. Only the answer format differs, so the app needs one more line parser and nothing else.

Try it by hand

fm serve --port 1976

In a second terminal:

curl http://127.0.0.1:1976/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"system","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Hi"}]}'

The answer arrives as server-sent events: one data: {...} line per piece, ending with data: [DONE]. With include_usage, the last useful line carries usage.completion_tokens, which gives the tokens per second. Stop the server with Ctrl+C.

Copy in the Apple module

git checkout step-2-apple -- src-tauri/src

This adds src-tauri/src/apple.rs and updates ollama.rs and main.rs.

Full file on GitHub: apple.rs

Start the server when you need it

src-tauri/src/apple.rs

pub fn available() -> bool {
    cfg!(target_os = "macos") && std::path::Path::new(FM).exists()
}

/// Start `fm serve` on a free port the first time it is needed.
fn port() -> Result<u16, String> {
    let mut server = SERVER.lock().unwrap();
    if let Some((_, port)) = server.as_ref() {
        return Ok(*port);
    }
    // Ask the OS for a free port, then hand it to fm.
    let port = TcpListener::bind("127.0.0.1:0")
        .and_then(|listener| listener.local_addr())
        .map_err(|e| e.to_string())?
        .port();
    let child = Command::new(FM)
        .args(["serve", "--host", "127.0.0.1", "--port", &port.to_string()])
        .stdin(Stdio::null())
        .stdout(Stdio::null())
        .stderr(Stdio::null())
        .spawn()
        .map_err(|e| format!("Could not start fm serve: {e}"))?;
    *server = Some((child, port));

    let deadline = Instant::now() + Duration::from_secs(10);
    while Instant::now() < deadline {
        if ureq::get(format!("http://127.0.0.1:{port}/health"))
            .call()
            .is_ok()
        {
            return Ok(port);
        }
        std::thread::sleep(Duration::from_millis(200));
    }
    Err("fm serve did not start".to_string())
}

available checks for /usr/bin/fm, so the option only exists where the command does. port asks the operating system for a free port, starts fm serve on it the first time you pick the model, and waits until /health answers.

Read a different stream

src-tauri/src/apple.rs

    for line in BufReader::new(response.into_body().into_reader()).lines() {
        if stop.load(Ordering::SeqCst) {
            let _ = events.send(ChatEvent::stopped());
            return Ok(());
        }
        let line = line.map_err(|e| e.to_string())?;
        let Some(data) = line.strip_prefix("data: ") else {
            continue;
        };
        if data == "[DONE]" {
            break;
        }
        let chunk: Value = serde_json::from_str(data).map_err(|e| e.to_string())?;
        if let Some(tokens) = chunk["usage"]["completion_tokens"].as_u64() {
            usage_tokens = Some(tokens);
        }
        let text = chunk["choices"][0]["delta"]["content"]
            .as_str()
            .unwrap_or("");
        if !text.is_empty() {
            first_token.get_or_insert_with(Instant::now);
            chunks += 1;
            let _ = events.send(ChatEvent::Text {
                delta: text.to_string(),
            });
        }
    }

The same loop as the Ollama one, with a different line format. Skip lines that do not start with data: , stop at [DONE], and send the text as text events on the same Channel. React cannot tell the two models apart.

fm reports no load time and no generation time. So the app measures the time from the first token to the last one itself, and divides completion_tokens by it.

Route Apple requests

ollama.rs adds apple-on-device at the end of the model list when it is available, and sends its requests to apple::stream_answer.

src-tauri/src/ollama.rs

    std::thread::spawn(move || {
        let result = if request.model == crate::apple::MODEL {
            crate::apple::stream_answer(&request, &on_event, &stop)
        } else {
            stream_answer(&request, &on_event, &stop)
        };
        if let Err(message) = result {
            let _ = on_event.send(ChatEvent::Error { message });
        }
    });

main.rs gets one extra line in the exit handler, apple::shutdown(), so the server stops when the app quits.

Check

pnpm tauri dev

On macOS 27, open the model button. apple-on-device is at the bottom of the list.
Pick it and ask a short question. The answer streams in, and the speed line shows its tokens per second.
Quit the app and check that the server stopped:

pgrep -fl "fm serve"

No output means it is gone.

Apple's model has a small context window of about 8,000 tokens and no thinking levels, so it suits short questions. It is not a good fit for Code mode.