Use your own model

Ollama serves open models on 127.0.0.1:11434 with an OpenAI-compatible API. Elyra treats it as an upstream: any OpenAI-compatible server works, but only localhost/LAN upstreams count as local/private. Hosted upstreams (OpenRouter, OpenAI) are labelled HOSTED · CLEAR.

Verified on Ubuntu 22.04 x86_64, CPU only, Ollama 0.40.14, model llama3.2:1b.

1. Install#

curl -fsSL https://ollama.com/install.sh -o ollama-install.sh
less ollama-install.sh            # read it first; it only downloads from ollama.com (and NVIDIA repos if a GPU is found)
sh ollama-install.sh
systemctl is-active ollama        # active
ss -ltnp | grep 11434             # 127.0.0.1:11434 only

The installer creates a systemd service. Keep Ollama on loopback; never expose port 11434.

2. Pull a model that fits your RAM#

ollama pull llama3.2:1b           # about 1.3 GB
curl -s http://127.0.0.1:11434/v1/models

Models live in /usr/share/ollama/.ollama/models by default. For another disk set OLLAMA_MODELS=/path in the service environment (systemctl edit ollama).

3. Smoke-test Ollama directly#

curl -s http://127.0.0.1:11434/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"llama3.2:1b","messages":[{"role":"user","content":"Reply with exactly: ok"}],"max_tokens":8}'

The first call loads the model into RAM: on the test machine (RAM nearly full) it took 82 s; the model itself needed 1.2 s. Warm calls are about one second.

4. Point Elyra at it#

Environment (see CONFIGURATION):

ELYRA_UPSTREAM_URL=http://127.0.0.1:11434/v1
ELYRA_UPSTREAM_NAME=ollama

or in the panel: Provide → add source. Then:

curl -s http://127.0.0.1:8545/v1/models                 # includes llama3.2:1b, owned_by "SELF - ollama"
curl -s http://127.0.0.1:8545/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"llama3.2:1b","messages":[{"role":"user","content":"Say hi in 3 words"}],"max_tokens":16}'

Result on the test machine: HTTP 200 in 1.0 s, and the response carries an elyra block: route: "self:SELF - ollama", simulated: false, tee: "none", price_per_mtok: 0 (your own model costs you nothing to call).

5. Sell it (optional)#

Setting ELYRA_PROVIDE=1, a public HTTPS URL and a price makes the node announce the model to peers. A non-TEE seller can read prompts, so it must never be labelled TEE (the node only claims TEE inside Confidential Space). Price per MTok must exceed your cost per MTok served. Start on a cheap CPU box; rent a GPU only after a buyer exists. Selling over the network was not tested with a real model yet; see STATUS.

Troubleshooting#

• Could not connect to a running Ollama instance: systemctl status ollama; on containers without systemd run ollama serve.
• Out-of-memory kills: pick a smaller model; check free -h while loaded. Ollama unloads idle models after a few minutes.
• Router shows the model but chat hangs: cold start, wait; or the model name in the request does not match ollama list.