Vision¶
Native multimodal support: send images and short video to the model - and see them in an on-demand preview using your terminal's graphics protocol.
Overview¶
NurCLI attaches workspace media on the Responses-style multimodal path (input_image /
input_video) when the active provider supports it. The model can see workspace images
and short video clips directly.
Pasted images stay in a compact composer attachment row until you send. F4 opens a preview and cycles through multiple images; Shift+F4 removes the previewed image (or the last attachment when no preview is open). Esc closes the preview. Enter can send an image-only message. After sending, a small attachment row offers click-to-peek; pixels never cover the running prompt header or consume a large transcript area. Images queued while a turn runs travel with that follow-up rather than entering the active request.
Previews use kitty, sixel, or iTerm2 graphics with a halfblocks fallback.
Image attachments and previews¶
| Input | What happens |
|---|---|
| Ctrl+V an image (screenshot, copied file) | Saved under .nur/media/paste/, staged in the composer, sent for vision with your message |
/image <path> |
Stages the workspace image in the composer |
look tool call |
Click the tool card to peek the full-size render |
| LaTeX equations | Rendered to PNG and shown inline in answer cards (existing behavior) |
Protocol selection: auto by default. Force one via config.toml:
[theme]
protocol = "kitty" # kitty | sixel | iterm2 | halfblocks | auto
inline_images = true # set false to disable all inline pixel rendering
Or per-launch env: NUR_IMAGE_PROTOCOL=kitty nur. The full capability probe
(kitty/sixel query + font size) is opt-in because it can block ~1 s on some
terminals: NUR_IMAGE_QUERY=1 nur.
Terminal support
Real pixels need a terminal that speaks one of the protocols - kitty, WezTerm, iTerm2, foot, Konsole, mintty, VS Code integrated terminal, rio. Windows Terminal / conhost fall back to unicode halfblocks art; everything else behaves exactly as before.
Tools¶
look¶
Attach workspace image(s) or a video so the model sees them on the next turn.
| Input type | Formats | Notes |
|---|---|---|
| Images | png, jpg, webp, gif | Direct attachment (no ffmpeg needed) |
| Video | mp4, webm, mov | mp4 accepted directly up to ~20 MB; webm/mov go through extract_frames |
Usage in TUI: The agent calls look automatically, or you can reference media paths in your prompt.
mp4 is accepted directly
Unlike other video formats, .mp4 files under ~20 MB are sent directly to the model via input_video without needing ffmpeg. For webm/mov or larger files, use extract_frames first.
extract_frames¶
Extract sparse keyframes from video via ffmpeg.
| Setting | Default |
|---|---|
| Frame rate | ~1 fps |
| Max frames | ~8 |
| Output | .nur/frames/<name>/ |
After extraction, look is auto-queued with the extracted frames.
Auto-attach¶
Media paths in your user prompt are automatically attached when the file exists in the workspace:
If demo.mp4 exists in your project, it is automatically sent to the model.
Design from video¶
A typical workflow for extracting design tokens from a reference clip:
-
Short video (< 20 MB): Reference it directly in your prompt
-
Longer video: Extract frames first, then reference them
-
Manual control: The agent will use
extract_frames→ inspect stills → implement using design-eng skills
Best practices
- Prefer sparse frames over frame-by-frame
- Longer / huge videos: extract frames first; don't
looka giant file extract_framesrequires ffmpeg on PATH (check withnur doctor)lookstill works on short videos and images without ffmpeg
Requirements¶
| Tool | Requires |
|---|---|
look |
Nothing extra for images; ffmpeg optional for short video |
extract_frames |
ffmpeg on PATH |
Check vision readiness:
Install ffmpeg: