termwiz: Given raw output from a pty, how can I apply it to a Surface?

[rgwood-dd]

What Operating System(s) are you running on?

macOS

Which Wayland compositor or X11 Window manager(s) are you using?

No response

WezTerm version

wezterm 20240203-110809-5046fc22

Ask your question!

Hi! I am trying to do the following with the help of the excellent termwiz crate:

1. Run a shell in a pty
2. Parse the output of that shell into a stream of discrete VTE events
3. Render the output of that process to a Surface (so I can do interesting things with it)

I'm able to do 1 with portable-pty and 2 with a Parser, but I'm struggling with 3. Specifically: the main way of interacting with Surface is applying Change structs to it, but I don't have Change structs - I have the stream of raw bytes, and a stream of Actions from the parser.

I could probably roll my own code to convert Actions to Changes, but I suspect I'm probably missing an easier way. Given raw output from a pty, is there an easy way to apply that output to a Surface?

[wez]

There is no easy way to do this, because you need a terminal emulator to parse the bytes and compute the the model for the display, in order to apply that to a Surface.

You could use wezterm's term crate to run a terminal emulator, but note that it is not published to crates.io because its API is not intended to be public and has no API stability guarantees outside of its use inside wezterm itself. To use it, you would need to reference it in your Cargo.toml using a git repo reference with rev set to the git commit hash you are using, so that you are pinned to a known version.

In addition, there is no canned transform from the term crate to Surface, but you can access Lines and do some processing of your own to adapt those into a Surface.

[rgwood-dd]

Thank you for the prompt response! The term crate and a terminal emulator may work for my needs; I'll take a closer look at that.

[bolinfest]

I think I have roughly the same question: I want to spawn a shell and running in a pty and get what is currently on screen as a string (or some sort of buffer). Using pyte, this basically works:

import select

import ptyprocess
import pyte

screen = pyte.Screen(80, 8)  # 80x8 terminal size
stream = pyte.Stream(screen)

process = ptyprocess.PtyProcessUnicode.spawn(["zsh"])
process.write("ls -l\n")

while True:
    if process.isalive() and process.fd in select.select([process.fd], [], [], 0)[0]:
        output = process.read(1024)
        stream.feed(output)

        cursor_position = screen.cursor
        for y, line in enumerate(screen.display):
            if y == cursor_position.y:
                line = line[: cursor_position.x] + "█" + line[cursor_position.x + 1 :]
            print(f"{y}: {line}")

        print("---")

That is, if I create this as main.py in /tmp/pyteex and run it, I see:

0: ls -l
1: █
2:
3:
4:
5:
6:
7:
---
0: ls -l
1: █
2:
3:
4:
5:
6:
7:
---
0: ls -l
1: /tmp/pyteex % ls -l
2: █
3:
4:
5:
6:
7:
---
0: ls -l
1: /tmp/pyteex % ls -l
2: total 8
3: █
4:
5:
6:
7:
---
0: ls -l
1: /tmp/pyteex % ls -l
2: total 8
3: -rw-r--r--  1 mbolin  wheel  713 Aug 16 15:55 main.py
4: █
5:
6:
7:
---
0: ls -l
1: /tmp/pyteex % ls -l
2: total 8
3: -rw-r--r--  1 mbolin  wheel  713 Aug 16 15:55 main.py
4: █
5:
6:
7:
---
0: ls -l
1: /tmp/pyteex % ls -l
2: total 8
3: -rw-r--r--  1 mbolin  wheel  713 Aug 16 15:55 main.py
4: /tmp/pyteex % █
5:
6:
7:
---

I was hoping to do the equivalent in Rust, though I assume the answer is the same as the one for @rgwood-dd, which is I need to use the term internals to do this?

[bolinfest]

Reading https://github.com/wez/wezterm/blob/30345b36d8a00fed347e4df5dadd83915a7693fb/term/README.md, I get the impression I could approximate the above by using advance_bytes and std::io::Write as described in the README and then rendering the view via TerminalState.screen?

[wez]

Yeah, term.advance_bytes() to send the data into the terminal, which will parse it and apply to its model. Then term.screen() (term derefs to TerminalState) and you can then use the screen's methods to iterate over the lines.

For something in a similar ballpark to the pyte example you showed (in terms of extracting the screen data), take a look at the TestTerm implementation in the unit tests.
This particular method is used to assert that the screen contents match expectations:

wezterm/term/src/test/mod.rs

Lines 291 to 302 in 30345b3

	/// Asserts that the visible lines of the terminal have the
	/// same character contents as the expected lines.
	/// The other cell attributes are not compared; this is
	/// a convenience for writing visually understandable tests.
	fn assert_visible_contents(term: &Terminal, file: &str, line: u32, expect_lines: &[&str]) {
	print_visible_lines(&term);
	let screen = term.screen();
	let expect: Vec<Line> = expect_lines.iter().map(|s| (*s).into()).collect();
	assert_lines_equal(file, line, &screen.visible_lines(), &expect, Compare::TEXT);
	}

[wez]

(edited: I pasted the wrong link previously)

[bolinfest]

Thanks, I didn’t have a chance to post an update, but I got something roughly equivalent to this working late last night, though visible_lines and some other helpful methods for my purposes are limited to #[test]:

wezterm/term/src/screen.rs

Lines 346 to 347 in 30345b3

	#[cfg(test)]
	pub fn visible_lines(&self) -> Vec<Line> {

I ended up trying for_each_phys_line_mut for now since it’s public and I didn’t want to mess with forks/branches yet:

wezterm/term/src/screen.rs

Line 970 in 30345b3

pub fn for_each_phys_line_mut<F>(&mut self, mut f: F)

I was also a bit confused because the initial ls command I sent appeared twice: once before the prompt and once after. I now realize that this was likely because I was writing to zsh before it was “ready” (definitely too much stuff in my .zshrc), so I guess I have to poll the output until it looks like the shell is “ready” unless there’s a signal or something else I can use instead of heuristics based on the presence of % and the cursor position?

Admittedly the command still gets run if i don’t wait: it just looks a little funny.

[wez]

re: your command showing up multiple times, that sounds like local echo is turned on at the start, then gets turned off when zsh is ready to read your prompt.

Rather than scraping the output, you can probably poll the termios to see when local echo is disabled. If you're using portable-pty for this, then this is the method to use:

https://docs.rs/portable-pty/latest/portable_pty/trait.MasterPty.html#method.get_termios

and I believe that local echo lives in the local flags; there are a couple of related flags and I didn't verify which is which:
https://docs.rs/nix/0.25.1/nix/sys/termios/struct.Termios.html#structfield.local_flags
https://docs.rs/nix/0.25.1/nix/sys/termios/struct.LocalFlags.html#associatedconstant.ECHO

this technique will likely work with any shell with any output, but the downside is that I don't think there is a mechanism to sleep and have the kernel wake you up when the termios change, so I think you're faced with a choice between inelegantly screen scraping, or inelegantly polling for this flag change.

[bolinfest]

This seems to work, though it's possible there's a race condition since I'm only checking after invoking advance_bytes() rather than on an interval?

In the example, term is a mut wezterm_term::Terminal. As discussed above, this crate is not published to crates.io, so one must do something like the following in order to implement a create_term() function to construct an instance of it:

[dependencies]
wezterm-term = { git = "https://github.com/wez/wezterm", rev = "30345b36d8a00fed347e4df5dadd83915a7693fb" }

Test for change to local echo:

let mut reader = pair.master.try_clone_reader()?;
let mut term: wezterm_term::Terminal = create_term(*rows as usize, *cols as usize);
let mut buf = [0_u8; 1024];
loop {
    let n = reader.read(&mut buf)?;
    term.advance_bytes(&buf[..n]);

    // Poll until local echo is disabled before handing over the terminal,
    // as we use that as a heuristic to determine when the shell is ready
    // for user input:
    // https://github.com/wez/wezterm/discussions/5217
    let termios = pair
        .master
        .get_termios()
        .ok_or_else(|| anyhow!("failed to read termios"))?;
    let is_echo = termios
        .local_flags
        .contains(nix::sys::termios::LocalFlags::ECHO);
    if !is_echo {
        // The terminal is ready.
        break;
    }
}

[wez]

it's probably fine, but it does make me feel uneasy as well.

If you did want to periodically check the termios independent of the blocking read there's some more effort required because the pty crate doesn't expose an async interface: the Windows pty documentation explicitly states that non-blocking (or rather, overlapped) IO is not supported, and we'd need some kind of runtime to to cater for either/both the posix and windows approaches. I'm avoiding forming an opinion about which runtime to choose from in the pty crate because none of the existing major ones work for wezterm.

If you're using tokio you probably want to wrap the reader.read() in a tokio::spawn_blocking call.

If you're not using async/await, the approach taken in wezterm is to spawn the reads into another thread that sends the buffers over to your terminal emulation thread and then use something like https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html#method.recv_timeout to let you tick over the reads. Wezterm actually does something a lot more complicated than this to produce a series of Actions that are pre-parsed and fed into the terminal instead of the raw bytes, but the high level idea is the same. It's probably not necessary for you to copy what wezterm is doing, I'm sharing this just in case you find yourself going even further down the rabbit hole:

wezterm/mux/src/lib.rs

Lines 268 to 272 in 30345b3

	/// This function is run in a separate thread; its purpose is to perform
	/// blocking reads from the pty (non-blocking reads are not portable to
	/// all platforms and pty/tty types), parse the escape sequences and
	/// relay the actions to the mux thread to apply them to the pane.
	fn read_from_pane_pty(