Minimal shell with tab completion

[MDr164]

Hi everyone,
I'm already pretty amazed by what this library can do and I found out about it via Task. For a different project we were looking for a small shell library and this one checks all the boxes. The only feature the proof-of-concept implementation in this repo lacks is basic tab completion. We do have a small library for that in our project but making it work with this one seems to be harder than expected. Is there a recommendation on where to hook into in the shell to enable tab completion or maybe it already exists here and I just missed it? Any ideas or help are greatly appreciated!

[mvdan]

Do you mean https://github.com/mvdan/sh/blob/master/cmd/gosh/main.go, or the internal libraries like syntax and interp?

The libraries don't implement the user-facing shell with the prompt its features like readline or tab completion, so in terms of compatibility, they should be fine - you implement a "full shell" with those libraries as building blocks, much like cmd/gosh does.

As far as cmd/gosh, like you said, it's a bit of a proof of concept and testing tool rather than a complete shell. I think it's a bit overkill to pull in heavier dependencies like readline or tab completion into this module. But writing such a complete shell in a third-party module would certainly be useful.

I would suggest to take cmd/gosh as a starting point, but to simply copy the code and adapt it as needed. It's not meant to be a library in any way, but rather a pretty simple and short package main implementation.

[MDr164]

Ah yes I should have clarified further. So I took gosh/main.go as a starting point and started modifying runInteractive() in there to accommodate for that extra feature. Our codebase already has a package for readline and completion that I wanted to use https://github.com/u-root/u-root/tree/main/pkg/complete which should suffice. When talking about the library of sh I meant syntax and interp for that matter. Generally this shell is meant to be packaged into an initramfs and not as a full on runtime shell so having something that can parse shell syntax and be as small as your sh library is perfect. It's just that some people there want tab completion for that matter.

So to sum up: Your recommended way to go here is take gosh/main.go as a starting point and implement tab completion in e.g. runInteractive() but not make use of syntax.Parser.Interactive but rather do that with external code and use interp.Runner on the output, right?

[mvdan]

So to sum up: Your recommended way to go here is take gosh/main.go as a starting point and implement tab completion in e.g. runInteractive() but not make use of syntax.Parser.Interactive but rather do that with external code and use interp.Runner on the output, right?

That sounds about right. I'm not sure whether or not using Parser.Interactive or its lower level sibling, Parser.Stmts, is going to be a good idea or not. With Parser.Stmts as an example, you only get callbacks when a) a statement is fully parsed, and b) read calls come in.

If your "complete" package wants to own the input reader, then that seems at odds with our Interactive and Stmts APIs. You could instead use your complete API to handle input while the user is typing or hitting the tab key, and when the user hits "enter" to input a whole line, pass that line to Parser.Stmts. That will either tell you more input is needed (then your interactive shell handles this with a "continue" prompt), or it tells you the input is a series of complete statements, in which case you call the interpreter to run them.

That logic is roughly what Parser.Interactive does. It's really just a bit of glue code over Parser.Stmts with a custom reader middle-man to track input lines. See

sh/syntax/parser.go

Lines 182 to 257 in ac775df

	type wrappedReader struct {
	*Parser
	io.Reader
	lastLine int
	accumulated []*Stmt
	fn func([]*Stmt) bool
	}
	func (w *wrappedReader) Read(p []byte) (n int, err error) {
	// If we lexed a newline for the first time, we just finished a line, so
	// we may need to give a callback for the edge cases below not covered
	// by Parser.Stmts.
	if (w.r == '\n' || w.r == escNewl) && w.line > w.lastLine {
	if w.Incomplete() {
	// Incomplete statement; call back to print "> ".
	if !w.fn(w.accumulated) {
	return 0, io.EOF
	}
	} else if len(w.accumulated) == 0 {
	// Nothing was parsed; call back to print another "$ ".
	if !w.fn(nil) {
	return 0, io.EOF
	}
	}
	w.lastLine = w.line
	}
	return w.Reader.Read(p)
	}
	// Interactive implements what is necessary to parse statements in an
	// interactive shell. The parser will call the given function under two
	// circumstances outlined below.
	//
	// If a line containing any number of statements is parsed, the function will be
	// called with said statements.
	//
	// If a line ending in an incomplete statement is parsed, the function will be
	// called with any fully parsed statents, and Parser.Incomplete will return
	// true.
	//
	// One can imagine a simple interactive shell implementation as follows:
	//
	// fmt.Fprintf(os.Stdout, "$ ")
	// parser.Interactive(os.Stdin, func(stmts []*syntax.Stmt) bool {
	// if parser.Incomplete() {
	// fmt.Fprintf(os.Stdout, "> ")
	// return true
	// }
	// run(stmts)
	// fmt.Fprintf(os.Stdout, "$ ")
	// return true
	// }
	//
	// If the callback function returns false, parsing is stopped and the function
	// is not called again.
	func (p *Parser) Interactive(r io.Reader, fn func([]*Stmt) bool) error {
	w := wrappedReader{Parser: p, Reader: r, fn: fn}
	return p.Stmts(&w, func(stmt *Stmt) bool {
	w.accumulated = append(w.accumulated, stmt)
	// We finished parsing a statement and we're at a newline token,
	// so we finished fully parsing a number of statements. Call
	// back to run the statements and print "$ ".
	if p.tok == _Newl {
	if !fn(w.accumulated) {
	return false
	}
	w.accumulated = w.accumulated[:0]
	// The callback above would already print "$ ", so we
	// don't want the subsequent wrappedReader.Read to cause
	// another "$ " print thinking that nothing was parsed.
	w.lastLine = w.line + 1
	}
	return true
	})
	}

.

[mvdan]

As an addition to the comment above - you will want to at least use Parser.Stmts I think, as it's the only parser API that allows you to parse a series of statements. Parser.Parse does a whole file, which is similar, but not really - it will just error if the input is incomplete, for instance.

[MDr164]

Great, thanks!