Standard Headless Widgets

[viridia]

Games thrive on novelty: nearly every game ever published has a unique artistic style. Unfortunately, this often results in developers having to re-implement basic UI components such as sliders, checkboxes, and spinners, in order to get the distinctive look that they want. Even worse, building robust, high-quality, cross-platform widgets requires knowledge of accessibility technologies, input mapping techniques, and other esoteric concerns, which many developers may not be experts in.

Headless Libraries

In the web world, this problem is solved by the a number of "headless" UI component libraries which provide high-quality widget implementations with no built-in styling, such as Headless UI and ReaKit. Users can easily add their own custom look and feel on top of these widgets.

Bevy has very recently gained a number of new capabilities (such as the input_focus crate) which facilitates the construction of headless widgets. The next logical step is to build out a selection of a few curated widgets. I refer to these as "headless" or "core" widgets.

Even though the world of UI has thousands of different kinds of widgets, the headless libraries mentioned previously have less than a dozen widgets in them. The reason is because what often distinguishes one kind of widget from another is styling. Since the headless widget library isn't concerned with styling, it does not need to distinguish widgets by style.

Also, many standard widgets are assemblies of smaller widgets: for example, a calendar widget is composed of many buttons. The headless widget libraries generally only provide low-level "atomic" components.

Accessibility Support

A key value of headless widgets is first-class support for accessibility. In Bevy, accessibility is provided by the bevy_a11y crate, which uses AccessKit to integrate with the platform's native OS capabilities for things like speech output. However, these features are not enabled unless you include the proper metadata. For example, when a widget is in a state like disabled or checked, adding in the proper metadata will allow the screen reader to say the words "disabled" or "checked" when the widget is focused.

Unfortunately, most developers aren't familiar with these technologies, many do not even know how to enable the screen reading feature integrated in their OS for testing.

By ensuring that the core widgets fully support the accessibility APIs, we can make it so that developers get most of the benefits "for free".

Platform support

With respect to widget design, platforms fall into three classes:

• Desktop
• Console
• Mobile

Desktop widgets are generally driven by mouse events and keypresses; console widgets by gamepad buttons, and mobile by touch events.

Some kinds of widgets are more "universal" than others. For example, a button or toggle widget works pretty much the same everywhere; the main difference is the type of event used to trigger it. The existence proof here is the standard HTML input fields: you can use <button> on both desktop and mobile, and it will adhere to platform conventions.

However, some widgets are quite different on different platforms. Take for example text input: on a console or mobile device this normally happens by popping up some auxiliary keyboard interface which is part of the OS. Unlike a desktop the text widget doesn't accept keystrokes.

These kinds of behaviors can likely be controlled by feature flags. In rare cases, a widget may not be available on all platforms.

Technical Requirements

Framework Agnostic

It is envisioned that the core widgets will not depend on any particularly templating system or reactive framework. They will be vanilla Bevy UI components that use observers and bubbling events. Keyboard shortcuts will go through the FocusInput mechanism.

Use of observers

Core widgets are implemented using a set of global observers - that is, we don't register a separate observer for each widget instance. There's a plugin which registers the observers (perhaps a separate plugin for each type). Alternatively, we might be able to use register_component_hooks to automatically set up the observers the first time the user inserts the component.

Component States

Widget states will be represented using components. Dynamic styles can be implemented by the user via normal Bevy change detection on the state components.

Controlled vs. Uncontrolled

To maximize flexibility, widgets will be "controlled" rather than "uncontrolled". This is a term from React; a controlled widget does not update its own state, but simply emits events containing the proposed new state, which the receiver can choose to ignore. For example, a checkbox widget, when clicked, emits an event with the new proposed checkbox state, but does not update itself. It's up to the parent component to update the component with the new state.

The choice between controlled and uncontrolled is somewhat of a personal preference, but most professional UI developers I have talked to prefer controlled widgets. These have several advantages:

• It's easier to integrate them with reactive frameworks or external state management.
• It's easier to do input validation if the receiver can simply ignore the proposed change, or flash an error indicator in response to the event.
• It's easier to programmatically change the state, or synchronize state between multiple widgets, if the state is not buried inside the widget.
• It's simple to transform a controlled widget into an uncontrolled one by adding an event handler which applies the proposed state.

However, controlled widgets do have a downside in that they are not self-sufficient, which may confuse novice programmers.

Hover Agnosticism

Hover states: in general, hover highlights are purely stylistic, which means that the headless library need not concern itself with hovering. Now, normally in headless libraries we do have to concern ourselves with "roll-off" behavior - that is, if you click on a button and then move the mouse outside of the button before releasing, there is no "click" event emitted. However, in the case of Bevy, the picking crate already handles this behavior for us, so we don't actually need to do anything.

As a result, the core widgets have no need to integrate with the hover status provided by picking. That is purely up to the style layer provided by the user.

Focus Behavior

Most widgets support shortcut keys when focused.

On desktop, widgets will set focus to themselves when clicked. They will also set the FocusVisible resource to false to hide the focus rect. Console games will generally ignore this, instead showing the focus indicator unconditionally.

Disabled states

Widgets can be disabled by adding an InteractionDisabled marker component. Disabled widgets do not respond to user input, but they can still have keyboard focus - this is important for accessibility, otherwise sight-impaired users cannot "see" the widget.

An entity hook will ensure that the a11y "disabled" metadata is kept up to date.

Widget Types

Widgets will be divided into a number of tiers or cohorts, with the first tier consisting of widgets that are (a) very common, and (b) easy to implement.

Tier 1

• Button - the standard push button. On desktop, Enter and Space will trigger the button when focused.
• Toggle - for checkboxes and toggle switches. On desktop, Enter and Space will toggle the button when focused.
• Choice Spinner - this is a widget that offers a set of choices which you can select by incrementing or decrementing, for example game difficulty: < [ veteran ] >. The value may be edited by pressing buttons or by dragging. On console, this will be edited by the direction buttons, whereas on desktop arrow keys will change the value.
• Number Spinner - like the choice spinner but allows editing of a numeric value, within a range from min to max.

Tier 2

• Slider - a very basic slider that consists of a thumb and a track. More complex sliders which have increment / decrement arrows are built on top of this.
• Scrollbar - similar to a slider, but uses different math (has no min property, but does have a span or visible_range property).
• Text Input - on desktop, this would (hopefully) use the cosmic-edit library to support full-featured editing by the keyboard. On console and mobile, this would use the OS keyboard.

Tier 3

• Disclosure - a toggle widget which shows or hides some other UI content.
• Radio - radio buttons are in the third tier because they have a complex focus behavior (according to the WAI authoring guidelines) and are a challenge. They are also not common in games, as they take up a lot of screen real-estate.
• List Box - lists have similar focus behavior to radio buttons, but are quite common in games.
• Menu / MenuItem - menus are a challenge due to the need for popping up and intelligent positioning of the popup. (This will be easier once we have support for Position::Fixed.)

Open Questions

• One piece of a11y information that cannot be derived automatically is the alt-text, that is, the text that should be read to the user when that widget is focused. Should we emit a warning when a widget lacks alt-text? Maybe as an opt-in feature?
• On console, widgets generally will use the platform conventions for button mapping. For example on Xbox, the A button will trigger a button or toggle. Focus navigation will be determined by the focus system, so the widget need not concern itself with which button changes focus. However, what about the spinner increment / decrement? Which game pad button will we use for that?
  • The obvious answer here is to use an input mapper like LWIM. However, the current thinking is that the event mapper is the last element in the event chain - focused widgets get first crack at button presses before the event mapper gets to see them. The reason for this is to allow widgets such as text input to override keystrokes that might be mapped to global shortcuts.
  • One solution might be to do this using a double-mapping: the widget allows direction button presses to pass through to the event mapper, the event mapper then maps these to "increment / decrement" events, and then re-dispatches the events to the focus widget. With observers, this shouldn't introduce an additional 1-frame delay.
• Where should these widgets live? In bevy_ui? Or a new crate such as bevy_ui_headless or bevy_ui_widgets?
• Even though the functionality of checkbox and toggle switch are identical, they have different ARIA roles ("checkbox" vs. "switch"). This suggests that they should be separate widgets rather than combined into a single "Toggle" widget type.