Implement Mesh Shader support for Rendering Device Vulkan and DirectX12

[thimenesup]

Tested to work on Windows 10 with Vulkan driver.
With the DirectX 12 driver, since SPIR-V to DXIL via godot-nir doesn't have Mesh Shaders capabilities (yet?), it remains untested.
Test project used available here

• Production edit: This closes Implement Vulkan Mesh Shading and Meshlets godot-proposals#6822.

[clayjohn]

This looks really good so far!

Just to properly set expectations, this is arriving a bit unexpected towards the end of our 4.3 feature merging window. It needs to be discussed with rendering contributors, so there is a good chance this will end up waiting for 4.3 to be released before it is merged.

On the other hand, there isn't too much code and it is fairly non-invasive. So there is little risk of breaking other things if we merge it.

[Saul2022]

Wow, that's so cool, though will there be a fallback n this or is it planned for follow up pr, because even nanite increased my fps, even on a igpu(considering it uses mesh shader in some sense for tiny triangles)

[DarioSamo]

I gave it a quick glance and it seems pretty well implemented to me. Nicely done.

Like Clay said this will probably need some discussion on the next meeting and potentially get delayed past the feature freeze for 4.3.

[rossbridger]

Any update on this?

[Calinou]

@thimenesup Could you look into rebasing this PR to fix merge conflicts?

[clayjohn]

@stuartcarnie Do you have a sense for how mesh shaders work on Metal (or if they are supported)? It would be nice for us to get this in for 4.4

[stuartcarnie]

@stuartcarnie Do you have a sense for how mesh shaders work on Metal (or if they are supported)? It would be nice for us to get this in for 4.4

Fortunately, the Apple M1 (Apple7 GPU) support mesh shaders, so I can take a look. Seems like the functionality is similar, so it will depend on whether SPIRV-Cross supports it.

[stuartcarnie]

Looks like recent (last few months) commits to SPIRV-Cross have added mesh shader support, so I will be able to try it.

[thimenesup]

Rebased with current master branch (and fixed the typos), let me know if anything else needs to be done.

[Ansraer]

As requested during the last rendering meeting I took the time to write an (hopefully) easy to understand proposal for this PR: godot-proposals/issues/11272

[thimenesup]

Noticed that the DirectX12 path needed some changes besides the rebasing, also took the chance to add documentation.

[Mickeon]

Docs are just fine as far as I can understand them to the best of my abilities.

[Ansraer]

@thimenesup Looks like some of the builds are currently failing because there are unimplemented functions on Apple devices.
Could you maybe take the time to add dummy functions, so this PR passes the tests?

The Mono build is failing due to GDExtension related issues, should also be a quick fix.

I plan to take some time this weekend to test and review this PR so we can hopefully get it merged before Christmas.

[thimenesup]

Made changes to preserve compatibility with GDExtension and also implemented the abstract methods for Metal Rendering Device as asked.

[Ansraer]

@thimenesup Uhm, did you test this PR after pushing the latest changes? While it looks like godot correctly identifies and compiles mesh shaders both my own Compositor Effects Project and the project linked in your PR just did nothing on my HW when I issued an actual draw command, not even report any kind of errors.

FYI I am on Windows with a Radeon RX 6900XT.

Here is a snippet of my demo project and the resulting Renderdoc capture:

Details

	# Actually use the mesh shader
	rd.draw_command_begin_label("MeshShaderStuff", Color.RED)
	
	var clear_colors = PackedColorArray([Color(0, 0, 0, 0), Color(0, 0, 0, 0), Color(0, 0, 0, 0)])
	var list := rd.draw_list_begin(framebuffer, RenderingDevice.INITIAL_ACTION_KEEP, clear_colors)
	rd.draw_list_bind_render_pipeline(list, pipeline) # pipeline is the mesh pipeline, which compiles fine
	rd.draw_list_bind_uniform_set(list, uniform_set, 0)
	rd.draw_list_set_push_constant(list, stream.data_array, stream.data_array.size())
	
	print(sphere_count)
	rd.draw_list_dispatch_mesh(list, 5, 1, 1)
	
	rd.draw_list_end()
	rd.draw_command_end_label()

[thimenesup]

@Ansraer I can confirm that it does work after the rebase, the confusion may come due to the fact that the test project is outdated, particularly the depth clear has changed in the master branch, making it not display anything (changing this does show the triangles), as for RenderDoc not showing the calls, it simply may not be behaving correctly (I did spot some unimplemented stuff about mesh shaders with it) but Radeon GPU profiler does indeed show the calls.

[stuartcarnie]

If / when this is merged, I can look at adding support for Metal. As I mentioned earlier, there has been work in SPIRV-Cross to add mesh and task shader support:

• MSL: added mesh-shader support KhronosGroup/SPIRV-Cross#2400
• MSL: added task shader support KhronosGroup/SPIRV-Cross#2402

MoltenVK currently doesn't have support, so the macOS build is failing due to undefined symbols¹:

Undefined symbols for architecture x86_64:
  "_vkCmdDrawMeshTasksEXT", referenced from:
      RenderingDeviceDriverVulkan::command_render_dispatch_mesh(RenderingDeviceDriver::CommandBufferID, unsigned int, unsigned int, unsigned int) in libdrivers.macos.template_release.x86_64.a[23](rendering_device_driver_vulkan.macos.template_release.x86_64.o)

Footnotes

1. Unclear why iOS is passing, as I would have expected the same issue. ↩

[ShalokShalom]

Can anybody explain us, how this will be enabled?

Do we compile two separate versions?
Or will the engine simply fallback automatically, when the GPU does not support the feature?

Some implementations seem to strictly require it, so will we have to decide this for the whole project as well?

[Calinou]

Can anybody explain us, how this will be enabled?

Do we compile two separate versions? Or will the engine simply fallback automatically, when the GPU does not support the feature?

Some implementations seem to strictly require it, so will we have to decide this for the whole project as well?

This is a feature you're meant to use in custom shaders, as the engine doesn't use mesh shaders out of the box with this PR. If you start to use it in custom shaders, you're expected to provide a fallback yourself (it's not possible to do so automatically). If you don't provide a fallback, the shader will fail compilation on unsupported GPUs, leading to broken visuals.

I don't know if this PR exposes a method to scripting to query whether mesh shaders are supported on the current GPU like #99119 does for traycing.