Completed

Author: mattkae

doc/sphinx/how-to/how-to-enable-graphics-core22-on-a-device.md

@@ -7,9 +7,9 @@ to run graphical snaps like `ubuntu-frame` or any other *Mir*-based
 compositor in a snapped environment.
 
 To this end, we will accomplish the following in order:
-- Assemble a `graphics-core22` snap
-- Test the assembled snap
-- Confirm that `ubuntu-frame` runs in your environment
+- How to assemble a `graphics-core22` provider snap
+- How to test a `graphics-core22` provider snap
+- How to test the confinement of your *Mir*-based snap that relies on the provider snap
 
 ## Prerequisites
 Before we get started, let's make sure that we have everything that we require.
@@ -96,7 +96,7 @@ While *Mir* can be enabled on platforms that do not support `GBM/KMS`, this
 requires a paltform implementation to be written in *Mir*. This work is out
 of the scope of this guide.
 
-## Assemble a `graphics-core22` snap
+## How to assemble a `graphics-core22` provider snap
 With the prerequisites out of the way, it is time to assemble a
 `graphics-core22` snap.
 
@@ -190,7 +190,9 @@ parts:
     - graphics/lib/*
 ```
 
-### Possible complications
+When you're ready, build your snap using `snapcraft`.
+
+### Troubleshooting
 There are a number of ways the vendor drivers can differ from this simplest case, requiring different solutions.
 Let's explore a few of them.
 
@@ -258,4 +260,196 @@ The [mesa-core22](https://github.com/canonical/mesa-core22/blob/main/snap/snapcr
 directory is shipped in a top-level `glvnd` path, under the path exported by the
 `graphics-core22` slot.
 
-.
+## How to test a `graphics-core22` provider snap
+Now we have our `graphics-core22` provider snap built. However, we don't yet know if it works
+properly. Testing this snap is our next task.
+
+For starters, install the [graphics-test-tools](https://snapcraft.io/graphics-test-tools) snap:
+
+```shell
+sudo snap install graphics-test-tools --channel 22/stable
+```
+
+Next, connect the snap that you've built to `graphics-test-tools`:
+
+```shell
+sudo snap connect graphics-test-tools:graphics-core22 <your-snap>:graphics-core22
+```
+
+With that installed, we can begin testing.
+
+### Use `eglinfo` to test
+This is the most basic test that can be performed, and is a good indication of baseline GPU
+driver setup. This should list _at least_ a GBM platform with the expected vendor and 
+`OpenGL_ES` client API. An abbreviated example (on a `mesa-core22` system):
+
+```
+$ graphics-test-tools.eglinfo
+EGL client extensions string:
+	EGL_EXT_device_base EGL_EXT_device_enumeration EGL_EXT_device_query
+	EGL_EXT_platform_base EGL_KHR_client_get_all_proc_addresses
+	EGL_EXT_client_extensions EGL_KHR_debug EGL_EXT_platform_device
+	EGL_EXT_platform_wayland EGL_KHR_platform_wayland
+	EGL_EXT_platform_x11 EGL_KHR_platform_x11 EGL_EXT_platform_xcb
+	EGL_MESA_platform_gbm EGL_KHR_platform_gbm
+	EGL_MESA_platform_surfaceless
+
+
+GBM platform:
+EGL API version: 1.5
+EGL vendor string: Mesa Project
+EGL version string: 1.5
+EGL client APIs: OpenGL OpenGL_ES
+EGL driver name: iris
+EGL extensions string:
+	EGL_ANDROID_blob_cache EGL_ANDROID_native_fence_sync
+	EGL_EXT_buffer_age EGL_EXT_create_context_robustness
+	EGL_EXT_image_dma_buf_import EGL_EXT_image_dma_buf_import_modifiers
+	EGL_IMG_context_priority EGL_KHR_cl_event2 EGL_KHR_config_attribs
+	EGL_KHR_context_flush_control EGL_KHR_create_context
+	EGL_KHR_create_context_no_error EGL_KHR_fence_sync
+	EGL_KHR_get_all_proc_addresses EGL_KHR_gl_colorspace
+	EGL_KHR_gl_renderbuffer_image EGL_KHR_gl_texture_2D_image
+	EGL_KHR_gl_texture_3D_image EGL_KHR_gl_texture_cubemap_image
+	EGL_KHR_image EGL_KHR_image_base EGL_KHR_image_pixmap
+	EGL_KHR_no_config_context EGL_KHR_reusable_sync
+	EGL_KHR_surfaceless_context EGL_EXT_pixel_format_float
+	EGL_KHR_wait_sync EGL_MESA_configless_context EGL_MESA_drm_image
+	EGL_MESA_image_dma_buf_export EGL_MESA_query_driver
+	EGL_WL_bind_wayland_display
+Configurations:
+ 	bf lv colorbuffer dp st  ms	vis   cav bi  renderable  supported
+  id sz  l  r  g  b  a th cl ns b	id   eat nd gl es es2 vg surfaces
+---------------------------------------------------------------------
+0x01 32  0 10 10 10  2  0  0  0 0 0x30335241--     	y  y  y 	win
+0x02 32  0 10 10 10  2 16  0  0 0 0x30335241--     	y  y  y 	win
+0x03 32  0 10 10 10  2 24  0  0 0 0x30335241--     	y  y  y 	win
+0x04 32  0 10 10 10  2 24  8  0 0 0x30335241--     	y  y  y 	win
+0x05 32  0 10 10 10  2  0  0  2 1 0x30335241--     	y  y  y 	win
+… lots more configurations, then other platform details …
+```
+
+If `eglinfo` does **NOT** list a GBM platform, or generates errors then you want to look
+at  [possible complications](#possible-complications). If `eglinfo` does list a GBM platform
+then we can proceed to testing `ubuntu-frame`.
+
+### Test if `ubuntu-frame` works
+First, install `ubuntu-frame`:
+```shell
+sudo snap install ubuntu-frame --devmode
+```
+
+Next, disconnect it from the default graphics interface:
+```shell
+sudo snap disconnect ubuntu-frame:graphics-core22
+```
+
+Then, connect it to your new provider snap:
+```shell
+sudo snap connect ubuntu-frame:graphics-core22 <your-snap>:graphics-core22
+```
+
+Finally, run `ubuntu-frame`:
+```shell
+ubuntu-frame
+```
+
+If everything previously has gone correctly this should result in `ubuntu-frame` coming
+up on the connected outputs. If not, the log messages will hopefully help identify issues.
+
+## How to test the confinement of your *Mir*-based snap
+Once you have your `graphics-core22` provider snap setup, and you are confident that it works
+well, you'll want to make sure that you can run the snap that *depends on* your provider snap
+in a confined mode. Since we already have `ubuntu-frame` setup, we will use it as an example.
+However, you should be able to run these tests on any *Mir*-based snap.
+
+First, let's uninstall `ubuntu-frame` if it is installed:
+
+```shell
+sudo snap remove ubuntu-frame
+```
+
+Next, we can install `ubuntu-frame` without the `--devmode` flag:
+```shell
+sudo snap install ubuntu-frame
+sudo snap disconnect ubuntu-frame:graphics-core22
+```
+
+Finally, we will run `ubuntu-frame` like before:
+```shell
+ubuntu-frame
+```
+
+If everything works, then `ubuntu-frame` is ready. If not, we'll have to troubleshoot.
+
+### Troubleshooting
+When bringing up a confined ubuntu-frame snap on a new board with new drivers there are two
+separate access control mechanisms:
+- AppArmor
+- The devices cgroup
+
+#### AppArmor
+The first is relatively easy to debug. When AppArmor would deny access to a resource, it outputs a nice
+message in `dmesg`:
+
+```
+apparmor="DENIED" operation="open" profile="snap.ubuntu-frame.ubuntu-frame" name="/run/udev/data/c189:1" <other stuff>
+```
+
+You can find the AppArmor profile at `/var/lib/snapd/apparmor/profiles/snap.ubuntu-frame.ubuntu-frame`.
+Make modifications to the profile using your favorite text editor. Finally, apply the change to replace
+the default confinement:
+
+```shell
+sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.ubuntu-frame.ubuntu-frame
+```
+
+#### The devices cgroup
+The cgroup confinement is less easy to debug. The kernel emits no logs when the devices cgroup
+denies access to a device node. The device node on the filesystem will appear to have the correct
+permissions (and, for example, you will be able to touch it), but calls to `open()` will fail
+with `EPERM`. If everything seems to be set up correctly, and there are no errors in `dmesg`,
+but `Mir` is failing it’s likely that the devices cgroup confinement is to blame.
+
+Additionally, it is more difficult to test as the cgroup is only set up during snap run startup,
+and so the knobs we need to twiddle only exist while the snap is running.
+
+We can get around this by running:
+```shell
+snap run --shell ubuntu-frame
+```
+This command will get snapd to do all the initialisation and drop us into a shell.
+Once the shell exists, there will be two(?!) cgroup folders found under `/sys/fs/cgroup/devices`:
+1. /sys/fs/cgroup/devices/system.slice/snap.ubuntu-frame.ubuntu-frame.${UUID}.scope, and
+2. /sys/fs/cgroup/devices/snap.ubuntu-frame.ubuntu-frame
+
+It is (2) that we’re after. You can check that you’ve got the right directory, because the
+`devices.list` file will contain a bunch of lines like:
+
+```
+c 5:1 rwm
+c 5:2 rwm
+c 136:* rwm
+c 137:* rwm
+c 138:* rwm
+```
+
+You now need to give the cgroup permission to access the necessary devices, for which you need 
+the `major:minor` of the device nodes. You can find that with ls:
+```
+$ ls -la /sys/fs/cgroup/devices/snap.ubuntu-frame.ubuntu-frame
+crw-rw-rw- 1 root root 10, 60 Jan 10 04:56 /dev/mali0
+```
+
+Here we see that the `/dev/mali0` device node has `major:minor` equal to `10:60`.
+
+Now we can enable access to the relevant device node, via:
+```shell
+echo "c 10:60 rw" | sudo tee /sys/fs/cgroup/devices/snap.ubuntu-frame.ubuntu-frame/devices.allow
+```
+
+And now we can go back to the shell, and try running `ubuntu-frame`:
+
+```shell
+$SNAP/usr/local/bin/frame
+```