• Any program you run can use the pipe or the socket, so the remote control is off by default; other machines cannot, nor, on Linux, other users. Turn it on for single-player games only: anti-cheat software watches injected DLLs.
• While you type to the assistant, the game does not have the focus: the keys then reach it as window messages, or X11 events on Linux, which games that read the keyboard state directly (DirectInput, raw input) do not see. Click the game window after sending your request, or let the assistant play on a second screen.
• While the overlay menu is open, the game sees no keys: the assistant can close it with insert.
• The assistant plays turn by turn, a screenshot and a few keys at a time: fine for menus, puzzles and slow games, not for reflexes.
• On Linux, keys go through the XTEST extension of the X server; without it, they are sent as events to the window. Native Wayland games get no keys.
• One game per pipe or socket name: a second game started with the same name logs that the remote control is unavailable.
• unloadKey stops the remote control with the overlay.
• Screenshots of HDR back buffers, with more than 10 bits per channel, fail.

remote/uih_mcp.py speaks MCP with the assistant on its standard input and output, and forwards each tool call as one line to the library, which serves the pipe or the socket on a thread of its own (src/remote.cpp; the transport and the keys in src/windows/remote.cpp and src/linux/remote.cpp):

Failures answer error <reason>. The screenshots come from uih::CaptureFrame(); the keys go through SendInput on Windows and XTEST on Linux when the game has the focus.

Start() launches a worker thread. On Windows it hooks every graphics and input DLL already loaded, then the others as they load (LdrRegisterDllNotification). The hooks are placed with MinHook on functions found through dummy objects: a hidden window, a D3D device, a Vulkan device per GPU. On Linux, the library exports the hooked functions under their own names and overrides dlsym(), so the game calls them whether it links them or looks them up like SDL and GLFW do.

uih::Frame() registers new textures, polls the hotkeys, applies the DPI scale and runs render; the backend then draws the ImGui data on the back buffer, uploading textures as needed, and the game's present goes on. Vertex colors are converted to linear for back buffers that store linear values; OpenGL and D3D9 turn sRGB writes off instead. Vulkan submits its own command buffer, which waits on the game's semaphores and signals the one the present waits on.

Resolution changes are followed on every API: D3D9 Reset/ResetEx, DXGI ResizeBuffers, a new Vulkan swap chain, or a new size of the OpenGL window; the overlay takes the new size on the next frame.

Windows. The game window is subclassed to feed ImGui, and its input messages are swallowed while the menu is open, except releases, keyboard and mouse raw input (filtered below) and Alt+F4. GetRawInputData, GetRawInputBuffer, GetAsyncKeyState, GetKeyState, GetKeyboardState, DirectInput and XInput then report no input to the game, and GetCursorPos a frozen position. The cursor is freed from the game's ClipCursor rectangle, which is applied again when the menu closes. Releases always go through, so no key stays stuck; calls made by the hook itself see the real state.

Linux. The pointer and the keyboard are polled from the X server on a separate XCB connection.

Stop() asks the rendering thread to release the backend at its next present, or does it itself after one second, then removes the hooks once none of them is running. When the library cannot leave the process (Linux, dinput8.dll proxy, no module), the unload key runs Stop() on a thread that then polls the menu key and buttons, and starts the hook again with the menu open.

The overlay is drawn on one swap chain, its target, by one thread at a time; the others present untouched. The target changes when it stops presenting for half a second, when the window of another swap chain has the focus and the target's does not, or when the game replaces its swap chain on the same window. A Vulkan surface created before the hook is matched to the window of the swap chain's size. EGL does not tell the window of a surface: there the overlay only moves when its target stops presenting.

On Windows, what the overlay runs in the game's threads (frames, teardown, window messages) is protected by structured exception handling. An exception there, such as an access violation in the render callback, removes the overlay from another thread, as the unload key does without a module: the game keeps running, and the menu key or buttons bring the overlay back. The frame runs before the backends take a reference to the back buffer, so a fault leaks none.

When CaptureFrame() waits, the backend copies its back buffer at the next present, before it draws the overlay: GetRenderTargetData on D3D9, a staging texture on D3D10 and 11, a readback buffer on D3D12 and Vulkan, glReadPixels on OpenGL. Multisampled back buffers are resolved first. To allow the copy, Vulkan swap chains are created with TRANSFER_SRC usage, and again without it if the surface refuses.

With remoteControl set, a thread of the library serves the named pipe with overlapped I/O on Windows, or the Unix socket with poll() and an eventfd on Linux, so that Stop() ends a connection or a command at once. Messages are drawn after the render callback, in the frame of the overlay; keys go through SendInput (Windows) or XTEST on a connection of its own to the X server (Linux) when the game has the focus, and as window messages or X11 events otherwise.

On Linux, every process the game starts preloads the library, so the socket is opened at the first frame of the overlay by the process that draws it. The socket file is only open to its owner (0600), and the server also checks the user of each client (SO_PEERCRED), for the /tmp fallback. A file left by a game that crashed is replaced when nobody answers on it.

From an MSYS2 UCRT64 shell or with a winlibs toolchain in PATH:

pacman -S mingw-w64-ucrt-x86_64-toolchain mingw-w64-ucrt-x86_64-cmake mingw-w64-ucrt-x86_64-ninja
cmake --preset mingw-x64 && cmake --build --preset mingw-x64

With the mingw-x64 and mingw-x86 presets and the toolchain files in cmake/:

With the linux preset:

It only needs libxcb and glibc at run time, but a glibc at least as recent as the build machine's: to share it or to use it in the Steam Linux Runtime, build on an old distribution or take the prebuilt one. Set CC=clang CXX=clang++ before configuring to use Clang.

Options are passed at configure time, such as -DUIH_BUILD_TESTS=ON:

Minimum versions: CMake 3.21, Visual Studio 2022, mingw-w64 11 with GCC 13, GCC 13 or Clang 18 on Linux; Windows 10 at run time. The mingw-w64 headers of Ubuntu 22.04 are too old for D3D12.

Without a GPU, as on the CI runners, Direct3D uses WARP, and Mesa's software drivers from mesa-dist-win cover the rest: copy opengl32.dll, libgallium_wgl.dll and dxil.dll next to uih_host.exe with GALLIUM_DRIVER=llvmpipe (or d3d12 for the D3D12-based driver of Microsoft's OpenGL compatibility pack, which needs dxil.dll to sign its shaders), and register lvp_icd.*.json under HKLM\SOFTWARE\Khronos\Vulkan\Drivers with a vulkan-1.dll from the Vulkan runtime.

UIH_TEST_VULKAN_VALIDATION=1 runs the Vulkan host under VK_LAYER_KHRONOS_validation, from the Vulkan SDK on Windows or vulkan-validationlayers on Linux, and UIH_TEST_D3D_DEBUG=1 creates the Direct3D 10, 11 and 12 devices with the debug layer, from the Graphics Tools optional feature of Windows. Any error they report, from the host or from the hook, fails the test.

uih_game --dll <path> --backend <name> -- game.exe [arguments] runs a game and injects the DLL once its window shows, as an injector does: the backend must become active, INSERT must show the menu on screen and F11 remove it while the game keeps running. It reads the log of the DLL as DebugView does, so DebugView must not run at the same time, and saves its captures as game-before.bmp, game-menu.bmp and game-after.bmp (--captures <prefix>).

tests/game.sh runs the same scenario on a Linux game, such as DDNet or a Godot project.

tests/game_mcp.py libuih_remote_preload.so <backend> mcp.png game [arguments] plays a Linux game through remote/uih_mcp.py, as an assistant does, with the remote control on: game description, screenshot, message, INSERT to open the menu and F11 to remove the overlay, after which the socket must be gone and the game still running.

On Linux, uih_host does not link the hook: each test preloads the .so, opens the menu, checks the pixels and removes the overlay. They cover:

• GLX, EGL, dlsym() lookups and Vulkan;
• --resize, --reload, --second-window, --recreate-context and --bound-framebuffer variants;
• the remote control (--remote), with tests/remote_preload.cpp, the hook built with it on: game description, screenshot without the message of the assistant, message on screen, keys through XTEST with the focus and as X11 events without it;
• presents through eglSwapBuffersWithDamage that redraw and post a single pixel once EGL_EXT_buffer_age allows it (--swap-with-damage);
• the Vulkan loader opened as libvulkan.so (--vulkan-library).

They need libx11-dev libxtst-dev libgl-dev libegl-dev; without a display, run xvfb-run -a ctest --test-dir build/linux. With -DUIH_SANITIZE=ON, the preload tests load the sanitizer runtime before the library, and any memory error or undefined behavior in the hook fails them.

The CI runs on every push and pull request:

Pushing a v* tag publishes the MSVC and Linux portable libraries as a release, once every job above has passed.