bevy_ui backend does not work with TargetCamera

[musjj]

Here's a minimal reproduction code. It's a combination of 2d/pixel_grid_snap.rs and ui/button.rs from the official examples:

Expand

use bevy::{
    prelude::*,
    render::{
        camera::RenderTarget,
        render_resource::{
            Extent3d, TextureDescriptor, TextureDimension, TextureFormat, TextureUsages,
        },
        view::RenderLayers,
    },
    window::WindowResized,
};
use bevy_mod_picking::prelude::*;

/// In-game resolution width.
const RES_WIDTH: u32 = 160;

/// In-game resolution height.
const RES_HEIGHT: u32 = 90;

/// Default render layers for pixel-perfect rendering.
/// You can skip adding this component, as this is the default.
const PIXEL_PERFECT_LAYERS: RenderLayers = RenderLayers::layer(0);

/// Render layers for high-resolution rendering.
const HIGH_RES_LAYERS: RenderLayers = RenderLayers::layer(1);

fn main() {
    App::new()
        .add_plugins(DefaultPlugins.set(ImagePlugin::default_nearest()))
        .add_plugins(DefaultPickingPlugins)
        .insert_resource(Msaa::Off)
        .insert_resource(DebugPickingMode::Normal)
        .add_systems(Startup, (setup_camera, setup_ui).chain())
        .add_systems(Update, fit_canvas)
        .run();
}

/// Low-resolution texture that contains the pixel-perfect world.
/// Canvas itself is rendered to the high-resolution world.
#[derive(Component)]
struct Canvas;

/// Camera that renders the pixel-perfect world to the [`Canvas`].
#[derive(Component)]
struct InGameCamera;

/// Camera that renders the [`Canvas`] (and other graphics on [`HIGH_RES_LAYERS`]) to the screen.
#[derive(Component)]
struct OuterCamera;

/// Our button
#[derive(Component)]
struct MyButton;

fn setup_ui(mut commands: Commands, query: Query<Entity, With<InGameCamera>>) {
    commands
        .spawn((
            NodeBundle {
                style: Style {
                    width: Val::Percent(100.0),
                    height: Val::Percent(100.0),
                    align_items: AlignItems::Center,
                    justify_content: JustifyContent::Center,
                    ..default()
                },
                ..default()
            },
            Pickable::IGNORE,
            TargetCamera(query.single()),
        ))
        .with_children(|parent| {
            parent
                .spawn((
                    MyButton,
                    ButtonBundle {
                        style: Style {
                            width: Val::Px(50.0),
                            height: Val::Px(20.0),
                            border: UiRect::all(Val::Px(2.0)),
                            // horizontally center child text
                            justify_content: JustifyContent::Center,
                            // vertically center child text
                            align_items: AlignItems::Center,
                            ..default()
                        },
                        border_color: BorderColor(Color::BLACK),
                        background_color: BackgroundColor(Color::rgb(0.15, 0.15, 0.15)),
                        ..default()
                    },
                    On::<Pointer<Click>>::run(|| info!("pressed!")),
                    On::<Pointer<Over>>::run(|| info!("hovered!")),
                ))
                .with_children(|parent| {
                    parent.spawn((
                        TextBundle::from_section(
                            "Button",
                            TextStyle {
                                font_size: 10.0,
                                color: Color::rgb(0.9, 0.9, 0.9),
                                ..default()
                            },
                        ),
                        Pickable::IGNORE,
                    ));
                });
        });
}

fn setup_camera(mut commands: Commands, mut images: ResMut<Assets<Image>>) {
    let canvas_size = Extent3d {
        width: RES_WIDTH,
        height: RES_HEIGHT,
        ..default()
    };

    // this Image serves as a canvas representing the low-resolution game screen
    let mut canvas = Image {
        texture_descriptor: TextureDescriptor {
            label: None,
            size: canvas_size,
            dimension: TextureDimension::D2,
            format: TextureFormat::Bgra8UnormSrgb,
            mip_level_count: 1,
            sample_count: 1,
            usage: TextureUsages::TEXTURE_BINDING
                | TextureUsages::COPY_DST
                | TextureUsages::RENDER_ATTACHMENT,
            view_formats: &[],
        },
        ..default()
    };

    // fill image.data with zeroes
    canvas.resize(canvas_size);

    let image_handle = images.add(canvas);

    // this camera renders whatever is on `PIXEL_PERFECT_LAYERS` to the canvas
    commands.spawn((
        Camera2dBundle {
            camera: Camera {
                // render before the "main pass" camera
                order: -1,
                target: RenderTarget::Image(image_handle.clone()),
                ..default()
            },
            ..default()
        },
        InGameCamera,
        PIXEL_PERFECT_LAYERS,
    ));

    // spawn the canvas
    commands.spawn((
        SpriteBundle {
            texture: image_handle,
            ..default()
        },
        Canvas,
        Pickable::IGNORE,
        HIGH_RES_LAYERS,
    ));

    // the "outer" camera renders whatever is on `HIGH_RES_LAYERS` to the screen.
    // here, the canvas and one of the sample sprites will be rendered by this camera
    commands.spawn((Camera2dBundle::default(), OuterCamera, HIGH_RES_LAYERS));
}

fn fit_canvas(
    mut resize_events: EventReader<WindowResized>,
    mut projections: Query<&mut OrthographicProjection, With<OuterCamera>>,
) {
    for event in resize_events.read() {
        let h_scale = event.width / RES_WIDTH as f32;
        let v_scale = event.height / RES_HEIGHT as f32;
        let mut projection = projections.single_mut();
        projection.scale = 1. / h_scale.min(v_scale).round();
    }
}

I tried to read through the UI backend and I think this is the problem. We have two cameras:

commands.spawn((
    Camera2dBundle {
        camera: Camera {
            order: -1,
            target: RenderTarget::Image(image_handle.clone()),
            ..default()
        },
        ..default()
    },
    InGameCamera,
    PIXEL_PERFECT_LAYERS,
));

commands.spawn((Camera2dBundle::default(), OuterCamera, HIGH_RES_LAYERS));

The first one targets an image handle, while the second one targets the main window. The UI node is associated with the first camera, while the main pointer is associated with the second camera.

The bevy_ui backend looks for pointers associated with the camera that the UI node is rendering to, before testing if those pointers intersects with the UI node. So our UI here is always skipped because our UI camera has no associated pointers.

It seems that the sprite backend does not have this problem. I think it's because the sprite backend does not care to which camera the sprite is getting rendered to.

Do you think that there is a general way to solve this problem upstream? Some pointers would be appreciated.

[wandbrandon]

I am also having the same issue. Using the same exact code I cannot get any pointers to work. When using the noisy debugger, It's obvious that It sees the Pixel Perfect camera as a large texture and not an actual ui element. So the target Camera is the outer camera. Not sure how to fix.

[aevyrie]

If you are trying to pick into a texture, then you need to add a pointer to that render target. #327

This all comes back to how the picking pipeline works: https://docs.rs/bevy_mod_picking/latest/bevy_mod_picking/#the-picking-pipeline

All you need to do is inform the backends that there is a pointer located at some x/y position on the target, and they can handle everything else automatically. Doing this correctly will probably require raycasting: raycast onto the render-to-texture quad, find the uv coordinate that the pointer is over, and send a PointerMove of that pointer over that render target.

[wandbrandon]

So in that example they aren't using raycasting. Is there not a way to simply attach the current pointer to the original in game camera? As opposed to translating the positions of the mouse to the render texture? Apologies as I'm a new to bevy and trying to learn a bit more about how it works.

[aevyrie]

That won't work. The pointer's position on the window is not the same as the pointer's position on the lower resolution render target. For example, if the window is 1920x1080, and you put the pointer in the bottom right, the pointer's position on the low resolution target is not (1920,1080), it depends on the size of that render target and its position on screen. If it is half resolution, the position 1920, 1080 on the window would map to 960, 540 on the half resolution render target. However, this ignores that the main camera could move around such that the render target does not fill the window, in which case you also need to compute this offset. You could hardcode this and assume the render target always fills the screen and has some fixed resolution ratio, but it might be brittle.

[wandbrandon]

Thank you for the write up, that makes sense. Perhaps rendering to a texture isn’t the greatest method for achieving this? Maybe there are better methods for achieving pixel perfect rendering?

[aevyrie]

I don't think there is anything wrong with this approach, it's just not immediately obvious how to handle this in a robust way. Considering this is 2D-only, it shouldn't be too hard though. All you need is the position of the pointer inside the texture. If it is in bevy_ui, I think it already computes that for you. If you are doing this manually with two 2d cameras, that also shouldn't be too hard. The only steps you need to follow are:

1. Compute the position of the pointer in the texture quad
   • This is the only tricky part. In 2d, this is just comparing the location of the two camera's viewports, and mapping the pointer from the outer camera's viewport to the inner camera's viewport. If you are showing this texture in bevy_ui, you should be able to just use https://docs.rs/bevy/latest/bevy/ui/struct.RelativeCursorPosition.html
   • If your render-to-texture is in 3d space, you would need to do a raycast.
2. Send an InputMove event for that pointer, with the quad's RenderTarget and the position computed in step 1

[wandbrandon]

This is very helpful, I appreciate you taking the time to do this! Hopefully this will help others as well.