[rlsw] Segfaulting some example and resolution combos on RPI with PLATFORM_DRM and OPENGL_SOFTWARE

[NighthowlerStudios]

Issue description

This issue began investigation starting at here in the discord https://discord.com/channels/426912293134270465/426912293956222978/1501577144948490333

On all Raspberry Pi boards:

make PLATFORM=PLATFORM_DRM GRAPHICS=GRAPHICS_API_OPENGL_ES2

Works as expected. All examples build without errors and run cleanly.

However, between 32-bit and 64-bit platforms:

make PLATFORM=PLATFORM_DRM GRAPHICS=GRAPHICS_API_OPENGL_SOFTWARE

On the Raspberry Pi 2 (32-bit): Only some examples (textures_bunnymark) built and ran with this gets a Segmentation Fault when it tries to take control of the screen to render. There is no black flash to indicate it even got that far

On the Raspberry Pi 5 (64-bit): textures_bunnymark behaves as expected but only at 2560x1440.

Environment

Raspberry Pi 2 (32-bit) with Raspbian Trixie (Debian 13)

Boot into terminal mode with raspi-config > System > Boot then reboot

Code Example

First boot into terminal

then

sudo su
apt install libdrm-dev libegl1-mesa-dev libgles2-mesa-dev libgbm-dev
git clone https://github.com/raysan5/raylib
git checkout tags/6.0
cd raylib
cd src
make PLATFORM=PLATFORM_DRM GRAPHICS=GRAPHICS_API_OPENGL_SOFTWARE -j4
cd ..
cd examples
make PLATFORM=PLATFORM_DRM GRAPHICS=GRAPHICS_API_OPENGL_SOFTWARE -j4
cd textures
./textures_bunnymark

segfault occurs after a short time after bunnymark is executed, it never takes control of rendering.

If you change SOFTWARE with ES2, it works.

[NighthowlerStudios]

trace of the SEGFAULT

Program received signal SIGSEGV, Segmentation fault.
0x00013e48 in sw_color_to_color8 (dst=0x76d40000 "\177ELF\001\001\001",
    src=0x7effefec) at external/rlsw.h:1500
1500        dst[0] = (uint8_t)(src[0]*255.0f);
(gdb) bt
#0  0x00013e48 in sw_color_to_color8 (dst=0x76d40000 "\177ELF\001\001\001",
    src=0x7effefec) at external/rlsw.h:1500
#1  0x0001699c in sw_pixel_write_color_R8G8B8A8 (pixels=0x76d40000,
    color=0x7effefec, offset=0) at external/rlsw.h:2231
#2  0x0002bb30 in sw_raster_quad_TEX_BLEND (a=0x1029fc <RLSW+204>,
    b=0x102a24 <RLSW+244>, c=0x102a4c <RLSW+284>, d=0x102a74 <RLSW+324>)
    at ./external/rlsw.h:5654
#3  0x0003034c in sw_quad_render (state=18) at external/rlsw.h:3558
#4  0x00031638 in sw_poly_fill_render (state=26) at external/rlsw.h:3774
#5  0x00031ec0 in sw_immediate_push_vertex (position=0x7efff150)
    at external/rlsw.h:3872
#6  0x00034dd4 in swVertex2f (x=800, y=0) at external/rlsw.h:4605
#7  0x00037588 in rlVertex2f (x=800, y=0) at rlgl.h:1469
#8  0x00064c48 in DrawRectanglePro (rec=..., origin=..., rotation=0, color=...)
    at rshapes.c:796
#9  0x000646d4 in DrawRectangleV (position=..., size=..., color=...)
    at rshapes.c:728
#10 0x0006462c in DrawRectangle (posX=0, posY=0, width=800, height=40,
    color=...) at rshapes.c:721
#11 0x000121d8 in main () at textures/textures_bunnymark.c:117

[NighthowlerStudios]

was looking at the exact point of segfault by spamming continues in gdb and noticed something.

the bunnymark demo is segfaulting applying textures, specifically at that point.

so i went ahead and compiled other demos that use plain shapes and never look at a texture

such as shapes_basic_shapes.

those work fine.

but models_waving_cubes?

segfault. now that's confusing. it doesn't use textures.

so now we know that some demos do work in software. others don't.

[NighthowlerStudios]

Traced via Grok Code and Copilot, it eventually settled on bounds checks in sw_triangle_clip_and_project. And it found another one with sw_texture_sample_nearest.

I tested some patches on this problem, and thankfully, the segfault did stop occurring and those demos started working.

It is very very likely that Pi wasn't going to be the only platform affected by these out of bounds checks

These seem to simply be checks forgotten about during implementation.

Creating a PR on this now and I'll tag this issue when I have it up and ready.

[ZX41R]

The gdb trace is a good clue that this is earlier than sw_color_to_color8() itself. sw_pixel_write_color_R8G8B8A8() is called with offset=0, so the bad address has already become the framebuffer pointer passed down from the raster path.

In the quad path, SW_RASTER_QUAD derives xMin/yMin/xMax/yMax directly from the projected corners and then computes:

baseOffset = y * stride + xMin;
cPtr = cPixels + baseOffset * SW_FRAMEBUFFER_COLOR_SIZE;

without the clamp that the clear paths use. A negative xMin/yMin, or a projected quad extending outside the target, can therefore move cPtr before the color buffer before the inner loop even starts. On 32-bit ARM that underflow can plausibly land on a mapped ELF page, which matches dst=... "\177ELF" in the trace.

When you open the PR, I would make sure the fix clips the scan rectangle and adjusts the starting interpolation values (uRow/vRow, color, depth) by the clipped delta. A late if (x < 0 || x >= width) guard will stop the write but can still leave texture/depth interpolation inconsistent at the clipped edge.

[NighthowlerStudios]

When you open the PR, I would make sure the fix clips the scan rectangle and adjusts the starting interpolation values (uRow/vRow, color, depth) by the clipped delta. A late if (x < 0 || x >= width) guard will stop the write but can still leave texture/depth interpolation inconsistent at the clipped edge.

@ZX41R and this is your recommendations in response to #5843 correct? (A PR is already open on this)

So my code is incorrect then and should be different? It's fine if it is, like I said, I am inexperienced in graphics. Are you describing what I thought may happen where textures may warp at the buffer edge doing this check too late?

[ZX41R]

Yes, that is the point I was getting at.

Your patch is not “bad” in the sense that it explains why the crash stopped: it prevents the rasterizer from producing framebuffer pointers outside the color buffer. The part I would be careful with is where the clamp happens.

Clamping the projected vertex position changes the primitive before rasterization. For a triangle/quad that extends past the viewport, the new clipped vertex keeps the old color/UV/depth values, so the rasterizer is no longer interpolating the same edge. That is where texture stretching or edge warping can come from. It is especially visible on textured quads because the geometry got shortened but the endpoint attributes did not get recomputed for the shortened edge.

The safer shape of the fix is:

1. Keep the projected vertices and their attributes unchanged.
2. Intersect the scan rectangle/span with the framebuffer bounds.
3. When xMin or yMin is advanced by clipping, also advance the starting interpolants by the same delta: color, depth, u/v, etc.

So for the quad path, instead of clamping v->position[] in sw_quad_clip_and_project(), I would clip around the raster loops: clamp the y loop range, clamp the per-row x range, and apply dy/dx corrections to cRow, zRow, uRow, vRow, and per-pixel starting color/depth/UV. Same idea for the triangle span macro.

The sw_texture_sample_nearest() clamp is different and looks reasonable: u == 1.0 can become x == width, so clamping the final texel index to wMinus1/hMinus1 is a normal defensive fix. I just would not rely on texture-coordinate clamping to fix framebuffer writes; those are separate bounds.

[NighthowlerStudios]

I'm aware of the texture clamp being a separate issue yes. I should have opened a different PR for that one but when I was looking over esp32's optimizations I noticed that could be an issue ^^'

regardless this pr will shift focus and focus on fixing the rastering loop. give me a few minutes, i'll put some changes on the PR and reopen it when I'm ready

[NighthowlerStudios]

I need to be honest as a noob and admit I'm still stuck on what you mean by this

So for the quad path, instead of clamping v->position[] in sw_quad_clip_and_project(), I would clip around the raster loops: clamp
the y loop range, clamp the per-row x range, and apply dy/dx corrections to cRow, zRow, uRow, vRow, and per-pixel starting
color/depth/UV. Same idea for the triangle span macro.

Could you point me to which line you're saying to start this? I'm having trouble figuring out exactly how the dx (I assume you mean the change between before clamp and after clamp) corrections to cRow etc. Because I'm having trouble understanding exactly what cRow and them are doing. especially with how they don't exist in TRIANGLE_SPAN

EDIT: nvm, I missed the comment where Rect is axis aligned. That makes this much simpler to dissect