Navmesh 1.0 - NavMesh Integration (Recast & Detour)

https://www.youtube.com/watch?v=oiziTNmDPVI

One of the biggest AI movement milestones I tackled in my engine was integrating Recast & Detour and building a full workflow around offline-baked navigation meshes, debug visualization, path queries, and NavMesh-constrained movement.

At this stage, the goal was not crowd simulation, dynamic obstacles, or fancy steering. The point was to get the engine to the stage where it could:

• build a NavMesh from scene geometry,

• save it as an engine resource,

• load it per scene,

• query paths,

• and constrain movement to walkable surfaces.

It sounds straightforward written like that, but in practice it required touching the engine across multiple layers: third-party integration, scene data extraction, offline baking, runtime nav queries, editor UI tooling, and debug rendering.

What I wanted to achieve

The task was focused on establishing a strong navigation foundation:

• integrate Recast/Detour into the engine build,

• extract scene geometry as a triangle soup for baking,

• bake a NavMesh offline and store it as a resource,

• load the NavMesh when the scene is loaded,

• add debug visualization to validate the NavMesh in-editor,

• implement a click-based tool to test pathfinding,

• and constrain a moving GameObject to the NavMesh.

This setup is the base layer that everything else builds on: AI agents, steering, avoidance, and proper gameplay movement rules.

Integrating Recast & Detour into the engine

The first step was simply getting the RecastNavigation codebase compiled as part of the solution and linked into the engine.

That included:

• hooking up include directories and library outputs correctly,

• making sure build configs matched (x64, Debug/Release),

• and keeping paths relative to the solution so the project is portable.

Once that worked, the engine could include Detour headers and link to the functions needed for navmesh creation and runtime querying.

NavMesh as an engine resource (one per scene)

I wanted the NavMesh to behave like any other asset in the engine: something that is generated once, saved, and then loaded quickly at runtime.

So I introduced a simple binary navmesh resource format, stored per scene:

Assets/NavMeshes/<SceneName>.navmesh

The file stores:

• a header with magic/version and dtNavMeshParams

• a list of tiles (tileRef, data size)

• and the raw Detour tile data blob

This meant the runtime side is lightweight: it just reads the file, reconstructs the dtNavMesh, builds a dtNavMeshQuery, and the navigation system is ready.

Extracting walkable geometry from the scene (triangle soup)

The baking pipeline needs input geometry in the simplest possible form:

• verts: [x, y, z, x, y, z…]

• tris: [i0, i1, i2, i0, i1, i2…]

So I built an extraction pass that:

• traverses the scene hierarchy,

• collects meshes from ModelComponent,

• transforms vertex positions into world space using the GameObject transform,

• and appends indices with correct offsets into a global buffer.

To avoid baking everything, I introduced a clean rule: Only GameObjects on Layer::NAVMESH contribute geometry to the NavMesh bake.

That gave me full control in the editor: environment goes in, props and dynamic objects stay out.

Offline baking: Recast “Solo Mesh” pipeline

With triangle soup available, the next step was implementing the Recast build pipeline, modeled after the classic “solo mesh” approach:

• heightfield creation and rasterization

• walkable filtering (slope/climb)

• compact heightfield generation

• region building

• contour extraction

• polygon mesh + detailed mesh

• conversion into Detour navmesh data

The output is Detour-compatible navmesh tile data which can be loaded instantly later.

This part was critical: it turns raw scene geometry into an actual navigation representation that Detour can query and pathfind on.

Editor workflow: manual build button

Since baking is offline, I added an editor button:

Build NavMesh

This solves a common workflow reality: I often start the engine with an “empty” scene or load levels manually while testing.

So the bake happens when I want it, on the currently loaded scene — and the result is written to:

Assets/NavMeshes/<SceneName>.navmesh

Debugging the NavMesh visually (dd::line)

A NavMesh that exists but can’t be visualized is basically impossible to trust.

So I integrated a debug draw mode in the scene viewport:

• NavMesh edges drawn as green debug lines

• Toggleable via an editor checkbox (stored as a runtime flag in the navigation system)

This immediately answered the most important questions:

• Is the mesh being built at all?

• Does it match the actual level geometry?

• Are walkable areas and holes where I expect them?

Click-to-test path tool (Detour queries + visualization)

Once the NavMesh was visible, the next step was proving the runtime querying worked.

I implemented a simple but powerful editor tool:

• left click sets start

• right click sets end

• Detour computes a path:

  • findNearestPoly

  • findPath

  • findStraightPath

• the resulting path is drawn in yellow lines

This became my primary “sanity test” tool: if clicking two points gives a correct path, then the whole nav system is structurally sound.

NavMesh-constrained movement component

Finally, I implemented a movement component that behaves like the engine’s basic walk controller, but constrained to the NavMesh.

The important part is the translation step:

• compute a desired position from input

• project/move along the NavMesh surface using Detour:

  • findNearestPoly

  • moveAlongSurface

  • getPolyHeight

• apply the corrected position to the GameObject transform

The result is very practical: the object cannot drift outside walkable areas and will naturally “slide” along borders.

This is a key piece of gameplay movement logic and also a stepping stone for AI agents.

Final result

By the end of this phase, the engine could:

• build a NavMesh offline from scene geometry,

• store it as a binary resource per scene,

• load it at runtime into Detour (dtNavMesh, dtNavMeshQuery),

• visualize the NavMesh in-editor with a toggle,

• click two points to generate and draw a path,

• and constrain movement to the NavMesh surface.

In short: I now have a complete navigation foundation that supports both debugging and real gameplay movement rules.

https://www.youtube.com/watch?v=9sZJYQGYARY

What was still missing

At this stage, the system is foundational. Some major next steps are:

• agent profiles (radius/height/speed) so multiple character types can query correctly

• a true AI Agent component that moves autonomously toward targets

• optional DetourCrowd integration for avoidance

• optional tile cache / dynamic obstacles (future)

Closing thoughts

This task felt like one of those “engine glue” milestones: it wasn’t just one feature, it was a pipeline.

It required building infrastructure across:

• third-party integration

• scene geometry extraction

• offline resource baking

• runtime query setup

• editor tooling

• debug visualization

• and constrained movement

Once this is in place, everything AI-related becomes more manageable — because you can finally see what’s happening and verify correctness step by step.