Building a Rust Game Engine: Part 1 - Project Setup

So you want to build a game engine in Rust. Bold choice. Let’s start with the foundation: project structure.

The Vision

We want three main components:

• Engine - The core library. Rendering, audio, physics, the works.
• Editor - A visual tool for building games (think Unity or Unreal Editor).
• Sandbox - A lightweight test app for rapid iteration.

For development, we want fast iteration. For shipping, we want optimized single-file executables. And we want clean output directories—each app isolated with only its dependencies.

Cargo Workspaces

Cargo workspaces let you manage multiple related crates in one repository. Create a root Cargo.toml:

[workspace]
members = ["editor", "engine", "sandbox", "xtask"]
resolver = "2"

[workspace.package]
version = "0.1.0"
edition = "2021"

[workspace.dependencies]
engine = { path = "engine" }

[profile.release]
lto = true

The workspace.dependencies section is key—it lets child crates reference the engine with just engine.workspace = true instead of repeating the path everywhere.

The Engine Crate

Here’s where it gets interesting. The engine’s Cargo.toml:

[package]
name = "engine"
version.workspace = true
edition.workspace = true

[lib]
crate-type = ["dylib", "rlib"]

[dependencies]

We’re building both a dynamic library (dylib) and a static library (rlib). Why?

dylib vs rlib

• rlib (Rust library) - Static linking. Gets baked directly into the executable. One file, no dependencies, maximum optimization with LTO.

• dylib (Dynamic library) - Produces a .dll (Windows), .so (Linux), or .dylib (macOS). Separate file that executables load at runtime.

By building both, we keep our options open. The DLL is there when we want it for hot-reloading scenarios. The rlib is there for release builds.

A Note on prefer-dynamic

You might think “I’ll just use -C prefer-dynamic in debug builds!”

Tried it. Here’s the catch: prefer-dynamic makes your executables depend on the Rust standard library DLLs too, not just your engine. Those live in your Rust toolchain directory, making distribution a headache.

For a game engine, the practical approach is:

• Debug builds include the DLL alongside executables (for future hot-reload support)
• Release/Dist builds use static linking—single executable, no DLL needed
• Release builds use LTO for maximum optimization

When you’re ready for true hot-reloading, you’ll want to use cdylib (C-compatible DLL) with runtime loading via libloading. That’s a topic for another post.

Editor and Sandbox

These are straightforward binary crates:

# editor/Cargo.toml
[package]
name = "editor"
version.workspace = true
edition.workspace = true

[dependencies]
engine.workspace = true

// editor/src/main.rs
fn main() {
    engine::init();
    println!("Editor started");
    engine::run();
    engine::shutdown();
}

Same pattern for sandbox. They depend on the engine, Cargo handles the rest.

The xtask Pattern

Raw cargo build dumps everything into target/debug/ or target/release/. Fine for simple projects, but game engines need more structure. We want:

bin/
└── Debug-x64/
    ├── Engine/
    │   └── engine.dll
    ├── Editor/
    │   ├── editor.exe
    │   └── engine.dll
    └── Sandbox/
        ├── sandbox.exe
        └── engine.dll

Each app gets its own folder with only what it needs to run. Clean. Distributable.

Enter xtask—a convention where you add a helper binary to your workspace:

# xtask/Cargo.toml
[package]
name = "xtask"
version = "0.1.0"
edition = "2021"

Then alias it in .cargo/config.toml:

[alias]
xtask = "run --package xtask --"

Our xtask handles:

1. Running cargo build
2. Creating the output directory structure (Debug-x64/, Release-x64/)
3. Copying executables (and DLLs for debug builds only)
4. Including PDB files for debugging on Windows

The core logic:

fn build(args: &[String]) {
    let is_release = args.iter().any(|a| a == "--release");
    let profile = if is_release { "release" } else { "debug" };
    let arch = "x64"; // or detect from cfg!(target_arch)

    // Build with cargo
    Command::new("cargo").arg("build").status()?;

    // Create output structure under bin/
    let out_dir = format!("bin/{}-{}",
        if is_release { "Release" } else { "Debug" },
        arch
    );

    fs::create_dir_all(format!("{}/Editor", out_dir))?;
    fs::create_dir_all(format!("{}/Sandbox", out_dir))?;
    fs::create_dir_all(format!("{}/Engine", out_dir))?;

    // Copy artifacts
    fs::copy("target/debug/editor.exe", format!("{}/Editor/editor.exe", out_dir))?;

    // Only copy DLL for debug builds (release uses static linking)
    if !is_release {
        fs::copy("target/debug/engine.dll", format!("{}/Editor/engine.dll", out_dir))?;
    }
    // ... etc
}

This is the foundation for:

• Asset pipeline integration
• Shader compilation
• Platform-specific bundling
• CI/CD helpers

Build Commands

With everything set up:

# Debug build (with DLL + PDBs for development)
cargo xtask build

# Release build (static linking, LTO optimized)
cargo xtask build --release

# Distribution build (stripped, with assets)
cargo xtask build --dist

# Build and run
cargo xtask run --bin editor
cargo xtask run --bin sandbox --release

# Clean everything
cargo xtask clean

What We Get

bin/
├── Debug-x64/
│   ├── Engine/
│   │   ├── engine.dll
│   │   └── engine.pdb
│   ├── Editor/
│   │   ├── editor.exe
│   │   ├── editor.pdb
│   │   ├── engine.dll
│   │   └── engine.pdb
│   └── Sandbox/
│       ├── sandbox.exe
│       ├── sandbox.pdb
│       ├── engine.dll
│       └── engine.pdb
│
├── Release-x64/
│   ├── Editor/
│   │   └── editor.exe    (statically linked)
│   └── Sandbox/
│       └── sandbox.exe   (statically linked)
│
└── Dist-x64/
    ├── Editor/
    │   ├── editor.exe    (stripped)
    │   └── assets/
    │       ├── engine/   (from engine/assets)
    │       └── editor/   (from editor/assets)
    └── Sandbox/
        ├── sandbox.exe   (stripped)
        └── assets/
            ├── engine/   (from engine/assets)
            └── game/     (from sandbox/assets)

Three build configurations:

• Debug - DLL + PDBs for development and future hot-reload
• Release - Static linking for testing optimized builds
• Dist - Stripped symbols, bundled assets, ready to ship

Project Structure

GGEngine-rs/
├── .cargo/
│   └── config.toml       # xtask alias
├── .gitignore
├── Cargo.toml            # Workspace root
├── editor/
│   ├── Cargo.toml
│   └── src/main.rs
├── engine/
│   ├── Cargo.toml
│   └── src/lib.rs
├── sandbox/
│   ├── Cargo.toml
│   └── src/main.rs
└── xtask/
    ├── Cargo.toml
    └── src/main.rs

What’s Next

With the project structure in place, we can start building actual engine functionality:

• Window creation and input handling
• Rendering abstraction (Vulkan? wgpu?)
• Entity Component System
• Asset loading and hot-reloading

The foundation is boring but necessary. Now the fun begins.

---

This is part 1 of a series on building a game engine in Rust. Special thanks to Cherno for getting me interested in this stuff.