Priority Inversion on an RTOS, and How to Provoke It on Purpose

Priority inversion is usually introduced to people as a theoretical hazard, which is a shame, because it is far more useful as something you can reproduce on your desk in twenty minutes. Once you have seen it happen, you will not ship a bare mutex again.

What goes wrong

Three tasks, one shared resource:

• Low (prio 1) — does almost nothing, but occasionally takes bus_mutex
• Medium (prio 2) — does the real work
• High (prio 3) — must respond within 2 ms, also needs bus_mutex

The medium task holds the mutex. Then the high-priority task becomes ready and preempts it — and immediately blocks on the mutex. Now the low-priority task is the only thing that can run, and the high-priority task waits for it.

The inversion is not that the mutex is unfair. It is that a high-priority task’s deadline now depends on the lowest-priority task in the system, which by definition has more work queued behind it.

K_THREAD_DEFINE(low_id, low_fn, NULL, 256, 5, 0);
K_THREAD_DEFINE(mid_id, mid_fn, NULL, 256, 6, 0);
K_THREAD_DEFINE(high_id, high_fn, NULL, 256, 7, 0);

low_fn holds bus_mutex for 40 ms. high_fn asserts if it is not done inside 2 ms.

Proving it before you fix it

The fix is well known. The discipline is not: write the failing test first, so you know the fix actually did something.

static void test_high_latency(void)
{
    k_sleep(K_MSEC(100));            /* let low grab the mutex */
    k_thread_priority_set(high_id, 7); /* make high urgent now  */
    start = k_uptime_get_32();
    high_set_done(false);

    k_sem_take(&done_sem, K_FOREVER);
    elapsed = k_uptime_get_32() - start;

    zassert_true(elapsed < 2,
                 "high-priority task waited %u ms — inversion present", elapsed);
}

Run it, watch it fail with something like 40 ms, and only then reach for the fix. Without the failing run you are not fixing an inversion, you are adding a mutex protocol to a theory.

The fix, and its cost

Priority inheritance means a task holding a mutex temporarily borrows the priority of the highest-priority task waiting on it. low_fn runs at priority 7 for the duration of its critical section, finishes fast, and hands the mutex back.

The cost is scheduling unpredictability — low_fn now runs at a priority it was never designed for. Two things follow:

1. Keep critical sections short. Priority inheritance amplifies whatever is inside the mutex. A long critical section becomes a long, high-priority, non-preemptible block.
2. Do not hold a mutex across a blocking call. Priority inheritance plus a blocking read inside a critical section can deadlock in configurations that look fine on paper.

Where it is not a mutex problem

Two adjacent traps that get misdiagnosed as inversions:

• Priority inheritance can be masked. A middleware layer that already uses a different mutex discipline will not show the bug, and the bug is still there in your own task.
• A long non-preemptible region is not an inversion but looks identical from a latency trace. Check with k_thread_priority_get() in a debugger before reaching for the mutex API.

The habit

Every mutex in a system gets one of three labels in the code review: short and safe, long — inherited priority is a hazard, or crosses a blocking call, needs redesign. There is no fourth category and no “probably fine”. The labelling is cheap, and it turns a latent scheduling bug into a comment someone will actually read.