hazptr: Record per-CPU slot owner after claiming the slot

An interrupt that uses a hazard pointer while the interrupted task is
in the acquire slow path can leave its own context recorded as the
owner of the per-CPU slot the task then publishes into.  The next
context switch promotes the task's pointer with the interrupt's
context.

hazptr_get_free_percpu_slot() records the owner when it picks an empty
slot, but only Store B in __hazptr_try_acquire() claims the slot.  An
interrupt in between picks the same slot, as the lowest free one:

  Task on CPU 0                        IRQ on CPU 0
  -------------                        ------------
  hazptr_acquire(ctx)
    fast path fails
    __hazptr_acquire(ctx)
      picks item k (empty)
      item k->ctx.ctx = ctx
                                       hazptr_acquire(irq_ctx)
                                         claims item k
                                         item k->ctx.ctx = irq_ctx
                                       hazptr_release(irq_ctx)
                                         clears slot k
      __hazptr_try_acquire()
        Store B into slot k
      (item k still names irq_ctx)
  sleeps holding the pointer
  hazptr_note_context_switch()
    promotes item k with irq_ctx

The task gets there with k > 0 when it already holds an outer hazard
pointer in slot 0; no other task can, as its slot is promoted when it
is switched out.

The task gets there with k == 0 when the fast path leaves slot 0
empty: either its ptr_eq() check fails because an updater changed the
pointer, or the pointer is NULL and an updater publishes a non-NULL one
before the slow path loads it.

The promotion moves the task's pointer into irq_ctx's backup slot,
which nothing unchains, so hazptr_synchronize() on that pointer never
returns.  The task's ctx->slot still names slot k, so its
hazptr_release() clears whatever another reader on that CPU has since
published there.

If the interrupt detaches instead, as hazptr_torture_acquire() does,
the promotion chains irq_ctx's backup slot a second time while it is
still linked.  The overflow list then loops on that node, or follows
LIST_POISON1 once hazptr_unchain_backup_slot() removes it.

Record the owner after __hazptr_try_acquire() succeeds, as the fast
path in hazptr_acquire() does.  This should be safe because an
interrupt that used the slot has released or detached it by the time
the task resumes, and only hazptr_note_context_switch() reads the
owner, which cannot run before the task enables preemption.

Fixes: 50b568234163 ("hazptr: Implement Hazard Pointers")
Assisted-by: LLM
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Acked-by: Bradley Morgan <brads@mainlining.org>
Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
1 file changed