rcu/tree: Allocate a page when caller is preemptible
The current memory-allocation interface poses the following challenges:
a) In kernels built with CONFIG_PROVE_RAW_LOCK_NESTING, lockdep
complains ("BUG: Invalid wait context"). This complaint is due
to the memory allocator acquiring non-raw spinlocks while a raw
spinlocks is held. This problem can also arise if kvfree_rcu()
is invoked while holding a raw spinlock.
b) In -rt kernels built with CONFIG_PREEMPT_RT, the situation
described in (a) above results in an attempt to acquire a
sleeplock while holding a spinlock, which is of course forbidden.
This can lead to "BUG: scheduling while atomic".
c) Please note that call_rcu() is invoked from raw atomic context,
so that kfree_rcu() and kvfree_rcu() are therefore also expected
to be callable from atomic raw context as well.
However given that CONFIG_PREEMPT_COUNT is unconditionally enabled
by the earlier commits in this series, the preemptible() macro now
properly detects preempt-disable code regions even in kernels built
with CONFIG_PREEMPT_NONE.
This commit therefore uses preemptible() to determine whether allocation
is possible at all for double-argument kvfree_rcu(). If !preemptible(),
then allocation is not possible, and kvfree_rcu() falls back to using
the less cache-friendly rcu_head approach. Even when preemptible(),
the caller might be involved in reclaim, so the GFP_ flags used by
double-argument kvfree_rcu() must avoid invoking reclaim processing.
Note that single-argument kvfree_rcu() must be invoked in sleepable
contexts, and that its fallback is the relatively high latency
synchronize_rcu(). Single-argument kvfree_rcu() therefore uses
GFP_KERNEL|__GFP_RETRY_MAYFAIL to allow limited sleeping within the
memory allocator.
Link: https://lore.kernel.org/lkml/20200630164543.4mdcf6zb4zfclhln@linutronix.de/
Fixes: 3042f83f19be ("rcu: Support reclaim for head-less object")
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
[ paulmck: Add add_ptr_to_bulk_krc_lock header comment per Michal Hocko. ]
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
1 file changed
