| diff --git a/Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst b/Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst |
| index 72f0f6fbd53c0..6f89cf1e567d0 100644 |
| --- a/Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst |
| +++ b/Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst |
| @@ -38,7 +38,7 @@ sections. |
| RCU-preempt Expedited Grace Periods |
| =================================== |
| |
| -``CONFIG_PREEMPT=y`` kernels implement RCU-preempt. |
| +``CONFIG_PREEMPTION=y`` kernels implement RCU-preempt. |
| The overall flow of the handling of a given CPU by an RCU-preempt |
| expedited grace period is shown in the following diagram: |
| |
| @@ -112,7 +112,7 @@ things. |
| RCU-sched Expedited Grace Periods |
| --------------------------------- |
| |
| -``CONFIG_PREEMPT=n`` kernels implement RCU-sched. The overall flow of |
| +``CONFIG_PREEMPTION=n`` kernels implement RCU-sched. The overall flow of |
| the handling of a given CPU by an RCU-sched expedited grace period is |
| shown in the following diagram: |
| |
| diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst |
| index 1ae79a10a8de6..17d38480ef5c5 100644 |
| --- a/Documentation/RCU/Design/Requirements/Requirements.rst |
| +++ b/Documentation/RCU/Design/Requirements/Requirements.rst |
| @@ -78,7 +78,7 @@ RCU treats a nested set as one big RCU read-side critical section. |
| Production-quality implementations of ``rcu_read_lock()`` and |
| ``rcu_read_unlock()`` are extremely lightweight, and in fact have |
| exactly zero overhead in Linux kernels built for production use with |
| -``CONFIG_PREEMPT=n``. |
| +``CONFIG_PREEMPTION=n``. |
| |
| This guarantee allows ordering to be enforced with extremely low |
| overhead to readers, for example: |
| @@ -1182,7 +1182,7 @@ and has become decreasingly so as memory sizes have expanded and memory |
| costs have plummeted. However, as I learned from Matt Mackall's |
| `bloatwatch <http://elinux.org/Linux_Tiny-FAQ>`__ efforts, memory |
| footprint is critically important on single-CPU systems with |
| -non-preemptible (``CONFIG_PREEMPT=n``) kernels, and thus `tiny |
| +non-preemptible (``CONFIG_PREEMPTION=n``) kernels, and thus `tiny |
| RCU <https://lkml.kernel.org/g/20090113221724.GA15307@linux.vnet.ibm.com>`__ |
| was born. Josh Triplett has since taken over the small-memory banner |
| with his `Linux kernel tinification <https://tiny.wiki.kernel.org/>`__ |
| @@ -1498,7 +1498,7 @@ limitations. |
| |
| Implementations of RCU for which ``rcu_read_lock()`` and |
| ``rcu_read_unlock()`` generate no code, such as Linux-kernel RCU when |
| -``CONFIG_PREEMPT=n``, can be nested arbitrarily deeply. After all, there |
| +``CONFIG_PREEMPTION=n``, can be nested arbitrarily deeply. After all, there |
| is no overhead. Except that if all these instances of |
| ``rcu_read_lock()`` and ``rcu_read_unlock()`` are visible to the |
| compiler, compilation will eventually fail due to exhausting memory, |
| @@ -1771,7 +1771,7 @@ implementation can be a no-op. |
| |
| However, once the scheduler has spawned its first kthread, this early |
| boot trick fails for ``synchronize_rcu()`` (as well as for |
| -``synchronize_rcu_expedited()``) in ``CONFIG_PREEMPT=y`` kernels. The |
| +``synchronize_rcu_expedited()``) in ``CONFIG_PREEMPTION=y`` kernels. The |
| reason is that an RCU read-side critical section might be preempted, |
| which means that a subsequent ``synchronize_rcu()`` really does have to |
| wait for something, as opposed to simply returning immediately. |
| @@ -2010,7 +2010,7 @@ the following: |
| 5 rcu_read_unlock(); |
| 6 do_something_with(v, user_v); |
| |
| -If the compiler did make this transformation in a ``CONFIG_PREEMPT=n`` kernel |
| +If the compiler did make this transformation in a ``CONFIG_PREEMPTION=n`` kernel |
| build, and if ``get_user()`` did page fault, the result would be a quiescent |
| state in the middle of an RCU read-side critical section. This misplaced |
| quiescent state could result in line 4 being a use-after-free access, |
| @@ -2289,10 +2289,10 @@ decides to throw at it. |
| |
| The Linux kernel is used for real-time workloads, especially in |
| conjunction with the `-rt |
| -patchset <https://rt.wiki.kernel.org/index.php/Main_Page>`__. The |
| +patchset <https://wiki.linuxfoundation.org/realtime/>`__. The |
| real-time-latency response requirements are such that the traditional |
| approach of disabling preemption across RCU read-side critical sections |
| -is inappropriate. Kernels built with ``CONFIG_PREEMPT=y`` therefore use |
| +is inappropriate. Kernels built with ``CONFIG_PREEMPTION=y`` therefore use |
| an RCU implementation that allows RCU read-side critical sections to be |
| preempted. This requirement made its presence known after users made it |
| clear that an earlier `real-time |
| @@ -2414,7 +2414,7 @@ includes ``rcu_read_lock_bh()``, ``rcu_read_unlock_bh()``, |
| ``call_rcu_bh()``, ``rcu_barrier_bh()``, and |
| ``rcu_read_lock_bh_held()``. However, the update-side APIs are now |
| simple wrappers for other RCU flavors, namely RCU-sched in |
| -CONFIG_PREEMPT=n kernels and RCU-preempt otherwise. |
| +CONFIG_PREEMPTION=n kernels and RCU-preempt otherwise. |
| |
| Sched Flavor (Historical) |
| ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| @@ -2432,11 +2432,11 @@ not have this property, given that any point in the code outside of an |
| RCU read-side critical section can be a quiescent state. Therefore, |
| *RCU-sched* was created, which follows “classic” RCU in that an |
| RCU-sched grace period waits for pre-existing interrupt and NMI |
| -handlers. In kernels built with ``CONFIG_PREEMPT=n``, the RCU and |
| +handlers. In kernels built with ``CONFIG_PREEMPTION=n``, the RCU and |
| RCU-sched APIs have identical implementations, while kernels built with |
| -``CONFIG_PREEMPT=y`` provide a separate implementation for each. |
| +``CONFIG_PREEMPTION=y`` provide a separate implementation for each. |
| |
| -Note well that in ``CONFIG_PREEMPT=y`` kernels, |
| +Note well that in ``CONFIG_PREEMPTION=y`` kernels, |
| ``rcu_read_lock_sched()`` and ``rcu_read_unlock_sched()`` disable and |
| re-enable preemption, respectively. This means that if there was a |
| preemption attempt during the RCU-sched read-side critical section, |
| @@ -2599,10 +2599,10 @@ userspace execution also delimit tasks-RCU read-side critical sections. |
| |
| The tasks-RCU API is quite compact, consisting only of |
| ``call_rcu_tasks()``, ``synchronize_rcu_tasks()``, and |
| -``rcu_barrier_tasks()``. In ``CONFIG_PREEMPT=n`` kernels, trampolines |
| +``rcu_barrier_tasks()``. In ``CONFIG_PREEMPTION=n`` kernels, trampolines |
| cannot be preempted, so these APIs map to ``call_rcu()``, |
| ``synchronize_rcu()``, and ``rcu_barrier()``, respectively. In |
| -``CONFIG_PREEMPT=y`` kernels, trampolines can be preempted, and these |
| +``CONFIG_PREEMPTION=y`` kernels, trampolines can be preempted, and these |
| three APIs are therefore implemented by separate functions that check |
| for voluntary context switches. |
| |
| diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.rst |
| index 2efed9926c3f9..7ed4956043bd9 100644 |
| --- a/Documentation/RCU/checklist.rst |
| +++ b/Documentation/RCU/checklist.rst |
| @@ -214,7 +214,7 @@ over a rather long period of time, but improvements are always welcome! |
| the rest of the system. |
| |
| 7. As of v4.20, a given kernel implements only one RCU flavor, |
| - which is RCU-sched for PREEMPT=n and RCU-preempt for PREEMPT=y. |
| + which is RCU-sched for PREEMPTION=n and RCU-preempt for PREEMPTION=y. |
| If the updater uses call_rcu() or synchronize_rcu(), |
| then the corresponding readers my use rcu_read_lock() and |
| rcu_read_unlock(), rcu_read_lock_bh() and rcu_read_unlock_bh(), |
| diff --git a/Documentation/RCU/rcubarrier.rst b/Documentation/RCU/rcubarrier.rst |
| index f64f4413a47c4..3b4a248774961 100644 |
| --- a/Documentation/RCU/rcubarrier.rst |
| +++ b/Documentation/RCU/rcubarrier.rst |
| @@ -9,7 +9,7 @@ RCU (read-copy update) is a synchronization mechanism that can be thought |
| of as a replacement for read-writer locking (among other things), but with |
| very low-overhead readers that are immune to deadlock, priority inversion, |
| and unbounded latency. RCU read-side critical sections are delimited |
| -by rcu_read_lock() and rcu_read_unlock(), which, in non-CONFIG_PREEMPT |
| +by rcu_read_lock() and rcu_read_unlock(), which, in non-CONFIG_PREEMPTION |
| kernels, generate no code whatsoever. |
| |
| This means that RCU writers are unaware of the presence of concurrent |
| @@ -329,10 +329,10 @@ Answer: This cannot happen. The reason is that on_each_cpu() has its last |
| to smp_call_function() and further to smp_call_function_on_cpu(), |
| causing this latter to spin until the cross-CPU invocation of |
| rcu_barrier_func() has completed. This by itself would prevent |
| - a grace period from completing on non-CONFIG_PREEMPT kernels, |
| + a grace period from completing on non-CONFIG_PREEMPTION kernels, |
| since each CPU must undergo a context switch (or other quiescent |
| state) before the grace period can complete. However, this is |
| - of no use in CONFIG_PREEMPT kernels. |
| + of no use in CONFIG_PREEMPTION kernels. |
| |
| Therefore, on_each_cpu() disables preemption across its call |
| to smp_call_function() and also across the local call to |
| diff --git a/Documentation/RCU/stallwarn.rst b/Documentation/RCU/stallwarn.rst |
| index c9ab6af4d3be9..e97d1b4876ef4 100644 |
| --- a/Documentation/RCU/stallwarn.rst |
| +++ b/Documentation/RCU/stallwarn.rst |
| @@ -25,7 +25,7 @@ warnings: |
| |
| - A CPU looping with bottom halves disabled. |
| |
| -- For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel |
| +- For !CONFIG_PREEMPTION kernels, a CPU looping anywhere in the kernel |
| without invoking schedule(). If the looping in the kernel is |
| really expected and desirable behavior, you might need to add |
| some calls to cond_resched(). |
| @@ -44,7 +44,7 @@ warnings: |
| result in the ``rcu_.*kthread starved for`` console-log message, |
| which will include additional debugging information. |
| |
| -- A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might |
| +- A CPU-bound real-time task in a CONFIG_PREEMPTION kernel, which might |
| happen to preempt a low-priority task in the middle of an RCU |
| read-side critical section. This is especially damaging if |
| that low-priority task is not permitted to run on any other CPU, |
| diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst |
| index fb3ff76c3e737..3b2b1479fd0f7 100644 |
| --- a/Documentation/RCU/whatisRCU.rst |
| +++ b/Documentation/RCU/whatisRCU.rst |
| @@ -684,7 +684,7 @@ Quick Quiz #1: |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| This section presents a "toy" RCU implementation that is based on |
| "classic RCU". It is also short on performance (but only for updates) and |
| -on features such as hotplug CPU and the ability to run in CONFIG_PREEMPT |
| +on features such as hotplug CPU and the ability to run in CONFIG_PREEMPTION |
| kernels. The definitions of rcu_dereference() and rcu_assign_pointer() |
| are the same as those shown in the preceding section, so they are omitted. |
| :: |
| @@ -740,7 +740,7 @@ Quick Quiz #2: |
| Quick Quiz #3: |
| If it is illegal to block in an RCU read-side |
| critical section, what the heck do you do in |
| - PREEMPT_RT, where normal spinlocks can block??? |
| + CONFIG_PREEMPT_RT, where normal spinlocks can block??? |
| |
| :ref:`Answers to Quick Quiz <8_whatisRCU>` |
| |
| @@ -1094,7 +1094,7 @@ Quick Quiz #2: |
| overhead is **negative**. |
| |
| Answer: |
| - Imagine a single-CPU system with a non-CONFIG_PREEMPT |
| + Imagine a single-CPU system with a non-CONFIG_PREEMPTION |
| kernel where a routing table is used by process-context |
| code, but can be updated by irq-context code (for example, |
| by an "ICMP REDIRECT" packet). The usual way of handling |
| @@ -1121,10 +1121,10 @@ Answer: |
| Quick Quiz #3: |
| If it is illegal to block in an RCU read-side |
| critical section, what the heck do you do in |
| - PREEMPT_RT, where normal spinlocks can block??? |
| + CONFIG_PREEMPT_RT, where normal spinlocks can block??? |
| |
| Answer: |
| - Just as PREEMPT_RT permits preemption of spinlock |
| + Just as CONFIG_PREEMPT_RT permits preemption of spinlock |
| critical sections, it permits preemption of RCU |
| read-side critical sections. It also permits |
| spinlocks blocking while in RCU read-side critical |
| diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt |
| index d00618967854d..4ee4dca81fbda 100644 |
| --- a/Documentation/admin-guide/kernel-parameters.txt |
| +++ b/Documentation/admin-guide/kernel-parameters.txt |
| @@ -4097,6 +4097,10 @@ |
| value, meaning that RCU_SOFTIRQ is used by default. |
| Specify rcutree.use_softirq=0 to use rcuc kthreads. |
| |
| + But note that CONFIG_PREEMPT_RT=y kernels disable |
| + this kernel boot parameter, forcibly setting it |
| + to zero. |
| + |
| rcutree.rcu_fanout_exact= [KNL] |
| Disable autobalancing of the rcu_node combining |
| tree. This is used by rcutorture, and might |
| @@ -4475,6 +4479,13 @@ |
| only normal grace-period primitives. No effect |
| on CONFIG_TINY_RCU kernels. |
| |
| + But note that CONFIG_PREEMPT_RT=y kernels enables |
| + this kernel boot parameter, forcibly setting |
| + it to the value one, that is, converting any |
| + post-boot attempt at an expedited RCU grace |
| + period to instead use normal non-expedited |
| + grace-period processing. |
| + |
| rcupdate.rcu_task_ipi_delay= [KNL] |
| Set time in jiffies during which RCU tasks will |
| avoid sending IPIs, starting with the beginning |
| diff --git a/Documentation/driver-api/io-mapping.rst b/Documentation/driver-api/io-mapping.rst |
| index a966239f04e48..a7830c59481f6 100644 |
| --- a/Documentation/driver-api/io-mapping.rst |
| +++ b/Documentation/driver-api/io-mapping.rst |
| @@ -20,78 +20,64 @@ A mapping object is created during driver initialization using:: |
| mappable, while 'size' indicates how large a mapping region to |
| enable. Both are in bytes. |
| |
| -This _wc variant provides a mapping which may only be used |
| -with the io_mapping_map_atomic_wc or io_mapping_map_wc. |
| +This _wc variant provides a mapping which may only be used with |
| +io_mapping_map_local_wc() or io_mapping_map_wc(). |
| |
| -With this mapping object, individual pages can be mapped either atomically |
| -or not, depending on the necessary scheduling environment. Of course, atomic |
| -maps are more efficient:: |
| +With this mapping object, individual pages can be mapped either temporarily |
| +or long term, depending on the requirements. Of course, temporary maps are |
| +more efficient. |
| |
| - void *io_mapping_map_atomic_wc(struct io_mapping *mapping, |
| - unsigned long offset) |
| + void *io_mapping_map_local_wc(struct io_mapping *mapping, |
| + unsigned long offset) |
| |
| -'offset' is the offset within the defined mapping region. |
| -Accessing addresses beyond the region specified in the |
| -creation function yields undefined results. Using an offset |
| -which is not page aligned yields an undefined result. The |
| -return value points to a single page in CPU address space. |
| +'offset' is the offset within the defined mapping region. Accessing |
| +addresses beyond the region specified in the creation function yields |
| +undefined results. Using an offset which is not page aligned yields an |
| +undefined result. The return value points to a single page in CPU address |
| +space. |
| |
| -This _wc variant returns a write-combining map to the |
| -page and may only be used with mappings created by |
| -io_mapping_create_wc |
| +This _wc variant returns a write-combining map to the page and may only be |
| +used with mappings created by io_mapping_create_wc() |
| |
| -Note that the task may not sleep while holding this page |
| -mapped. |
| +Temporary mappings are only valid in the context of the caller. The mapping |
| +is not guaranteed to be globaly visible. |
| |
| -:: |
| +io_mapping_map_local_wc() has a side effect on X86 32bit as it disables |
| +migration to make the mapping code work. No caller can rely on this side |
| +effect. |
| |
| - void io_mapping_unmap_atomic(void *vaddr) |
| +Nested mappings need to be undone in reverse order because the mapping |
| +code uses a stack for keeping track of them:: |
| |
| -'vaddr' must be the value returned by the last |
| -io_mapping_map_atomic_wc call. This unmaps the specified |
| -page and allows the task to sleep once again. |
| + addr1 = io_mapping_map_local_wc(map1, offset1); |
| + addr2 = io_mapping_map_local_wc(map2, offset2); |
| + ... |
| + io_mapping_unmap_local(addr2); |
| + io_mapping_unmap_local(addr1); |
| |
| -If you need to sleep while holding the lock, you can use the non-atomic |
| -variant, although they may be significantly slower. |
| +The mappings are released with:: |
| |
| -:: |
| + void io_mapping_unmap_local(void *vaddr) |
| + |
| +'vaddr' must be the value returned by the last io_mapping_map_local_wc() |
| +call. This unmaps the specified mapping and undoes eventual side effects of |
| +the mapping function. |
| + |
| +If you need to sleep while holding a mapping, you can use the regular |
| +variant, although this may be significantly slower:: |
| |
| void *io_mapping_map_wc(struct io_mapping *mapping, |
| unsigned long offset) |
| |
| -This works like io_mapping_map_atomic_wc except it allows |
| -the task to sleep while holding the page mapped. |
| +This works like io_mapping_map_local_wc() except it has no side effects and |
| +the pointer is globaly visible. |
| |
| - |
| -:: |
| +The mappings are released with:: |
| |
| void io_mapping_unmap(void *vaddr) |
| |
| -This works like io_mapping_unmap_atomic, except it is used |
| -for pages mapped with io_mapping_map_wc. |
| +Use for pages mapped with io_mapping_map_wc(). |
| |
| At driver close time, the io_mapping object must be freed:: |
| |
| void io_mapping_free(struct io_mapping *mapping) |
| - |
| -Current Implementation |
| -====================== |
| - |
| -The initial implementation of these functions uses existing mapping |
| -mechanisms and so provides only an abstraction layer and no new |
| -functionality. |
| - |
| -On 64-bit processors, io_mapping_create_wc calls ioremap_wc for the whole |
| -range, creating a permanent kernel-visible mapping to the resource. The |
| -map_atomic and map functions add the requested offset to the base of the |
| -virtual address returned by ioremap_wc. |
| - |
| -On 32-bit processors with HIGHMEM defined, io_mapping_map_atomic_wc uses |
| -kmap_atomic_pfn to map the specified page in an atomic fashion; |
| -kmap_atomic_pfn isn't really supposed to be used with device pages, but it |
| -provides an efficient mapping for this usage. |
| - |
| -On 32-bit processors without HIGHMEM defined, io_mapping_map_atomic_wc and |
| -io_mapping_map_wc both use ioremap_wc, a terribly inefficient function which |
| -performs an IPI to inform all processors about the new mapping. This results |
| -in a significant performance penalty. |
| diff --git a/arch/Kconfig b/arch/Kconfig |
| index 632d60e13494c..c4713403ae3f7 100644 |
| --- a/arch/Kconfig |
| +++ b/arch/Kconfig |
| @@ -37,6 +37,7 @@ config OPROFILE |
| tristate "OProfile system profiling" |
| depends on PROFILING |
| depends on HAVE_OPROFILE |
| + depends on !PREEMPT_RT |
| select RING_BUFFER |
| select RING_BUFFER_ALLOW_SWAP |
| help |
| @@ -643,6 +644,12 @@ config HAVE_TIF_NOHZ |
| config HAVE_VIRT_CPU_ACCOUNTING |
| bool |
| |
| +config HAVE_VIRT_CPU_ACCOUNTING_IDLE |
| + bool |
| + help |
| + Architecture has its own way to account idle CPU time and therefore |
| + doesn't implement vtime_account_idle(). |
| + |
| config ARCH_HAS_SCALED_CPUTIME |
| bool |
| |
| @@ -657,7 +664,6 @@ config HAVE_VIRT_CPU_ACCOUNTING_GEN |
| some 32-bit arches may require multiple accesses, so proper |
| locking is needed to protect against concurrent accesses. |
| |
| - |
| config HAVE_IRQ_TIME_ACCOUNTING |
| bool |
| help |
| diff --git a/arch/alpha/include/asm/kmap_types.h b/arch/alpha/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index 651714b45729a..0000000000000 |
| --- a/arch/alpha/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,15 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0 */ |
| -#ifndef _ASM_KMAP_TYPES_H |
| -#define _ASM_KMAP_TYPES_H |
| - |
| -/* Dummy header just to define km_type. */ |
| - |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| -#define __WITH_KM_FENCE |
| -#endif |
| - |
| -#include <asm-generic/kmap_types.h> |
| - |
| -#undef __WITH_KM_FENCE |
| - |
| -#endif |
| diff --git a/arch/alpha/include/asm/spinlock_types.h b/arch/alpha/include/asm/spinlock_types.h |
| index 1d5716bc060be..6883bc952d224 100644 |
| --- a/arch/alpha/include/asm/spinlock_types.h |
| +++ b/arch/alpha/include/asm/spinlock_types.h |
| @@ -2,10 +2,6 @@ |
| #ifndef _ALPHA_SPINLOCK_TYPES_H |
| #define _ALPHA_SPINLOCK_TYPES_H |
| |
| -#ifndef __LINUX_SPINLOCK_TYPES_H |
| -# error "please don't include this file directly" |
| -#endif |
| - |
| typedef struct { |
| volatile unsigned int lock; |
| } arch_spinlock_t; |
| diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig |
| index 0a89cc9def651..d8804001d5507 100644 |
| --- a/arch/arc/Kconfig |
| +++ b/arch/arc/Kconfig |
| @@ -507,6 +507,7 @@ config LINUX_RAM_BASE |
| config HIGHMEM |
| bool "High Memory Support" |
| select ARCH_DISCONTIGMEM_ENABLE |
| + select KMAP_LOCAL |
| help |
| With ARC 2G:2G address split, only upper 2G is directly addressable by |
| kernel. Enable this to potentially allow access to rest of 2G and PAE |
| diff --git a/arch/arc/include/asm/highmem.h b/arch/arc/include/asm/highmem.h |
| index 6e5eafb3afdd4..a6b8e2c352c44 100644 |
| --- a/arch/arc/include/asm/highmem.h |
| +++ b/arch/arc/include/asm/highmem.h |
| @@ -9,17 +9,29 @@ |
| #ifdef CONFIG_HIGHMEM |
| |
| #include <uapi/asm/page.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| + |
| +#define FIXMAP_SIZE PGDIR_SIZE |
| +#define PKMAP_SIZE PGDIR_SIZE |
| |
| /* start after vmalloc area */ |
| #define FIXMAP_BASE (PAGE_OFFSET - FIXMAP_SIZE - PKMAP_SIZE) |
| -#define FIXMAP_SIZE PGDIR_SIZE /* only 1 PGD worth */ |
| -#define KM_TYPE_NR ((FIXMAP_SIZE >> PAGE_SHIFT)/NR_CPUS) |
| -#define FIXMAP_ADDR(nr) (FIXMAP_BASE + ((nr) << PAGE_SHIFT)) |
| + |
| +#define FIX_KMAP_SLOTS (KM_MAX_IDX * NR_CPUS) |
| +#define FIX_KMAP_BEGIN (0UL) |
| +#define FIX_KMAP_END ((FIX_KMAP_BEGIN + FIX_KMAP_SLOTS) - 1) |
| + |
| +#define FIXADDR_TOP (FIXMAP_BASE + (FIX_KMAP_END << PAGE_SHIFT)) |
| + |
| +/* |
| + * This should be converted to the asm-generic version, but of course this |
| + * is needlessly different from all other architectures. Sigh - tglx |
| + */ |
| +#define __fix_to_virt(x) (FIXADDR_TOP - ((x) << PAGE_SHIFT)) |
| +#define __virt_to_fix(x) (((FIXADDR_TOP - ((x) & PAGE_MASK))) >> PAGE_SHIFT) |
| |
| /* start after fixmap area */ |
| #define PKMAP_BASE (FIXMAP_BASE + FIXMAP_SIZE) |
| -#define PKMAP_SIZE PGDIR_SIZE |
| #define LAST_PKMAP (PKMAP_SIZE >> PAGE_SHIFT) |
| #define LAST_PKMAP_MASK (LAST_PKMAP - 1) |
| #define PKMAP_ADDR(nr) (PKMAP_BASE + ((nr) << PAGE_SHIFT)) |
| @@ -29,11 +41,13 @@ |
| |
| extern void kmap_init(void); |
| |
| +#define arch_kmap_local_post_unmap(vaddr) \ |
| + local_flush_tlb_kernel_range(vaddr, vaddr + PAGE_SIZE) |
| + |
| static inline void flush_cache_kmaps(void) |
| { |
| flush_cache_all(); |
| } |
| - |
| #endif |
| |
| #endif |
| diff --git a/arch/arc/include/asm/kmap_types.h b/arch/arc/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index fecf7851ec322..0000000000000 |
| --- a/arch/arc/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,14 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0-only */ |
| -/* |
| - * Copyright (C) 2015 Synopsys, Inc. (www.synopsys.com) |
| - */ |
| - |
| -#ifndef _ASM_KMAP_TYPES_H |
| -#define _ASM_KMAP_TYPES_H |
| - |
| -/* |
| - * We primarily need to define KM_TYPE_NR here but that in turn |
| - * is a function of PGDIR_SIZE etc. |
| - * To avoid circular deps issue, put everything in asm/highmem.h |
| - */ |
| -#endif |
| diff --git a/arch/arc/mm/highmem.c b/arch/arc/mm/highmem.c |
| index 1b9f473c63693..c79912a6b1961 100644 |
| --- a/arch/arc/mm/highmem.c |
| +++ b/arch/arc/mm/highmem.c |
| @@ -36,9 +36,8 @@ |
| * This means each only has 1 PGDIR_SIZE worth of kvaddr mappings, which means |
| * 2M of kvaddr space for typical config (8K page and 11:8:13 traversal split) |
| * |
| - * - fixmap anyhow needs a limited number of mappings. So 2M kvaddr == 256 PTE |
| - * slots across NR_CPUS would be more than sufficient (generic code defines |
| - * KM_TYPE_NR as 20). |
| + * - The fixed KMAP slots for kmap_local/atomic() require KM_MAX_IDX slots per |
| + * CPU. So the number of CPUs sharing a single PTE page is limited. |
| * |
| * - pkmap being preemptible, in theory could do with more than 256 concurrent |
| * mappings. However, generic pkmap code: map_new_virtual(), doesn't traverse |
| @@ -47,48 +46,6 @@ |
| */ |
| |
| extern pte_t * pkmap_page_table; |
| -static pte_t * fixmap_page_table; |
| - |
| -void *kmap_atomic_high_prot(struct page *page, pgprot_t prot) |
| -{ |
| - int idx, cpu_idx; |
| - unsigned long vaddr; |
| - |
| - cpu_idx = kmap_atomic_idx_push(); |
| - idx = cpu_idx + KM_TYPE_NR * smp_processor_id(); |
| - vaddr = FIXMAP_ADDR(idx); |
| - |
| - set_pte_at(&init_mm, vaddr, fixmap_page_table + idx, |
| - mk_pte(page, prot)); |
| - |
| - return (void *)vaddr; |
| -} |
| -EXPORT_SYMBOL(kmap_atomic_high_prot); |
| - |
| -void kunmap_atomic_high(void *kv) |
| -{ |
| - unsigned long kvaddr = (unsigned long)kv; |
| - |
| - if (kvaddr >= FIXMAP_BASE && kvaddr < (FIXMAP_BASE + FIXMAP_SIZE)) { |
| - |
| - /* |
| - * Because preemption is disabled, this vaddr can be associated |
| - * with the current allocated index. |
| - * But in case of multiple live kmap_atomic(), it still relies on |
| - * callers to unmap in right order. |
| - */ |
| - int cpu_idx = kmap_atomic_idx(); |
| - int idx = cpu_idx + KM_TYPE_NR * smp_processor_id(); |
| - |
| - WARN_ON(kvaddr != FIXMAP_ADDR(idx)); |
| - |
| - pte_clear(&init_mm, kvaddr, fixmap_page_table + idx); |
| - local_flush_tlb_kernel_range(kvaddr, kvaddr + PAGE_SIZE); |
| - |
| - kmap_atomic_idx_pop(); |
| - } |
| -} |
| -EXPORT_SYMBOL(kunmap_atomic_high); |
| |
| static noinline pte_t * __init alloc_kmap_pgtable(unsigned long kvaddr) |
| { |
| @@ -108,10 +65,9 @@ void __init kmap_init(void) |
| { |
| /* Due to recursive include hell, we can't do this in processor.h */ |
| BUILD_BUG_ON(PAGE_OFFSET < (VMALLOC_END + FIXMAP_SIZE + PKMAP_SIZE)); |
| + BUILD_BUG_ON(LAST_PKMAP > PTRS_PER_PTE); |
| + BUILD_BUG_ON(FIX_KMAP_SLOTS > PTRS_PER_PTE); |
| |
| - BUILD_BUG_ON(KM_TYPE_NR > PTRS_PER_PTE); |
| pkmap_page_table = alloc_kmap_pgtable(PKMAP_BASE); |
| - |
| - BUILD_BUG_ON(LAST_PKMAP > PTRS_PER_PTE); |
| - fixmap_page_table = alloc_kmap_pgtable(FIXMAP_BASE); |
| + alloc_kmap_pgtable(FIXMAP_BASE); |
| } |
| diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig |
| index a0eac00e2c81a..6bcbefad00911 100644 |
| --- a/arch/arm/Kconfig |
| +++ b/arch/arm/Kconfig |
| @@ -31,6 +31,7 @@ config ARM |
| select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX |
| select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT if CPU_V7 |
| select ARCH_SUPPORTS_ATOMIC_RMW |
| + select ARCH_SUPPORTS_RT if HAVE_POSIX_CPU_TIMERS_TASK_WORK |
| select ARCH_USE_BUILTIN_BSWAP |
| select ARCH_USE_CMPXCHG_LOCKREF |
| select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU |
| @@ -66,7 +67,7 @@ config ARM |
| select HARDIRQS_SW_RESEND |
| select HAVE_ARCH_AUDITSYSCALL if AEABI && !OABI_COMPAT |
| select HAVE_ARCH_BITREVERSE if (CPU_32v7M || CPU_32v7) && !CPU_32v6 |
| - select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL && !CPU_ENDIAN_BE32 && MMU |
| + select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL && !CPU_ENDIAN_BE32 && MMU && !PREEMPT_RT |
| select HAVE_ARCH_KGDB if !CPU_ENDIAN_BE32 && MMU |
| select HAVE_ARCH_MMAP_RND_BITS if MMU |
| select HAVE_ARCH_SECCOMP |
| @@ -106,6 +107,7 @@ config ARM |
| select HAVE_PERF_EVENTS |
| select HAVE_PERF_REGS |
| select HAVE_PERF_USER_STACK_DUMP |
| + select HAVE_PREEMPT_LAZY |
| select MMU_GATHER_RCU_TABLE_FREE if SMP && ARM_LPAE |
| select HAVE_REGS_AND_STACK_ACCESS_API |
| select HAVE_RSEQ |
| @@ -121,6 +123,7 @@ config ARM |
| select OLD_SIGSUSPEND3 |
| select PCI_SYSCALL if PCI |
| select PERF_USE_VMALLOC |
| + select HAVE_POSIX_CPU_TIMERS_TASK_WORK if !KVM |
| select RTC_LIB |
| select SET_FS |
| select SYS_SUPPORTS_APM_EMULATION |
| @@ -1500,6 +1503,7 @@ config HAVE_ARCH_PFN_VALID |
| config HIGHMEM |
| bool "High Memory Support" |
| depends on MMU |
| + select KMAP_LOCAL |
| help |
| The address space of ARM processors is only 4 Gigabytes large |
| and it has to accommodate user address space, kernel address |
| diff --git a/arch/arm/include/asm/fixmap.h b/arch/arm/include/asm/fixmap.h |
| index 9575b404019c9..707068f852c27 100644 |
| --- a/arch/arm/include/asm/fixmap.h |
| +++ b/arch/arm/include/asm/fixmap.h |
| @@ -7,14 +7,14 @@ |
| #define FIXADDR_TOP (FIXADDR_END - PAGE_SIZE) |
| |
| #include <linux/pgtable.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| |
| enum fixed_addresses { |
| FIX_EARLYCON_MEM_BASE, |
| __end_of_permanent_fixed_addresses, |
| |
| FIX_KMAP_BEGIN = __end_of_permanent_fixed_addresses, |
| - FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_TYPE_NR * NR_CPUS) - 1, |
| + FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_MAX_IDX * NR_CPUS) - 1, |
| |
| /* Support writing RO kernel text via kprobes, jump labels, etc. */ |
| FIX_TEXT_POKE0, |
| diff --git a/arch/arm/include/asm/hardirq.h b/arch/arm/include/asm/hardirq.h |
| index b95848ed2bc7f..706efafbf9722 100644 |
| --- a/arch/arm/include/asm/hardirq.h |
| +++ b/arch/arm/include/asm/hardirq.h |
| @@ -2,16 +2,11 @@ |
| #ifndef __ASM_HARDIRQ_H |
| #define __ASM_HARDIRQ_H |
| |
| -#include <linux/cache.h> |
| -#include <linux/threads.h> |
| #include <asm/irq.h> |
| |
| -typedef struct { |
| - unsigned int __softirq_pending; |
| -} ____cacheline_aligned irq_cpustat_t; |
| - |
| -#include <linux/irq_cpustat.h> /* Standard mappings for irq_cpustat_t above */ |
| - |
| #define __ARCH_IRQ_EXIT_IRQS_DISABLED 1 |
| +#define ack_bad_irq ack_bad_irq |
| + |
| +#include <asm-generic/hardirq.h> |
| |
| #endif /* __ASM_HARDIRQ_H */ |
| diff --git a/arch/arm/include/asm/highmem.h b/arch/arm/include/asm/highmem.h |
| index 31811be38d781..b22dffa8c7eb9 100644 |
| --- a/arch/arm/include/asm/highmem.h |
| +++ b/arch/arm/include/asm/highmem.h |
| @@ -2,7 +2,8 @@ |
| #ifndef _ASM_HIGHMEM_H |
| #define _ASM_HIGHMEM_H |
| |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| +#include <asm/fixmap.h> |
| |
| #define PKMAP_BASE (PAGE_OFFSET - PMD_SIZE) |
| #define LAST_PKMAP PTRS_PER_PTE |
| @@ -46,19 +47,32 @@ extern pte_t *pkmap_page_table; |
| |
| #ifdef ARCH_NEEDS_KMAP_HIGH_GET |
| extern void *kmap_high_get(struct page *page); |
| -#else |
| + |
| +static inline void *arch_kmap_local_high_get(struct page *page) |
| +{ |
| + if (IS_ENABLED(CONFIG_DEBUG_HIGHMEM) && !cache_is_vivt()) |
| + return NULL; |
| + return kmap_high_get(page); |
| +} |
| +#define arch_kmap_local_high_get arch_kmap_local_high_get |
| + |
| +#else /* ARCH_NEEDS_KMAP_HIGH_GET */ |
| static inline void *kmap_high_get(struct page *page) |
| { |
| return NULL; |
| } |
| -#endif |
| +#endif /* !ARCH_NEEDS_KMAP_HIGH_GET */ |
| |
| -/* |
| - * The following functions are already defined by <linux/highmem.h> |
| - * when CONFIG_HIGHMEM is not set. |
| - */ |
| -#ifdef CONFIG_HIGHMEM |
| -extern void *kmap_atomic_pfn(unsigned long pfn); |
| -#endif |
| +#define arch_kmap_local_post_map(vaddr, pteval) \ |
| + local_flush_tlb_kernel_page(vaddr) |
| + |
| +#define arch_kmap_local_pre_unmap(vaddr) \ |
| +do { \ |
| + if (cache_is_vivt()) \ |
| + __cpuc_flush_dcache_area((void *)vaddr, PAGE_SIZE); \ |
| +} while (0) |
| + |
| +#define arch_kmap_local_post_unmap(vaddr) \ |
| + local_flush_tlb_kernel_page(vaddr) |
| |
| #endif |
| diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h |
| index 46d41140df27d..1cbcc462b07e8 100644 |
| --- a/arch/arm/include/asm/irq.h |
| +++ b/arch/arm/include/asm/irq.h |
| @@ -31,6 +31,8 @@ void handle_IRQ(unsigned int, struct pt_regs *); |
| void init_IRQ(void); |
| |
| #ifdef CONFIG_SMP |
| +#include <linux/cpumask.h> |
| + |
| extern void arch_trigger_cpumask_backtrace(const cpumask_t *mask, |
| bool exclude_self); |
| #define arch_trigger_cpumask_backtrace arch_trigger_cpumask_backtrace |
| diff --git a/arch/arm/include/asm/kmap_types.h b/arch/arm/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index 5590940ee43d7..0000000000000 |
| --- a/arch/arm/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,10 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0 */ |
| -#ifndef __ARM_KMAP_TYPES_H |
| -#define __ARM_KMAP_TYPES_H |
| - |
| -/* |
| - * This is the "bare minimum". AIO seems to require this. |
| - */ |
| -#define KM_TYPE_NR 16 |
| - |
| -#endif |
| diff --git a/arch/arm/include/asm/spinlock_types.h b/arch/arm/include/asm/spinlock_types.h |
| index 5976958647fe1..a37c0803954ba 100644 |
| --- a/arch/arm/include/asm/spinlock_types.h |
| +++ b/arch/arm/include/asm/spinlock_types.h |
| @@ -2,10 +2,6 @@ |
| #ifndef __ASM_SPINLOCK_TYPES_H |
| #define __ASM_SPINLOCK_TYPES_H |
| |
| -#ifndef __LINUX_SPINLOCK_TYPES_H |
| -# error "please don't include this file directly" |
| -#endif |
| - |
| #define TICKET_SHIFT 16 |
| |
| typedef struct { |
| diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h |
| index 536b6b979f634..875aaf9af9465 100644 |
| --- a/arch/arm/include/asm/thread_info.h |
| +++ b/arch/arm/include/asm/thread_info.h |
| @@ -46,6 +46,7 @@ struct cpu_context_save { |
| struct thread_info { |
| unsigned long flags; /* low level flags */ |
| int preempt_count; /* 0 => preemptable, <0 => bug */ |
| + int preempt_lazy_count; /* 0 => preemptable, <0 => bug */ |
| mm_segment_t addr_limit; /* address limit */ |
| struct task_struct *task; /* main task structure */ |
| __u32 cpu; /* cpu */ |
| @@ -134,7 +135,8 @@ extern int vfp_restore_user_hwstate(struct user_vfp *, |
| #define TIF_SYSCALL_TRACE 4 /* syscall trace active */ |
| #define TIF_SYSCALL_AUDIT 5 /* syscall auditing active */ |
| #define TIF_SYSCALL_TRACEPOINT 6 /* syscall tracepoint instrumentation */ |
| -#define TIF_SECCOMP 7 /* seccomp syscall filtering active */ |
| +#define TIF_NEED_RESCHED_LAZY 7 |
| +#define TIF_SECCOMP 8 /* seccomp syscall filtering active */ |
| |
| #define TIF_USING_IWMMXT 17 |
| #define TIF_MEMDIE 18 /* is terminating due to OOM killer */ |
| @@ -143,6 +145,7 @@ extern int vfp_restore_user_hwstate(struct user_vfp *, |
| #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) |
| #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) |
| #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) |
| +#define _TIF_NEED_RESCHED_LAZY (1 << TIF_NEED_RESCHED_LAZY) |
| #define _TIF_UPROBE (1 << TIF_UPROBE) |
| #define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE) |
| #define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT) |
| @@ -158,7 +161,8 @@ extern int vfp_restore_user_hwstate(struct user_vfp *, |
| * Change these and you break ASM code in entry-common.S |
| */ |
| #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ |
| - _TIF_NOTIFY_RESUME | _TIF_UPROBE) |
| + _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ |
| + _TIF_NEED_RESCHED_LAZY) |
| |
| #endif /* __KERNEL__ */ |
| #endif /* __ASM_ARM_THREAD_INFO_H */ |
| diff --git a/arch/arm/kernel/asm-offsets.c b/arch/arm/kernel/asm-offsets.c |
| index 70993af22d80c..024c65c3a0f23 100644 |
| --- a/arch/arm/kernel/asm-offsets.c |
| +++ b/arch/arm/kernel/asm-offsets.c |
| @@ -43,6 +43,7 @@ int main(void) |
| BLANK(); |
| DEFINE(TI_FLAGS, offsetof(struct thread_info, flags)); |
| DEFINE(TI_PREEMPT, offsetof(struct thread_info, preempt_count)); |
| + DEFINE(TI_PREEMPT_LAZY, offsetof(struct thread_info, preempt_lazy_count)); |
| DEFINE(TI_ADDR_LIMIT, offsetof(struct thread_info, addr_limit)); |
| DEFINE(TI_TASK, offsetof(struct thread_info, task)); |
| DEFINE(TI_CPU, offsetof(struct thread_info, cpu)); |
| diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S |
| index 63fbcdc97ded9..4a9594fdb0048 100644 |
| --- a/arch/arm/kernel/entry-armv.S |
| +++ b/arch/arm/kernel/entry-armv.S |
| @@ -206,11 +206,18 @@ __irq_svc: |
| |
| #ifdef CONFIG_PREEMPTION |
| ldr r8, [tsk, #TI_PREEMPT] @ get preempt count |
| - ldr r0, [tsk, #TI_FLAGS] @ get flags |
| teq r8, #0 @ if preempt count != 0 |
| + bne 1f @ return from exeption |
| + ldr r0, [tsk, #TI_FLAGS] @ get flags |
| + tst r0, #_TIF_NEED_RESCHED @ if NEED_RESCHED is set |
| + blne svc_preempt @ preempt! |
| + |
| + ldr r8, [tsk, #TI_PREEMPT_LAZY] @ get preempt lazy count |
| + teq r8, #0 @ if preempt lazy count != 0 |
| movne r0, #0 @ force flags to 0 |
| - tst r0, #_TIF_NEED_RESCHED |
| + tst r0, #_TIF_NEED_RESCHED_LAZY |
| blne svc_preempt |
| +1: |
| #endif |
| |
| svc_exit r5, irq = 1 @ return from exception |
| @@ -225,8 +232,14 @@ svc_preempt: |
| 1: bl preempt_schedule_irq @ irq en/disable is done inside |
| ldr r0, [tsk, #TI_FLAGS] @ get new tasks TI_FLAGS |
| tst r0, #_TIF_NEED_RESCHED |
| + bne 1b |
| + tst r0, #_TIF_NEED_RESCHED_LAZY |
| reteq r8 @ go again |
| - b 1b |
| + ldr r0, [tsk, #TI_PREEMPT_LAZY] @ get preempt lazy count |
| + teq r0, #0 @ if preempt lazy count != 0 |
| + beq 1b |
| + ret r8 @ go again |
| + |
| #endif |
| |
| __und_fault: |
| diff --git a/arch/arm/kernel/entry-common.S b/arch/arm/kernel/entry-common.S |
| index 271cb8a1eba1e..fd039b1b37319 100644 |
| --- a/arch/arm/kernel/entry-common.S |
| +++ b/arch/arm/kernel/entry-common.S |
| @@ -53,7 +53,9 @@ __ret_fast_syscall: |
| cmp r2, #TASK_SIZE |
| blne addr_limit_check_failed |
| ldr r1, [tsk, #TI_FLAGS] @ re-check for syscall tracing |
| - tst r1, #_TIF_SYSCALL_WORK | _TIF_WORK_MASK |
| + tst r1, #((_TIF_SYSCALL_WORK | _TIF_WORK_MASK) & ~_TIF_SECCOMP) |
| + bne fast_work_pending |
| + tst r1, #_TIF_SECCOMP |
| bne fast_work_pending |
| |
| |
| @@ -90,8 +92,11 @@ __ret_fast_syscall: |
| cmp r2, #TASK_SIZE |
| blne addr_limit_check_failed |
| ldr r1, [tsk, #TI_FLAGS] @ re-check for syscall tracing |
| - tst r1, #_TIF_SYSCALL_WORK | _TIF_WORK_MASK |
| + tst r1, #((_TIF_SYSCALL_WORK | _TIF_WORK_MASK) & ~_TIF_SECCOMP) |
| + bne do_slower_path |
| + tst r1, #_TIF_SECCOMP |
| beq no_work_pending |
| +do_slower_path: |
| UNWIND(.fnend ) |
| ENDPROC(ret_fast_syscall) |
| |
| diff --git a/arch/arm/kernel/signal.c b/arch/arm/kernel/signal.c |
| index 2f81d3af5f9af..6e69f7b3d5813 100644 |
| --- a/arch/arm/kernel/signal.c |
| +++ b/arch/arm/kernel/signal.c |
| @@ -649,7 +649,8 @@ do_work_pending(struct pt_regs *regs, unsigned int thread_flags, int syscall) |
| */ |
| trace_hardirqs_off(); |
| do { |
| - if (likely(thread_flags & _TIF_NEED_RESCHED)) { |
| + if (likely(thread_flags & (_TIF_NEED_RESCHED | |
| + _TIF_NEED_RESCHED_LAZY))) { |
| schedule(); |
| } else { |
| if (unlikely(!user_mode(regs))) |
| diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c |
| index 8aa7fa949c232..3693706ba0a1a 100644 |
| --- a/arch/arm/kernel/smp.c |
| +++ b/arch/arm/kernel/smp.c |
| @@ -671,9 +671,7 @@ static void do_handle_IPI(int ipinr) |
| break; |
| |
| case IPI_CPU_BACKTRACE: |
| - printk_nmi_enter(); |
| nmi_cpu_backtrace(get_irq_regs()); |
| - printk_nmi_exit(); |
| break; |
| |
| default: |
| diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile |
| index 7cb1699fbfc4f..c4ce477c52611 100644 |
| --- a/arch/arm/mm/Makefile |
| +++ b/arch/arm/mm/Makefile |
| @@ -19,7 +19,6 @@ obj-$(CONFIG_MODULES) += proc-syms.o |
| obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o |
| |
| obj-$(CONFIG_ALIGNMENT_TRAP) += alignment.o |
| -obj-$(CONFIG_HIGHMEM) += highmem.o |
| obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o |
| obj-$(CONFIG_ARM_PV_FIXUP) += pv-fixup-asm.o |
| |
| diff --git a/arch/arm/mm/cache-feroceon-l2.c b/arch/arm/mm/cache-feroceon-l2.c |
| index 5c1b7a7b9af63..87328766e9102 100644 |
| --- a/arch/arm/mm/cache-feroceon-l2.c |
| +++ b/arch/arm/mm/cache-feroceon-l2.c |
| @@ -49,9 +49,9 @@ static inline unsigned long l2_get_va(unsigned long paddr) |
| * we simply install a virtual mapping for it only for the |
| * TLB lookup to occur, hence no need to flush the untouched |
| * memory mapping afterwards (note: a cache flush may happen |
| - * in some circumstances depending on the path taken in kunmap_atomic). |
| + * in some circumstances depending on the path taken in kunmap_local). |
| */ |
| - void *vaddr = kmap_atomic_pfn(paddr >> PAGE_SHIFT); |
| + void *vaddr = kmap_local_pfn(paddr >> PAGE_SHIFT); |
| return (unsigned long)vaddr + (paddr & ~PAGE_MASK); |
| #else |
| return __phys_to_virt(paddr); |
| @@ -61,7 +61,7 @@ static inline unsigned long l2_get_va(unsigned long paddr) |
| static inline void l2_put_va(unsigned long vaddr) |
| { |
| #ifdef CONFIG_HIGHMEM |
| - kunmap_atomic((void *)vaddr); |
| + kunmap_local((void *)vaddr); |
| #endif |
| } |
| |
| diff --git a/arch/arm/mm/cache-xsc3l2.c b/arch/arm/mm/cache-xsc3l2.c |
| index d20d7af02d10f..0e0a3abd81743 100644 |
| --- a/arch/arm/mm/cache-xsc3l2.c |
| +++ b/arch/arm/mm/cache-xsc3l2.c |
| @@ -59,7 +59,7 @@ static inline void l2_unmap_va(unsigned long va) |
| { |
| #ifdef CONFIG_HIGHMEM |
| if (va != -1) |
| - kunmap_atomic((void *)va); |
| + kunmap_local((void *)va); |
| #endif |
| } |
| |
| @@ -75,7 +75,7 @@ static inline unsigned long l2_map_va(unsigned long pa, unsigned long prev_va) |
| * in place for it. |
| */ |
| l2_unmap_va(prev_va); |
| - va = (unsigned long)kmap_atomic_pfn(pa >> PAGE_SHIFT); |
| + va = (unsigned long)kmap_local_pfn(pa >> PAGE_SHIFT); |
| } |
| return va + (pa_offset >> (32 - PAGE_SHIFT)); |
| #else |
| diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c |
| index efa4020250315..59487ee9fd612 100644 |
| --- a/arch/arm/mm/fault.c |
| +++ b/arch/arm/mm/fault.c |
| @@ -400,6 +400,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr, |
| if (addr < TASK_SIZE) |
| return do_page_fault(addr, fsr, regs); |
| |
| + if (interrupts_enabled(regs)) |
| + local_irq_enable(); |
| + |
| if (user_mode(regs)) |
| goto bad_area; |
| |
| @@ -470,6 +473,9 @@ do_translation_fault(unsigned long addr, unsigned int fsr, |
| static int |
| do_sect_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) |
| { |
| + if (interrupts_enabled(regs)) |
| + local_irq_enable(); |
| + |
| do_bad_area(addr, fsr, regs); |
| return 0; |
| } |
| diff --git a/arch/arm/mm/highmem.c b/arch/arm/mm/highmem.c |
| deleted file mode 100644 |
| index 187fab227b509..0000000000000 |
| --- a/arch/arm/mm/highmem.c |
| +++ /dev/null |
| @@ -1,121 +0,0 @@ |
| -// SPDX-License-Identifier: GPL-2.0-only |
| -/* |
| - * arch/arm/mm/highmem.c -- ARM highmem support |
| - * |
| - * Author: Nicolas Pitre |
| - * Created: september 8, 2008 |
| - * Copyright: Marvell Semiconductors Inc. |
| - */ |
| - |
| -#include <linux/module.h> |
| -#include <linux/highmem.h> |
| -#include <linux/interrupt.h> |
| -#include <asm/fixmap.h> |
| -#include <asm/cacheflush.h> |
| -#include <asm/tlbflush.h> |
| -#include "mm.h" |
| - |
| -static inline void set_fixmap_pte(int idx, pte_t pte) |
| -{ |
| - unsigned long vaddr = __fix_to_virt(idx); |
| - pte_t *ptep = virt_to_kpte(vaddr); |
| - |
| - set_pte_ext(ptep, pte, 0); |
| - local_flush_tlb_kernel_page(vaddr); |
| -} |
| - |
| -static inline pte_t get_fixmap_pte(unsigned long vaddr) |
| -{ |
| - pte_t *ptep = virt_to_kpte(vaddr); |
| - |
| - return *ptep; |
| -} |
| - |
| -void *kmap_atomic_high_prot(struct page *page, pgprot_t prot) |
| -{ |
| - unsigned int idx; |
| - unsigned long vaddr; |
| - void *kmap; |
| - int type; |
| - |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - /* |
| - * There is no cache coherency issue when non VIVT, so force the |
| - * dedicated kmap usage for better debugging purposes in that case. |
| - */ |
| - if (!cache_is_vivt()) |
| - kmap = NULL; |
| - else |
| -#endif |
| - kmap = kmap_high_get(page); |
| - if (kmap) |
| - return kmap; |
| - |
| - type = kmap_atomic_idx_push(); |
| - |
| - idx = FIX_KMAP_BEGIN + type + KM_TYPE_NR * smp_processor_id(); |
| - vaddr = __fix_to_virt(idx); |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - /* |
| - * With debugging enabled, kunmap_atomic forces that entry to 0. |
| - * Make sure it was indeed properly unmapped. |
| - */ |
| - BUG_ON(!pte_none(get_fixmap_pte(vaddr))); |
| -#endif |
| - /* |
| - * When debugging is off, kunmap_atomic leaves the previous mapping |
| - * in place, so the contained TLB flush ensures the TLB is updated |
| - * with the new mapping. |
| - */ |
| - set_fixmap_pte(idx, mk_pte(page, prot)); |
| - |
| - return (void *)vaddr; |
| -} |
| -EXPORT_SYMBOL(kmap_atomic_high_prot); |
| - |
| -void kunmap_atomic_high(void *kvaddr) |
| -{ |
| - unsigned long vaddr = (unsigned long) kvaddr & PAGE_MASK; |
| - int idx, type; |
| - |
| - if (kvaddr >= (void *)FIXADDR_START) { |
| - type = kmap_atomic_idx(); |
| - idx = FIX_KMAP_BEGIN + type + KM_TYPE_NR * smp_processor_id(); |
| - |
| - if (cache_is_vivt()) |
| - __cpuc_flush_dcache_area((void *)vaddr, PAGE_SIZE); |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - BUG_ON(vaddr != __fix_to_virt(idx)); |
| - set_fixmap_pte(idx, __pte(0)); |
| -#else |
| - (void) idx; /* to kill a warning */ |
| -#endif |
| - kmap_atomic_idx_pop(); |
| - } else if (vaddr >= PKMAP_ADDR(0) && vaddr < PKMAP_ADDR(LAST_PKMAP)) { |
| - /* this address was obtained through kmap_high_get() */ |
| - kunmap_high(pte_page(pkmap_page_table[PKMAP_NR(vaddr)])); |
| - } |
| -} |
| -EXPORT_SYMBOL(kunmap_atomic_high); |
| - |
| -void *kmap_atomic_pfn(unsigned long pfn) |
| -{ |
| - unsigned long vaddr; |
| - int idx, type; |
| - struct page *page = pfn_to_page(pfn); |
| - |
| - preempt_disable(); |
| - pagefault_disable(); |
| - if (!PageHighMem(page)) |
| - return page_address(page); |
| - |
| - type = kmap_atomic_idx_push(); |
| - idx = FIX_KMAP_BEGIN + type + KM_TYPE_NR * smp_processor_id(); |
| - vaddr = __fix_to_virt(idx); |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - BUG_ON(!pte_none(get_fixmap_pte(vaddr))); |
| -#endif |
| - set_fixmap_pte(idx, pfn_pte(pfn, kmap_prot)); |
| - |
| - return (void *)vaddr; |
| -} |
| diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig |
| index 3da71fe56b922..3ba7b561d2564 100644 |
| --- a/arch/arm64/Kconfig |
| +++ b/arch/arm64/Kconfig |
| @@ -76,6 +76,7 @@ config ARM64 |
| select ARCH_SUPPORTS_ATOMIC_RMW |
| select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 && (GCC_VERSION >= 50000 || CC_IS_CLANG) |
| select ARCH_SUPPORTS_NUMA_BALANCING |
| + select ARCH_SUPPORTS_RT if HAVE_POSIX_CPU_TIMERS_TASK_WORK |
| select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT |
| select ARCH_WANT_DEFAULT_BPF_JIT |
| select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT |
| @@ -173,6 +174,7 @@ config ARM64 |
| select HAVE_PERF_EVENTS |
| select HAVE_PERF_REGS |
| select HAVE_PERF_USER_STACK_DUMP |
| + select HAVE_PREEMPT_LAZY |
| select HAVE_REGS_AND_STACK_ACCESS_API |
| select HAVE_FUNCTION_ARG_ACCESS_API |
| select HAVE_FUTEX_CMPXCHG if FUTEX |
| @@ -194,6 +196,7 @@ config ARM64 |
| select PCI_DOMAINS_GENERIC if PCI |
| select PCI_ECAM if (ACPI && PCI) |
| select PCI_SYSCALL if PCI |
| + select HAVE_POSIX_CPU_TIMERS_TASK_WORK if !KVM |
| select POWER_RESET |
| select POWER_SUPPLY |
| select SET_FS |
| diff --git a/arch/arm64/include/asm/hardirq.h b/arch/arm64/include/asm/hardirq.h |
| index 5ffa4bacdad38..cbfa7b6f2e098 100644 |
| --- a/arch/arm64/include/asm/hardirq.h |
| +++ b/arch/arm64/include/asm/hardirq.h |
| @@ -13,11 +13,8 @@ |
| #include <asm/kvm_arm.h> |
| #include <asm/sysreg.h> |
| |
| -typedef struct { |
| - unsigned int __softirq_pending; |
| -} ____cacheline_aligned irq_cpustat_t; |
| - |
| -#include <linux/irq_cpustat.h> /* Standard mappings for irq_cpustat_t above */ |
| +#define ack_bad_irq ack_bad_irq |
| +#include <asm-generic/hardirq.h> |
| |
| #define __ARCH_IRQ_EXIT_IRQS_DISABLED 1 |
| |
| diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h |
| index e83f0982b99c1..7a5770d825b91 100644 |
| --- a/arch/arm64/include/asm/preempt.h |
| +++ b/arch/arm64/include/asm/preempt.h |
| @@ -70,17 +70,43 @@ static inline bool __preempt_count_dec_and_test(void) |
| * interrupt occurring between the non-atomic READ_ONCE/WRITE_ONCE |
| * pair. |
| */ |
| - return !pc || !READ_ONCE(ti->preempt_count); |
| + if (!pc || !READ_ONCE(ti->preempt_count)) |
| + return true; |
| +#ifdef CONFIG_PREEMPT_LAZY |
| + if ((pc & ~PREEMPT_NEED_RESCHED)) |
| + return false; |
| + if (current_thread_info()->preempt_lazy_count) |
| + return false; |
| + return test_thread_flag(TIF_NEED_RESCHED_LAZY); |
| +#else |
| + return false; |
| +#endif |
| } |
| |
| static inline bool should_resched(int preempt_offset) |
| { |
| +#ifdef CONFIG_PREEMPT_LAZY |
| + u64 pc = READ_ONCE(current_thread_info()->preempt_count); |
| + if (pc == preempt_offset) |
| + return true; |
| + |
| + if ((pc & ~PREEMPT_NEED_RESCHED) != preempt_offset) |
| + return false; |
| + |
| + if (current_thread_info()->preempt_lazy_count) |
| + return false; |
| + return test_thread_flag(TIF_NEED_RESCHED_LAZY); |
| +#else |
| u64 pc = READ_ONCE(current_thread_info()->preempt_count); |
| return pc == preempt_offset; |
| +#endif |
| } |
| |
| #ifdef CONFIG_PREEMPTION |
| void preempt_schedule(void); |
| +#ifdef CONFIG_PREEMPT_RT |
| +void preempt_schedule_lock(void); |
| +#endif |
| #define __preempt_schedule() preempt_schedule() |
| void preempt_schedule_notrace(void); |
| #define __preempt_schedule_notrace() preempt_schedule_notrace() |
| diff --git a/arch/arm64/include/asm/spinlock_types.h b/arch/arm64/include/asm/spinlock_types.h |
| index 18782f0c47212..6672b05350b4e 100644 |
| --- a/arch/arm64/include/asm/spinlock_types.h |
| +++ b/arch/arm64/include/asm/spinlock_types.h |
| @@ -5,10 +5,6 @@ |
| #ifndef __ASM_SPINLOCK_TYPES_H |
| #define __ASM_SPINLOCK_TYPES_H |
| |
| -#if !defined(__LINUX_SPINLOCK_TYPES_H) && !defined(__ASM_SPINLOCK_H) |
| -# error "please don't include this file directly" |
| -#endif |
| - |
| #include <asm-generic/qspinlock_types.h> |
| #include <asm-generic/qrwlock_types.h> |
| |
| diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h |
| index 1fbab854a51b0..148b53dc2840e 100644 |
| --- a/arch/arm64/include/asm/thread_info.h |
| +++ b/arch/arm64/include/asm/thread_info.h |
| @@ -29,6 +29,7 @@ struct thread_info { |
| #ifdef CONFIG_ARM64_SW_TTBR0_PAN |
| u64 ttbr0; /* saved TTBR0_EL1 */ |
| #endif |
| + int preempt_lazy_count; /* 0 => preemptable, <0 => bug */ |
| union { |
| u64 preempt_count; /* 0 => preemptible, <0 => bug */ |
| struct { |
| @@ -68,6 +69,7 @@ void arch_release_task_struct(struct task_struct *tsk); |
| #define TIF_UPROBE 4 /* uprobe breakpoint or singlestep */ |
| #define TIF_FSCHECK 5 /* Check FS is USER_DS on return */ |
| #define TIF_MTE_ASYNC_FAULT 6 /* MTE Asynchronous Tag Check Fault */ |
| +#define TIF_NEED_RESCHED_LAZY 7 |
| #define TIF_SYSCALL_TRACE 8 /* syscall trace active */ |
| #define TIF_SYSCALL_AUDIT 9 /* syscall auditing */ |
| #define TIF_SYSCALL_TRACEPOINT 10 /* syscall tracepoint for ftrace */ |
| @@ -98,11 +100,14 @@ void arch_release_task_struct(struct task_struct *tsk); |
| #define _TIF_32BIT (1 << TIF_32BIT) |
| #define _TIF_SVE (1 << TIF_SVE) |
| #define _TIF_MTE_ASYNC_FAULT (1 << TIF_MTE_ASYNC_FAULT) |
| +#define _TIF_NEED_RESCHED_LAZY (1 << TIF_NEED_RESCHED_LAZY) |
| |
| #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ |
| _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ |
| - _TIF_UPROBE | _TIF_FSCHECK | _TIF_MTE_ASYNC_FAULT) |
| + _TIF_UPROBE | _TIF_FSCHECK | _TIF_MTE_ASYNC_FAULT | \ |
| + _TIF_NEED_RESCHED_LAZY) |
| |
| +#define _TIF_NEED_RESCHED_MASK (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY) |
| #define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ |
| _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \ |
| _TIF_SYSCALL_EMU) |
| diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c |
| index 7d32fc959b1a0..b2f29bd2ae871 100644 |
| --- a/arch/arm64/kernel/asm-offsets.c |
| +++ b/arch/arm64/kernel/asm-offsets.c |
| @@ -30,6 +30,7 @@ int main(void) |
| BLANK(); |
| DEFINE(TSK_TI_FLAGS, offsetof(struct task_struct, thread_info.flags)); |
| DEFINE(TSK_TI_PREEMPT, offsetof(struct task_struct, thread_info.preempt_count)); |
| + DEFINE(TSK_TI_PREEMPT_LAZY, offsetof(struct task_struct, thread_info.preempt_lazy_count)); |
| DEFINE(TSK_TI_ADDR_LIMIT, offsetof(struct task_struct, thread_info.addr_limit)); |
| #ifdef CONFIG_ARM64_SW_TTBR0_PAN |
| DEFINE(TSK_TI_TTBR0, offsetof(struct task_struct, thread_info.ttbr0)); |
| diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S |
| index fe83d6d67ec3d..cd2fc5556c8db 100644 |
| --- a/arch/arm64/kernel/entry.S |
| +++ b/arch/arm64/kernel/entry.S |
| @@ -515,9 +515,18 @@ alternative_if ARM64_HAS_IRQ_PRIO_MASKING |
| mrs x0, daif |
| orr x24, x24, x0 |
| alternative_else_nop_endif |
| - cbnz x24, 1f // preempt count != 0 || NMI return path |
| - bl arm64_preempt_schedule_irq // irq en/disable is done inside |
| + |
| + cbz x24, 1f // (need_resched + count) == 0 |
| + cbnz w24, 2f // count != 0 |
| + |
| + ldr w24, [tsk, #TSK_TI_PREEMPT_LAZY] // get preempt lazy count |
| + cbnz w24, 2f // preempt lazy count != 0 |
| + |
| + ldr x0, [tsk, #TSK_TI_FLAGS] // get flags |
| + tbz x0, #TIF_NEED_RESCHED_LAZY, 2f // needs rescheduling? |
| 1: |
| + bl arm64_preempt_schedule_irq // irq en/disable is done inside |
| +2: |
| #endif |
| |
| mov x0, sp |
| diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c |
| index a9bbfb800ec2b..aa631771e0dcd 100644 |
| --- a/arch/arm64/kernel/fpsimd.c |
| +++ b/arch/arm64/kernel/fpsimd.c |
| @@ -226,6 +226,16 @@ static void sve_free(struct task_struct *task) |
| __sve_free(task); |
| } |
| |
| +static void *sve_free_atomic(struct task_struct *task) |
| +{ |
| + void *sve_state = task->thread.sve_state; |
| + |
| + WARN_ON(test_tsk_thread_flag(task, TIF_SVE)); |
| + |
| + task->thread.sve_state = NULL; |
| + return sve_state; |
| +} |
| + |
| /* |
| * TIF_SVE controls whether a task can use SVE without trapping while |
| * in userspace, and also the way a task's FPSIMD/SVE state is stored |
| @@ -1022,6 +1032,7 @@ void fpsimd_thread_switch(struct task_struct *next) |
| void fpsimd_flush_thread(void) |
| { |
| int vl, supported_vl; |
| + void *mem = NULL; |
| |
| if (!system_supports_fpsimd()) |
| return; |
| @@ -1034,7 +1045,7 @@ void fpsimd_flush_thread(void) |
| |
| if (system_supports_sve()) { |
| clear_thread_flag(TIF_SVE); |
| - sve_free(current); |
| + mem = sve_free_atomic(current); |
| |
| /* |
| * Reset the task vector length as required. |
| @@ -1068,6 +1079,7 @@ void fpsimd_flush_thread(void) |
| } |
| |
| put_cpu_fpsimd_context(); |
| + kfree(mem); |
| } |
| |
| /* |
| diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c |
| index e62005317ce29..fe94a3e1f849b 100644 |
| --- a/arch/arm64/kernel/signal.c |
| +++ b/arch/arm64/kernel/signal.c |
| @@ -919,7 +919,7 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, |
| /* Check valid user FS if needed */ |
| addr_limit_user_check(); |
| |
| - if (thread_flags & _TIF_NEED_RESCHED) { |
| + if (thread_flags & _TIF_NEED_RESCHED_MASK) { |
| /* Unmask Debug and SError for the next task */ |
| local_daif_restore(DAIF_PROCCTX_NOIRQ); |
| |
| diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c |
| index 5bc978be80434..f262bb64e029a 100644 |
| --- a/arch/arm64/kvm/arm.c |
| +++ b/arch/arm64/kvm/arm.c |
| @@ -708,7 +708,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) |
| * involves poking the GIC, which must be done in a |
| * non-preemptible context. |
| */ |
| - preempt_disable(); |
| + migrate_disable(); |
| |
| kvm_pmu_flush_hwstate(vcpu); |
| |
| @@ -757,7 +757,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) |
| kvm_timer_sync_user(vcpu); |
| kvm_vgic_sync_hwstate(vcpu); |
| local_irq_enable(); |
| - preempt_enable(); |
| + migrate_enable(); |
| continue; |
| } |
| |
| @@ -829,7 +829,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) |
| /* Exit types that need handling before we can be preempted */ |
| handle_exit_early(vcpu, ret); |
| |
| - preempt_enable(); |
| + migrate_enable(); |
| |
| /* |
| * The ARMv8 architecture doesn't give the hypervisor |
| diff --git a/arch/csky/Kconfig b/arch/csky/Kconfig |
| index 7bf0a617e94c3..c9f2533cc53db 100644 |
| --- a/arch/csky/Kconfig |
| +++ b/arch/csky/Kconfig |
| @@ -286,6 +286,7 @@ config NR_CPUS |
| config HIGHMEM |
| bool "High Memory Support" |
| depends on !CPU_CK610 |
| + select KMAP_LOCAL |
| default y |
| |
| config FORCE_MAX_ZONEORDER |
| diff --git a/arch/csky/include/asm/fixmap.h b/arch/csky/include/asm/fixmap.h |
| index 81f9477d5330c..4b589cc209000 100644 |
| --- a/arch/csky/include/asm/fixmap.h |
| +++ b/arch/csky/include/asm/fixmap.h |
| @@ -8,7 +8,7 @@ |
| #include <asm/memory.h> |
| #ifdef CONFIG_HIGHMEM |
| #include <linux/threads.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| #endif |
| |
| enum fixed_addresses { |
| @@ -17,7 +17,7 @@ enum fixed_addresses { |
| #endif |
| #ifdef CONFIG_HIGHMEM |
| FIX_KMAP_BEGIN, |
| - FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_TYPE_NR * NR_CPUS) - 1, |
| + FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_MAX_IDX * NR_CPUS) - 1, |
| #endif |
| __end_of_fixed_addresses |
| }; |
| diff --git a/arch/csky/include/asm/highmem.h b/arch/csky/include/asm/highmem.h |
| index 14645e3d5cd52..1f4ed3f4c0d9b 100644 |
| --- a/arch/csky/include/asm/highmem.h |
| +++ b/arch/csky/include/asm/highmem.h |
| @@ -9,7 +9,7 @@ |
| #include <linux/init.h> |
| #include <linux/interrupt.h> |
| #include <linux/uaccess.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| #include <asm/cache.h> |
| |
| /* undef for production */ |
| @@ -32,10 +32,12 @@ extern pte_t *pkmap_page_table; |
| |
| #define ARCH_HAS_KMAP_FLUSH_TLB |
| extern void kmap_flush_tlb(unsigned long addr); |
| -extern void *kmap_atomic_pfn(unsigned long pfn); |
| |
| #define flush_cache_kmaps() do {} while (0) |
| |
| +#define arch_kmap_local_post_map(vaddr, pteval) kmap_flush_tlb(vaddr) |
| +#define arch_kmap_local_post_unmap(vaddr) kmap_flush_tlb(vaddr) |
| + |
| extern void kmap_init(void); |
| |
| #endif /* __KERNEL__ */ |
| diff --git a/arch/csky/mm/highmem.c b/arch/csky/mm/highmem.c |
| index 89c10800a002e..4161df3c6c152 100644 |
| --- a/arch/csky/mm/highmem.c |
| +++ b/arch/csky/mm/highmem.c |
| @@ -9,8 +9,6 @@ |
| #include <asm/tlbflush.h> |
| #include <asm/cacheflush.h> |
| |
| -static pte_t *kmap_pte; |
| - |
| unsigned long highstart_pfn, highend_pfn; |
| |
| void kmap_flush_tlb(unsigned long addr) |
| @@ -19,67 +17,7 @@ void kmap_flush_tlb(unsigned long addr) |
| } |
| EXPORT_SYMBOL(kmap_flush_tlb); |
| |
| -void *kmap_atomic_high_prot(struct page *page, pgprot_t prot) |
| -{ |
| - unsigned long vaddr; |
| - int idx, type; |
| - |
| - type = kmap_atomic_idx_push(); |
| - idx = type + KM_TYPE_NR*smp_processor_id(); |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - BUG_ON(!pte_none(*(kmap_pte - idx))); |
| -#endif |
| - set_pte(kmap_pte-idx, mk_pte(page, prot)); |
| - flush_tlb_one((unsigned long)vaddr); |
| - |
| - return (void *)vaddr; |
| -} |
| -EXPORT_SYMBOL(kmap_atomic_high_prot); |
| - |
| -void kunmap_atomic_high(void *kvaddr) |
| -{ |
| - unsigned long vaddr = (unsigned long) kvaddr & PAGE_MASK; |
| - int idx; |
| - |
| - if (vaddr < FIXADDR_START) |
| - return; |
| - |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - idx = KM_TYPE_NR*smp_processor_id() + kmap_atomic_idx(); |
| - |
| - BUG_ON(vaddr != __fix_to_virt(FIX_KMAP_BEGIN + idx)); |
| - |
| - pte_clear(&init_mm, vaddr, kmap_pte - idx); |
| - flush_tlb_one(vaddr); |
| -#else |
| - (void) idx; /* to kill a warning */ |
| -#endif |
| - kmap_atomic_idx_pop(); |
| -} |
| -EXPORT_SYMBOL(kunmap_atomic_high); |
| - |
| -/* |
| - * This is the same as kmap_atomic() but can map memory that doesn't |
| - * have a struct page associated with it. |
| - */ |
| -void *kmap_atomic_pfn(unsigned long pfn) |
| -{ |
| - unsigned long vaddr; |
| - int idx, type; |
| - |
| - pagefault_disable(); |
| - |
| - type = kmap_atomic_idx_push(); |
| - idx = type + KM_TYPE_NR*smp_processor_id(); |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| - set_pte(kmap_pte-idx, pfn_pte(pfn, PAGE_KERNEL)); |
| - flush_tlb_one(vaddr); |
| - |
| - return (void *) vaddr; |
| -} |
| - |
| -static void __init kmap_pages_init(void) |
| +void __init kmap_init(void) |
| { |
| unsigned long vaddr; |
| pgd_t *pgd; |
| @@ -96,14 +34,3 @@ static void __init kmap_pages_init(void) |
| pte = pte_offset_kernel(pmd, vaddr); |
| pkmap_page_table = pte; |
| } |
| - |
| -void __init kmap_init(void) |
| -{ |
| - unsigned long vaddr; |
| - |
| - kmap_pages_init(); |
| - |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN); |
| - |
| - kmap_pte = pte_offset_kernel((pmd_t *)pgd_offset_k(vaddr), vaddr); |
| -} |
| diff --git a/arch/hexagon/include/asm/spinlock_types.h b/arch/hexagon/include/asm/spinlock_types.h |
| index 19d233497ba52..de72fb23016dc 100644 |
| --- a/arch/hexagon/include/asm/spinlock_types.h |
| +++ b/arch/hexagon/include/asm/spinlock_types.h |
| @@ -8,10 +8,6 @@ |
| #ifndef _ASM_SPINLOCK_TYPES_H |
| #define _ASM_SPINLOCK_TYPES_H |
| |
| -#ifndef __LINUX_SPINLOCK_TYPES_H |
| -# error "please don't include this file directly" |
| -#endif |
| - |
| typedef struct { |
| volatile unsigned int lock; |
| } arch_spinlock_t; |
| diff --git a/arch/ia64/include/asm/kmap_types.h b/arch/ia64/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index 5c268cf7c2bd5..0000000000000 |
| --- a/arch/ia64/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,13 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0 */ |
| -#ifndef _ASM_IA64_KMAP_TYPES_H |
| -#define _ASM_IA64_KMAP_TYPES_H |
| - |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| -#define __WITH_KM_FENCE |
| -#endif |
| - |
| -#include <asm-generic/kmap_types.h> |
| - |
| -#undef __WITH_KM_FENCE |
| - |
| -#endif /* _ASM_IA64_KMAP_TYPES_H */ |
| diff --git a/arch/ia64/include/asm/spinlock_types.h b/arch/ia64/include/asm/spinlock_types.h |
| index 6e345fefcdcab..681408d6816fb 100644 |
| --- a/arch/ia64/include/asm/spinlock_types.h |
| +++ b/arch/ia64/include/asm/spinlock_types.h |
| @@ -2,10 +2,6 @@ |
| #ifndef _ASM_IA64_SPINLOCK_TYPES_H |
| #define _ASM_IA64_SPINLOCK_TYPES_H |
| |
| -#ifndef __LINUX_SPINLOCK_TYPES_H |
| -# error "please don't include this file directly" |
| -#endif |
| - |
| typedef struct { |
| volatile unsigned int lock; |
| } arch_spinlock_t; |
| diff --git a/arch/ia64/kernel/time.c b/arch/ia64/kernel/time.c |
| index 7abc5f37bfaf9..733e0e3324b80 100644 |
| --- a/arch/ia64/kernel/time.c |
| +++ b/arch/ia64/kernel/time.c |
| @@ -138,12 +138,8 @@ void vtime_account_kernel(struct task_struct *tsk) |
| struct thread_info *ti = task_thread_info(tsk); |
| __u64 stime = vtime_delta(tsk); |
| |
| - if ((tsk->flags & PF_VCPU) && !irq_count()) |
| + if (tsk->flags & PF_VCPU) |
| ti->gtime += stime; |
| - else if (hardirq_count()) |
| - ti->hardirq_time += stime; |
| - else if (in_serving_softirq()) |
| - ti->softirq_time += stime; |
| else |
| ti->stime += stime; |
| } |
| @@ -156,6 +152,20 @@ void vtime_account_idle(struct task_struct *tsk) |
| ti->idle_time += vtime_delta(tsk); |
| } |
| |
| +void vtime_account_softirq(struct task_struct *tsk) |
| +{ |
| + struct thread_info *ti = task_thread_info(tsk); |
| + |
| + ti->softirq_time += vtime_delta(tsk); |
| +} |
| + |
| +void vtime_account_hardirq(struct task_struct *tsk) |
| +{ |
| + struct thread_info *ti = task_thread_info(tsk); |
| + |
| + ti->hardirq_time += vtime_delta(tsk); |
| +} |
| + |
| #endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ |
| |
| static irqreturn_t |
| diff --git a/arch/microblaze/Kconfig b/arch/microblaze/Kconfig |
| index 33925ffed68fd..7f6ca0ab4f81f 100644 |
| --- a/arch/microblaze/Kconfig |
| +++ b/arch/microblaze/Kconfig |
| @@ -155,6 +155,7 @@ config XILINX_UNCACHED_SHADOW |
| config HIGHMEM |
| bool "High memory support" |
| depends on MMU |
| + select KMAP_LOCAL |
| help |
| The address space of Microblaze processors is only 4 Gigabytes large |
| and it has to accommodate user address space, kernel address |
| diff --git a/arch/microblaze/include/asm/fixmap.h b/arch/microblaze/include/asm/fixmap.h |
| index 0379ce5229e3a..e6e9288bff761 100644 |
| --- a/arch/microblaze/include/asm/fixmap.h |
| +++ b/arch/microblaze/include/asm/fixmap.h |
| @@ -20,7 +20,7 @@ |
| #include <asm/page.h> |
| #ifdef CONFIG_HIGHMEM |
| #include <linux/threads.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| #endif |
| |
| #define FIXADDR_TOP ((unsigned long)(-PAGE_SIZE)) |
| @@ -47,7 +47,7 @@ enum fixed_addresses { |
| FIX_HOLE, |
| #ifdef CONFIG_HIGHMEM |
| FIX_KMAP_BEGIN, /* reserved pte's for temporary kernel mappings */ |
| - FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_TYPE_NR * num_possible_cpus()) - 1, |
| + FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_MAX_IDX * num_possible_cpus()) - 1, |
| #endif |
| __end_of_fixed_addresses |
| }; |
| diff --git a/arch/microblaze/include/asm/highmem.h b/arch/microblaze/include/asm/highmem.h |
| index 284ca8fb54c1d..4418633fb1632 100644 |
| --- a/arch/microblaze/include/asm/highmem.h |
| +++ b/arch/microblaze/include/asm/highmem.h |
| @@ -25,7 +25,6 @@ |
| #include <linux/uaccess.h> |
| #include <asm/fixmap.h> |
| |
| -extern pte_t *kmap_pte; |
| extern pte_t *pkmap_page_table; |
| |
| /* |
| @@ -52,6 +51,11 @@ extern pte_t *pkmap_page_table; |
| |
| #define flush_cache_kmaps() { flush_icache(); flush_dcache(); } |
| |
| +#define arch_kmap_local_post_map(vaddr, pteval) \ |
| + local_flush_tlb_page(NULL, vaddr); |
| +#define arch_kmap_local_post_unmap(vaddr) \ |
| + local_flush_tlb_page(NULL, vaddr); |
| + |
| #endif /* __KERNEL__ */ |
| |
| #endif /* _ASM_HIGHMEM_H */ |
| diff --git a/arch/microblaze/mm/Makefile b/arch/microblaze/mm/Makefile |
| index 1b16875cea704..8ced711000471 100644 |
| --- a/arch/microblaze/mm/Makefile |
| +++ b/arch/microblaze/mm/Makefile |
| @@ -6,4 +6,3 @@ |
| obj-y := consistent.o init.o |
| |
| obj-$(CONFIG_MMU) += pgtable.o mmu_context.o fault.o |
| -obj-$(CONFIG_HIGHMEM) += highmem.o |
| diff --git a/arch/microblaze/mm/highmem.c b/arch/microblaze/mm/highmem.c |
| deleted file mode 100644 |
| index 92e0890416c91..0000000000000 |
| --- a/arch/microblaze/mm/highmem.c |
| +++ /dev/null |
| @@ -1,78 +0,0 @@ |
| -// SPDX-License-Identifier: GPL-2.0 |
| -/* |
| - * highmem.c: virtual kernel memory mappings for high memory |
| - * |
| - * PowerPC version, stolen from the i386 version. |
| - * |
| - * Used in CONFIG_HIGHMEM systems for memory pages which |
| - * are not addressable by direct kernel virtual addresses. |
| - * |
| - * Copyright (C) 1999 Gerhard Wichert, Siemens AG |
| - * Gerhard.Wichert@pdb.siemens.de |
| - * |
| - * |
| - * Redesigned the x86 32-bit VM architecture to deal with |
| - * up to 16 Terrabyte physical memory. With current x86 CPUs |
| - * we now support up to 64 Gigabytes physical RAM. |
| - * |
| - * Copyright (C) 1999 Ingo Molnar <mingo@redhat.com> |
| - * |
| - * Reworked for PowerPC by various contributors. Moved from |
| - * highmem.h by Benjamin Herrenschmidt (c) 2009 IBM Corp. |
| - */ |
| - |
| -#include <linux/export.h> |
| -#include <linux/highmem.h> |
| - |
| -/* |
| - * The use of kmap_atomic/kunmap_atomic is discouraged - kmap/kunmap |
| - * gives a more generic (and caching) interface. But kmap_atomic can |
| - * be used in IRQ contexts, so in some (very limited) cases we need |
| - * it. |
| - */ |
| -#include <asm/tlbflush.h> |
| - |
| -void *kmap_atomic_high_prot(struct page *page, pgprot_t prot) |
| -{ |
| - |
| - unsigned long vaddr; |
| - int idx, type; |
| - |
| - type = kmap_atomic_idx_push(); |
| - idx = type + KM_TYPE_NR*smp_processor_id(); |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - BUG_ON(!pte_none(*(kmap_pte-idx))); |
| -#endif |
| - set_pte_at(&init_mm, vaddr, kmap_pte-idx, mk_pte(page, prot)); |
| - local_flush_tlb_page(NULL, vaddr); |
| - |
| - return (void *) vaddr; |
| -} |
| -EXPORT_SYMBOL(kmap_atomic_high_prot); |
| - |
| -void kunmap_atomic_high(void *kvaddr) |
| -{ |
| - unsigned long vaddr = (unsigned long) kvaddr & PAGE_MASK; |
| - int type; |
| - unsigned int idx; |
| - |
| - if (vaddr < __fix_to_virt(FIX_KMAP_END)) |
| - return; |
| - |
| - type = kmap_atomic_idx(); |
| - |
| - idx = type + KM_TYPE_NR * smp_processor_id(); |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - BUG_ON(vaddr != __fix_to_virt(FIX_KMAP_BEGIN + idx)); |
| -#endif |
| - /* |
| - * force other mappings to Oops if they'll try to access |
| - * this pte without first remap it |
| - */ |
| - pte_clear(&init_mm, vaddr, kmap_pte-idx); |
| - local_flush_tlb_page(NULL, vaddr); |
| - |
| - kmap_atomic_idx_pop(); |
| -} |
| -EXPORT_SYMBOL(kunmap_atomic_high); |
| diff --git a/arch/microblaze/mm/init.c b/arch/microblaze/mm/init.c |
| index 45da639bd22ca..1f4b5b34e6004 100644 |
| --- a/arch/microblaze/mm/init.c |
| +++ b/arch/microblaze/mm/init.c |
| @@ -49,17 +49,11 @@ unsigned long lowmem_size; |
| EXPORT_SYMBOL(min_low_pfn); |
| EXPORT_SYMBOL(max_low_pfn); |
| |
| -#ifdef CONFIG_HIGHMEM |
| -pte_t *kmap_pte; |
| -EXPORT_SYMBOL(kmap_pte); |
| - |
| static void __init highmem_init(void) |
| { |
| pr_debug("%x\n", (u32)PKMAP_BASE); |
| map_page(PKMAP_BASE, 0, 0); /* XXX gross */ |
| pkmap_page_table = virt_to_kpte(PKMAP_BASE); |
| - |
| - kmap_pte = virt_to_kpte(__fix_to_virt(FIX_KMAP_BEGIN)); |
| } |
| |
| static void highmem_setup(void) |
| diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig |
| index 23d756fe0fd6c..4ea224e9add20 100644 |
| --- a/arch/mips/Kconfig |
| +++ b/arch/mips/Kconfig |
| @@ -2726,6 +2726,7 @@ config WAR_MIPS34K_MISSED_ITLB |
| config HIGHMEM |
| bool "High Memory Support" |
| depends on 32BIT && CPU_SUPPORTS_HIGHMEM && SYS_SUPPORTS_HIGHMEM && !CPU_MIPS32_3_5_EVA |
| + select KMAP_LOCAL |
| |
| config CPU_SUPPORTS_HIGHMEM |
| bool |
| diff --git a/arch/mips/include/asm/fixmap.h b/arch/mips/include/asm/fixmap.h |
| index 743535be7528c..beea14761cef0 100644 |
| --- a/arch/mips/include/asm/fixmap.h |
| +++ b/arch/mips/include/asm/fixmap.h |
| @@ -17,7 +17,7 @@ |
| #include <spaces.h> |
| #ifdef CONFIG_HIGHMEM |
| #include <linux/threads.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| #endif |
| |
| /* |
| @@ -52,7 +52,7 @@ enum fixed_addresses { |
| #ifdef CONFIG_HIGHMEM |
| /* reserved pte's for temporary kernel mappings */ |
| FIX_KMAP_BEGIN = FIX_CMAP_END + 1, |
| - FIX_KMAP_END = FIX_KMAP_BEGIN+(KM_TYPE_NR*NR_CPUS)-1, |
| + FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_MAX_IDX * NR_CPUS) - 1, |
| #endif |
| __end_of_fixed_addresses |
| }; |
| diff --git a/arch/mips/include/asm/highmem.h b/arch/mips/include/asm/highmem.h |
| index 9f021cf51aa71..1716181ea66dd 100644 |
| --- a/arch/mips/include/asm/highmem.h |
| +++ b/arch/mips/include/asm/highmem.h |
| @@ -24,7 +24,7 @@ |
| #include <linux/interrupt.h> |
| #include <linux/uaccess.h> |
| #include <asm/cpu-features.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| |
| /* declarations for highmem.c */ |
| extern unsigned long highstart_pfn, highend_pfn; |
| @@ -48,11 +48,11 @@ extern pte_t *pkmap_page_table; |
| |
| #define ARCH_HAS_KMAP_FLUSH_TLB |
| extern void kmap_flush_tlb(unsigned long addr); |
| -extern void *kmap_atomic_pfn(unsigned long pfn); |
| |
| #define flush_cache_kmaps() BUG_ON(cpu_has_dc_aliases) |
| |
| -extern void kmap_init(void); |
| +#define arch_kmap_local_post_map(vaddr, pteval) local_flush_tlb_one(vaddr) |
| +#define arch_kmap_local_post_unmap(vaddr) local_flush_tlb_one(vaddr) |
| |
| #endif /* __KERNEL__ */ |
| |
| diff --git a/arch/mips/include/asm/kmap_types.h b/arch/mips/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index 16665dc2431b6..0000000000000 |
| --- a/arch/mips/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,13 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0 */ |
| -#ifndef _ASM_KMAP_TYPES_H |
| -#define _ASM_KMAP_TYPES_H |
| - |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| -#define __WITH_KM_FENCE |
| -#endif |
| - |
| -#include <asm-generic/kmap_types.h> |
| - |
| -#undef __WITH_KM_FENCE |
| - |
| -#endif |
| diff --git a/arch/mips/kernel/crash_dump.c b/arch/mips/kernel/crash_dump.c |
| index 01b2bd95ba1f7..9aba83e1eeb4e 100644 |
| --- a/arch/mips/kernel/crash_dump.c |
| +++ b/arch/mips/kernel/crash_dump.c |
| @@ -5,8 +5,6 @@ |
| #include <linux/uaccess.h> |
| #include <linux/slab.h> |
| |
| -static void *kdump_buf_page; |
| - |
| /** |
| * copy_oldmem_page - copy one page from "oldmem" |
| * @pfn: page frame number to be copied |
| @@ -17,51 +15,25 @@ static void *kdump_buf_page; |
| * @userbuf: if set, @buf is in user address space, use copy_to_user(), |
| * otherwise @buf is in kernel address space, use memcpy(). |
| * |
| - * Copy a page from "oldmem". For this page, there is no pte mapped |
| + * Copy a page from "oldmem". For this page, there might be no pte mapped |
| * in the current kernel. |
| - * |
| - * Calling copy_to_user() in atomic context is not desirable. Hence first |
| - * copying the data to a pre-allocated kernel page and then copying to user |
| - * space in non-atomic context. |
| */ |
| -ssize_t copy_oldmem_page(unsigned long pfn, char *buf, |
| - size_t csize, unsigned long offset, int userbuf) |
| +ssize_t copy_oldmem_page(unsigned long pfn, char *buf, size_t csize, |
| + unsigned long offset, int userbuf) |
| { |
| void *vaddr; |
| |
| if (!csize) |
| return 0; |
| |
| - vaddr = kmap_atomic_pfn(pfn); |
| + vaddr = kmap_local_pfn(pfn); |
| |
| if (!userbuf) { |
| - memcpy(buf, (vaddr + offset), csize); |
| - kunmap_atomic(vaddr); |
| + memcpy(buf, vaddr + offset, csize); |
| } else { |
| - if (!kdump_buf_page) { |
| - pr_warn("Kdump: Kdump buffer page not allocated\n"); |
| - |
| - return -EFAULT; |
| - } |
| - copy_page(kdump_buf_page, vaddr); |
| - kunmap_atomic(vaddr); |
| - if (copy_to_user(buf, (kdump_buf_page + offset), csize)) |
| - return -EFAULT; |
| + if (copy_to_user(buf, vaddr + offset, csize)) |
| + csize = -EFAULT; |
| } |
| |
| return csize; |
| } |
| - |
| -static int __init kdump_buf_page_init(void) |
| -{ |
| - int ret = 0; |
| - |
| - kdump_buf_page = kmalloc(PAGE_SIZE, GFP_KERNEL); |
| - if (!kdump_buf_page) { |
| - pr_warn("Kdump: Failed to allocate kdump buffer page\n"); |
| - ret = -ENOMEM; |
| - } |
| - |
| - return ret; |
| -} |
| -arch_initcall(kdump_buf_page_init); |
| diff --git a/arch/mips/mm/highmem.c b/arch/mips/mm/highmem.c |
| index 5fec7f45d79a8..57e2f08f00d0c 100644 |
| --- a/arch/mips/mm/highmem.c |
| +++ b/arch/mips/mm/highmem.c |
| @@ -8,8 +8,6 @@ |
| #include <asm/fixmap.h> |
| #include <asm/tlbflush.h> |
| |
| -static pte_t *kmap_pte; |
| - |
| unsigned long highstart_pfn, highend_pfn; |
| |
| void kmap_flush_tlb(unsigned long addr) |
| @@ -17,78 +15,3 @@ void kmap_flush_tlb(unsigned long addr) |
| flush_tlb_one(addr); |
| } |
| EXPORT_SYMBOL(kmap_flush_tlb); |
| - |
| -void *kmap_atomic_high_prot(struct page *page, pgprot_t prot) |
| -{ |
| - unsigned long vaddr; |
| - int idx, type; |
| - |
| - type = kmap_atomic_idx_push(); |
| - idx = type + KM_TYPE_NR*smp_processor_id(); |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - BUG_ON(!pte_none(*(kmap_pte - idx))); |
| -#endif |
| - set_pte(kmap_pte-idx, mk_pte(page, prot)); |
| - local_flush_tlb_one((unsigned long)vaddr); |
| - |
| - return (void*) vaddr; |
| -} |
| -EXPORT_SYMBOL(kmap_atomic_high_prot); |
| - |
| -void kunmap_atomic_high(void *kvaddr) |
| -{ |
| - unsigned long vaddr = (unsigned long) kvaddr & PAGE_MASK; |
| - int type __maybe_unused; |
| - |
| - if (vaddr < FIXADDR_START) |
| - return; |
| - |
| - type = kmap_atomic_idx(); |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - { |
| - int idx = type + KM_TYPE_NR * smp_processor_id(); |
| - |
| - BUG_ON(vaddr != __fix_to_virt(FIX_KMAP_BEGIN + idx)); |
| - |
| - /* |
| - * force other mappings to Oops if they'll try to access |
| - * this pte without first remap it |
| - */ |
| - pte_clear(&init_mm, vaddr, kmap_pte-idx); |
| - local_flush_tlb_one(vaddr); |
| - } |
| -#endif |
| - kmap_atomic_idx_pop(); |
| -} |
| -EXPORT_SYMBOL(kunmap_atomic_high); |
| - |
| -/* |
| - * This is the same as kmap_atomic() but can map memory that doesn't |
| - * have a struct page associated with it. |
| - */ |
| -void *kmap_atomic_pfn(unsigned long pfn) |
| -{ |
| - unsigned long vaddr; |
| - int idx, type; |
| - |
| - preempt_disable(); |
| - pagefault_disable(); |
| - |
| - type = kmap_atomic_idx_push(); |
| - idx = type + KM_TYPE_NR*smp_processor_id(); |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| - set_pte(kmap_pte-idx, pfn_pte(pfn, PAGE_KERNEL)); |
| - flush_tlb_one(vaddr); |
| - |
| - return (void*) vaddr; |
| -} |
| - |
| -void __init kmap_init(void) |
| -{ |
| - unsigned long kmap_vstart; |
| - |
| - /* cache the first kmap pte */ |
| - kmap_vstart = __fix_to_virt(FIX_KMAP_BEGIN); |
| - kmap_pte = virt_to_kpte(kmap_vstart); |
| -} |
| diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c |
| index 07e84a7749387..bc80893e5c0f8 100644 |
| --- a/arch/mips/mm/init.c |
| +++ b/arch/mips/mm/init.c |
| @@ -36,7 +36,6 @@ |
| #include <asm/cachectl.h> |
| #include <asm/cpu.h> |
| #include <asm/dma.h> |
| -#include <asm/kmap_types.h> |
| #include <asm/maar.h> |
| #include <asm/mmu_context.h> |
| #include <asm/sections.h> |
| @@ -402,9 +401,6 @@ void __init paging_init(void) |
| |
| pagetable_init(); |
| |
| -#ifdef CONFIG_HIGHMEM |
| - kmap_init(); |
| -#endif |
| #ifdef CONFIG_ZONE_DMA |
| max_zone_pfns[ZONE_DMA] = MAX_DMA_PFN; |
| #endif |
| diff --git a/arch/nds32/Kconfig.cpu b/arch/nds32/Kconfig.cpu |
| index f88a12fdf0f35..c107599524855 100644 |
| --- a/arch/nds32/Kconfig.cpu |
| +++ b/arch/nds32/Kconfig.cpu |
| @@ -157,6 +157,7 @@ config HW_SUPPORT_UNALIGNMENT_ACCESS |
| config HIGHMEM |
| bool "High Memory Support" |
| depends on MMU && !CPU_CACHE_ALIASING |
| + select KMAP_LOCAL |
| help |
| The address space of Andes processors is only 4 Gigabytes large |
| and it has to accommodate user address space, kernel address |
| diff --git a/arch/nds32/include/asm/fixmap.h b/arch/nds32/include/asm/fixmap.h |
| index 5a4bf11e58003..2fa09a2de4281 100644 |
| --- a/arch/nds32/include/asm/fixmap.h |
| +++ b/arch/nds32/include/asm/fixmap.h |
| @@ -6,7 +6,7 @@ |
| |
| #ifdef CONFIG_HIGHMEM |
| #include <linux/threads.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| #endif |
| |
| enum fixed_addresses { |
| @@ -14,7 +14,7 @@ enum fixed_addresses { |
| FIX_KMAP_RESERVED, |
| FIX_KMAP_BEGIN, |
| #ifdef CONFIG_HIGHMEM |
| - FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_TYPE_NR * NR_CPUS), |
| + FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_MAX_IDX * NR_CPUS) - 1, |
| #endif |
| FIX_EARLYCON_MEM_BASE, |
| __end_of_fixed_addresses |
| diff --git a/arch/nds32/include/asm/highmem.h b/arch/nds32/include/asm/highmem.h |
| index fe986d0e6e3ff..16159a8716f22 100644 |
| --- a/arch/nds32/include/asm/highmem.h |
| +++ b/arch/nds32/include/asm/highmem.h |
| @@ -5,7 +5,6 @@ |
| #define _ASM_HIGHMEM_H |
| |
| #include <asm/proc-fns.h> |
| -#include <asm/kmap_types.h> |
| #include <asm/fixmap.h> |
| |
| /* |
| @@ -45,11 +44,22 @@ extern pte_t *pkmap_page_table; |
| extern void kmap_init(void); |
| |
| /* |
| - * The following functions are already defined by <linux/highmem.h> |
| - * when CONFIG_HIGHMEM is not set. |
| + * FIXME: The below looks broken vs. a kmap_atomic() in task context which |
| + * is interupted and another kmap_atomic() happens in interrupt context. |
| + * But what do I know about nds32. -- tglx |
| */ |
| -#ifdef CONFIG_HIGHMEM |
| -extern void *kmap_atomic_pfn(unsigned long pfn); |
| -#endif |
| +#define arch_kmap_local_post_map(vaddr, pteval) \ |
| + do { \ |
| + __nds32__tlbop_inv(vaddr); \ |
| + __nds32__mtsr_dsb(vaddr, NDS32_SR_TLB_VPN); \ |
| + __nds32__tlbop_rwr(pteval); \ |
| + __nds32__isb(); \ |
| + } while (0) |
| + |
| +#define arch_kmap_local_pre_unmap(vaddr) \ |
| + do { \ |
| + __nds32__tlbop_inv(vaddr); \ |
| + __nds32__isb(); \ |
| + } while (0) |
| |
| #endif |
| diff --git a/arch/nds32/mm/Makefile b/arch/nds32/mm/Makefile |
| index 897ecaf5cf54b..14fb2e8eb0368 100644 |
| --- a/arch/nds32/mm/Makefile |
| +++ b/arch/nds32/mm/Makefile |
| @@ -3,7 +3,6 @@ obj-y := extable.o tlb.o fault.o init.o mmap.o \ |
| mm-nds32.o cacheflush.o proc.o |
| |
| obj-$(CONFIG_ALIGNMENT_TRAP) += alignment.o |
| -obj-$(CONFIG_HIGHMEM) += highmem.o |
| |
| ifdef CONFIG_FUNCTION_TRACER |
| CFLAGS_REMOVE_proc.o = $(CC_FLAGS_FTRACE) |
| diff --git a/arch/nds32/mm/highmem.c b/arch/nds32/mm/highmem.c |
| deleted file mode 100644 |
| index 4284cd59e21ad..0000000000000 |
| --- a/arch/nds32/mm/highmem.c |
| +++ /dev/null |
| @@ -1,48 +0,0 @@ |
| -// SPDX-License-Identifier: GPL-2.0 |
| -// Copyright (C) 2005-2017 Andes Technology Corporation |
| - |
| -#include <linux/export.h> |
| -#include <linux/highmem.h> |
| -#include <linux/sched.h> |
| -#include <linux/smp.h> |
| -#include <linux/interrupt.h> |
| -#include <linux/memblock.h> |
| -#include <asm/fixmap.h> |
| -#include <asm/tlbflush.h> |
| - |
| -void *kmap_atomic_high_prot(struct page *page, pgprot_t prot) |
| -{ |
| - unsigned int idx; |
| - unsigned long vaddr, pte; |
| - int type; |
| - pte_t *ptep; |
| - |
| - type = kmap_atomic_idx_push(); |
| - |
| - idx = type + KM_TYPE_NR * smp_processor_id(); |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| - pte = (page_to_pfn(page) << PAGE_SHIFT) | prot; |
| - ptep = pte_offset_kernel(pmd_off_k(vaddr), vaddr); |
| - set_pte(ptep, pte); |
| - |
| - __nds32__tlbop_inv(vaddr); |
| - __nds32__mtsr_dsb(vaddr, NDS32_SR_TLB_VPN); |
| - __nds32__tlbop_rwr(pte); |
| - __nds32__isb(); |
| - return (void *)vaddr; |
| -} |
| -EXPORT_SYMBOL(kmap_atomic_high_prot); |
| - |
| -void kunmap_atomic_high(void *kvaddr) |
| -{ |
| - if (kvaddr >= (void *)FIXADDR_START) { |
| - unsigned long vaddr = (unsigned long)kvaddr; |
| - pte_t *ptep; |
| - kmap_atomic_idx_pop(); |
| - __nds32__tlbop_inv(vaddr); |
| - __nds32__isb(); |
| - ptep = pte_offset_kernel(pmd_off_k(vaddr), vaddr); |
| - set_pte(ptep, 0); |
| - } |
| -} |
| -EXPORT_SYMBOL(kunmap_atomic_high); |
| diff --git a/arch/openrisc/mm/init.c b/arch/openrisc/mm/init.c |
| index 5e88c351e6a45..f3fa02b8838af 100644 |
| --- a/arch/openrisc/mm/init.c |
| +++ b/arch/openrisc/mm/init.c |
| @@ -33,7 +33,6 @@ |
| #include <asm/io.h> |
| #include <asm/tlb.h> |
| #include <asm/mmu_context.h> |
| -#include <asm/kmap_types.h> |
| #include <asm/fixmap.h> |
| #include <asm/tlbflush.h> |
| #include <asm/sections.h> |
| diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c |
| index a978590d802d0..5aed97a18bac9 100644 |
| --- a/arch/openrisc/mm/ioremap.c |
| +++ b/arch/openrisc/mm/ioremap.c |
| @@ -15,7 +15,6 @@ |
| #include <linux/io.h> |
| #include <linux/pgtable.h> |
| #include <asm/pgalloc.h> |
| -#include <asm/kmap_types.h> |
| #include <asm/fixmap.h> |
| #include <asm/bug.h> |
| #include <linux/sched.h> |
| diff --git a/arch/parisc/include/asm/hardirq.h b/arch/parisc/include/asm/hardirq.h |
| index 7f7039516e53f..fad29aa6f45fa 100644 |
| --- a/arch/parisc/include/asm/hardirq.h |
| +++ b/arch/parisc/include/asm/hardirq.h |
| @@ -32,7 +32,6 @@ typedef struct { |
| DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); |
| |
| #define __ARCH_IRQ_STAT |
| -#define __IRQ_STAT(cpu, member) (irq_stat[cpu].member) |
| #define inc_irq_stat(member) this_cpu_inc(irq_stat.member) |
| #define __inc_irq_stat(member) __this_cpu_inc(irq_stat.member) |
| #define ack_bad_irq(irq) WARN(1, "unexpected IRQ trap at vector %02x\n", irq) |
| diff --git a/arch/parisc/include/asm/kmap_types.h b/arch/parisc/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index 3e70b5cd11231..0000000000000 |
| --- a/arch/parisc/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,13 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0 */ |
| -#ifndef _ASM_KMAP_TYPES_H |
| -#define _ASM_KMAP_TYPES_H |
| - |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| -#define __WITH_KM_FENCE |
| -#endif |
| - |
| -#include <asm-generic/kmap_types.h> |
| - |
| -#undef __WITH_KM_FENCE |
| - |
| -#endif |
| diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig |
| index 5afa0ebd78ca5..1d3e6f351bf2f 100644 |
| --- a/arch/powerpc/Kconfig |
| +++ b/arch/powerpc/Kconfig |
| @@ -146,6 +146,7 @@ config PPC |
| select ARCH_MIGHT_HAVE_PC_SERIO |
| select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX |
| select ARCH_SUPPORTS_ATOMIC_RMW |
| + select ARCH_SUPPORTS_RT if HAVE_POSIX_CPU_TIMERS_TASK_WORK |
| select ARCH_USE_BUILTIN_BSWAP |
| select ARCH_USE_CMPXCHG_LOCKREF if PPC64 |
| select ARCH_USE_QUEUED_RWLOCKS if PPC_QUEUED_SPINLOCKS |
| @@ -230,6 +231,7 @@ config PPC |
| select HAVE_HARDLOCKUP_DETECTOR_PERF if PERF_EVENTS && HAVE_PERF_EVENTS_NMI && !HAVE_HARDLOCKUP_DETECTOR_ARCH |
| select HAVE_PERF_REGS |
| select HAVE_PERF_USER_STACK_DUMP |
| + select HAVE_PREEMPT_LAZY |
| select MMU_GATHER_RCU_TABLE_FREE |
| select MMU_GATHER_PAGE_SIZE |
| select HAVE_REGS_AND_STACK_ACCESS_API |
| @@ -237,6 +239,7 @@ config PPC |
| select HAVE_SYSCALL_TRACEPOINTS |
| select HAVE_VIRT_CPU_ACCOUNTING |
| select HAVE_IRQ_TIME_ACCOUNTING |
| + select HAVE_POSIX_CPU_TIMERS_TASK_WORK if !KVM |
| select HAVE_RSEQ |
| select IOMMU_HELPER if PPC64 |
| select IRQ_DOMAIN |
| @@ -410,6 +413,7 @@ menu "Kernel options" |
| config HIGHMEM |
| bool "High memory support" |
| depends on PPC32 |
| + select KMAP_LOCAL |
| |
| source "kernel/Kconfig.hz" |
| |
| diff --git a/arch/powerpc/include/asm/cmpxchg.h b/arch/powerpc/include/asm/cmpxchg.h |
| index cf091c4c22e53..7371f7e23c357 100644 |
| --- a/arch/powerpc/include/asm/cmpxchg.h |
| +++ b/arch/powerpc/include/asm/cmpxchg.h |
| @@ -5,7 +5,7 @@ |
| #ifdef __KERNEL__ |
| #include <linux/compiler.h> |
| #include <asm/synch.h> |
| -#include <linux/bug.h> |
| +#include <linux/bits.h> |
| |
| #ifdef __BIG_ENDIAN |
| #define BITOFF_CAL(size, off) ((sizeof(u32) - size - off) * BITS_PER_BYTE) |
| diff --git a/arch/powerpc/include/asm/fixmap.h b/arch/powerpc/include/asm/fixmap.h |
| index 591b2f4deed53..947b5b9c44241 100644 |
| --- a/arch/powerpc/include/asm/fixmap.h |
| +++ b/arch/powerpc/include/asm/fixmap.h |
| @@ -20,7 +20,7 @@ |
| #include <asm/page.h> |
| #ifdef CONFIG_HIGHMEM |
| #include <linux/threads.h> |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| #endif |
| |
| #ifdef CONFIG_PPC64 |
| @@ -61,7 +61,7 @@ enum fixed_addresses { |
| FIX_EARLY_DEBUG_BASE = FIX_EARLY_DEBUG_TOP+(ALIGN(SZ_128K, PAGE_SIZE)/PAGE_SIZE)-1, |
| #ifdef CONFIG_HIGHMEM |
| FIX_KMAP_BEGIN, /* reserved pte's for temporary kernel mappings */ |
| - FIX_KMAP_END = FIX_KMAP_BEGIN+(KM_TYPE_NR*NR_CPUS)-1, |
| + FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_MAX_IDX * NR_CPUS) - 1, |
| #endif |
| #ifdef CONFIG_PPC_8xx |
| /* For IMMR we need an aligned 512K area */ |
| diff --git a/arch/powerpc/include/asm/highmem.h b/arch/powerpc/include/asm/highmem.h |
| index 104026f7d6bc2..80a5ae771c653 100644 |
| --- a/arch/powerpc/include/asm/highmem.h |
| +++ b/arch/powerpc/include/asm/highmem.h |
| @@ -24,12 +24,10 @@ |
| #ifdef __KERNEL__ |
| |
| #include <linux/interrupt.h> |
| -#include <asm/kmap_types.h> |
| #include <asm/cacheflush.h> |
| #include <asm/page.h> |
| #include <asm/fixmap.h> |
| |
| -extern pte_t *kmap_pte; |
| extern pte_t *pkmap_page_table; |
| |
| /* |
| @@ -60,6 +58,11 @@ extern pte_t *pkmap_page_table; |
| |
| #define flush_cache_kmaps() flush_cache_all() |
| |
| +#define arch_kmap_local_post_map(vaddr, pteval) \ |
| + local_flush_tlb_page(NULL, vaddr) |
| +#define arch_kmap_local_post_unmap(vaddr) \ |
| + local_flush_tlb_page(NULL, vaddr) |
| + |
| #endif /* __KERNEL__ */ |
| |
| #endif /* _ASM_HIGHMEM_H */ |
| diff --git a/arch/powerpc/include/asm/kmap_types.h b/arch/powerpc/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index c8fa182d48c8e..0000000000000 |
| --- a/arch/powerpc/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,13 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0-or-later */ |
| -#ifndef _ASM_POWERPC_KMAP_TYPES_H |
| -#define _ASM_POWERPC_KMAP_TYPES_H |
| - |
| -#ifdef __KERNEL__ |
| - |
| -/* |
| - */ |
| - |
| -#define KM_TYPE_NR 16 |
| - |
| -#endif /* __KERNEL__ */ |
| -#endif /* _ASM_POWERPC_KMAP_TYPES_H */ |
| diff --git a/arch/powerpc/include/asm/simple_spinlock_types.h b/arch/powerpc/include/asm/simple_spinlock_types.h |
| index 0f3cdd8faa959..d45561e9e6bab 100644 |
| --- a/arch/powerpc/include/asm/simple_spinlock_types.h |
| +++ b/arch/powerpc/include/asm/simple_spinlock_types.h |
| @@ -2,7 +2,7 @@ |
| #ifndef _ASM_POWERPC_SIMPLE_SPINLOCK_TYPES_H |
| #define _ASM_POWERPC_SIMPLE_SPINLOCK_TYPES_H |
| |
| -#ifndef __LINUX_SPINLOCK_TYPES_H |
| +#if !defined(__LINUX_SPINLOCK_TYPES_H) && !defined(__LINUX_RT_MUTEX_H) |
| # error "please don't include this file directly" |
| #endif |
| |
| diff --git a/arch/powerpc/include/asm/spinlock_types.h b/arch/powerpc/include/asm/spinlock_types.h |
| index c5d742f18021d..cc6922a011ba1 100644 |
| --- a/arch/powerpc/include/asm/spinlock_types.h |
| +++ b/arch/powerpc/include/asm/spinlock_types.h |
| @@ -2,10 +2,6 @@ |
| #ifndef _ASM_POWERPC_SPINLOCK_TYPES_H |
| #define _ASM_POWERPC_SPINLOCK_TYPES_H |
| |
| -#ifndef __LINUX_SPINLOCK_TYPES_H |
| -# error "please don't include this file directly" |
| -#endif |
| - |
| #ifdef CONFIG_PPC_QUEUED_SPINLOCKS |
| #include <asm-generic/qspinlock_types.h> |
| #include <asm-generic/qrwlock_types.h> |
| diff --git a/arch/powerpc/include/asm/stackprotector.h b/arch/powerpc/include/asm/stackprotector.h |
| index 1c8460e235838..b1653c160bab9 100644 |
| --- a/arch/powerpc/include/asm/stackprotector.h |
| +++ b/arch/powerpc/include/asm/stackprotector.h |
| @@ -24,7 +24,11 @@ static __always_inline void boot_init_stack_canary(void) |
| unsigned long canary; |
| |
| /* Try to get a semi random initial value. */ |
| +#ifdef CONFIG_PREEMPT_RT |
| + canary = (unsigned long)&canary; |
| +#else |
| canary = get_random_canary(); |
| +#endif |
| canary ^= mftb(); |
| canary ^= LINUX_VERSION_CODE; |
| canary &= CANARY_MASK; |
| diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h |
| index 46a210b03d2b8..0e316b44b2d73 100644 |
| --- a/arch/powerpc/include/asm/thread_info.h |
| +++ b/arch/powerpc/include/asm/thread_info.h |
| @@ -48,6 +48,8 @@ |
| struct thread_info { |
| int preempt_count; /* 0 => preemptable, |
| <0 => BUG */ |
| + int preempt_lazy_count; /* 0 => preemptable, |
| + <0 => BUG */ |
| unsigned long local_flags; /* private flags for thread */ |
| #ifdef CONFIG_LIVEPATCH |
| unsigned long *livepatch_sp; |
| @@ -97,11 +99,12 @@ void arch_setup_new_exec(void); |
| #define TIF_SINGLESTEP 8 /* singlestepping active */ |
| #define TIF_NOHZ 9 /* in adaptive nohz mode */ |
| #define TIF_SECCOMP 10 /* secure computing */ |
| -#define TIF_RESTOREALL 11 /* Restore all regs (implies NOERROR) */ |
| -#define TIF_NOERROR 12 /* Force successful syscall return */ |
| + |
| +#define TIF_NEED_RESCHED_LAZY 11 /* lazy rescheduling necessary */ |
| +#define TIF_SYSCALL_TRACEPOINT 12 /* syscall tracepoint instrumentation */ |
| + |
| #define TIF_NOTIFY_RESUME 13 /* callback before returning to user */ |
| #define TIF_UPROBE 14 /* breakpointed or single-stepping */ |
| -#define TIF_SYSCALL_TRACEPOINT 15 /* syscall tracepoint instrumentation */ |
| #define TIF_EMULATE_STACK_STORE 16 /* Is an instruction emulation |
| for stack store? */ |
| #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ |
| @@ -110,6 +113,9 @@ void arch_setup_new_exec(void); |
| #endif |
| #define TIF_POLLING_NRFLAG 19 /* true if poll_idle() is polling TIF_NEED_RESCHED */ |
| #define TIF_32BIT 20 /* 32 bit binary */ |
| +#define TIF_RESTOREALL 21 /* Restore all regs (implies NOERROR) */ |
| +#define TIF_NOERROR 22 /* Force successful syscall return */ |
| + |
| |
| /* as above, but as bit values */ |
| #define _TIF_SYSCALL_TRACE (1<<TIF_SYSCALL_TRACE) |
| @@ -129,6 +135,7 @@ void arch_setup_new_exec(void); |
| #define _TIF_SYSCALL_TRACEPOINT (1<<TIF_SYSCALL_TRACEPOINT) |
| #define _TIF_EMULATE_STACK_STORE (1<<TIF_EMULATE_STACK_STORE) |
| #define _TIF_NOHZ (1<<TIF_NOHZ) |
| +#define _TIF_NEED_RESCHED_LAZY (1<<TIF_NEED_RESCHED_LAZY) |
| #define _TIF_SYSCALL_EMU (1<<TIF_SYSCALL_EMU) |
| #define _TIF_SYSCALL_DOTRACE (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ |
| _TIF_SECCOMP | _TIF_SYSCALL_TRACEPOINT | \ |
| @@ -136,8 +143,10 @@ void arch_setup_new_exec(void); |
| |
| #define _TIF_USER_WORK_MASK (_TIF_SIGPENDING | _TIF_NEED_RESCHED | \ |
| _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ |
| - _TIF_RESTORE_TM | _TIF_PATCH_PENDING) |
| + _TIF_RESTORE_TM | _TIF_PATCH_PENDING | \ |
| + _TIF_NEED_RESCHED_LAZY) |
| #define _TIF_PERSYSCALL_MASK (_TIF_RESTOREALL|_TIF_NOERROR) |
| +#define _TIF_NEED_RESCHED_MASK (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY) |
| |
| /* Bits in local_flags */ |
| /* Don't move TLF_NAPPING without adjusting the code in entry_32.S */ |
| diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c |
| index 5c125255571cd..597379121407a 100644 |
| --- a/arch/powerpc/kernel/asm-offsets.c |
| +++ b/arch/powerpc/kernel/asm-offsets.c |
| @@ -189,6 +189,7 @@ int main(void) |
| OFFSET(TI_FLAGS, thread_info, flags); |
| OFFSET(TI_LOCAL_FLAGS, thread_info, local_flags); |
| OFFSET(TI_PREEMPT, thread_info, preempt_count); |
| + OFFSET(TI_PREEMPT_LAZY, thread_info, preempt_lazy_count); |
| |
| #ifdef CONFIG_PPC64 |
| OFFSET(DCACHEL1BLOCKSIZE, ppc64_caches, l1d.block_size); |
| diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S |
| index 459f5d00b9904..fc9517a976407 100644 |
| --- a/arch/powerpc/kernel/entry_32.S |
| +++ b/arch/powerpc/kernel/entry_32.S |
| @@ -414,7 +414,9 @@ ret_from_syscall: |
| mtmsr r10 |
| lwz r9,TI_FLAGS(r2) |
| li r8,-MAX_ERRNO |
| - andi. r0,r9,(_TIF_SYSCALL_DOTRACE|_TIF_SINGLESTEP|_TIF_USER_WORK_MASK|_TIF_PERSYSCALL_MASK) |
| + lis r0,(_TIF_SYSCALL_DOTRACE|_TIF_SINGLESTEP|_TIF_USER_WORK_MASK|_TIF_PERSYSCALL_MASK)@h |
| + ori r0,r0, (_TIF_SYSCALL_DOTRACE|_TIF_SINGLESTEP|_TIF_USER_WORK_MASK|_TIF_PERSYSCALL_MASK)@l |
| + and. r0,r9,r0 |
| bne- syscall_exit_work |
| cmplw 0,r3,r8 |
| blt+ syscall_exit_cont |
| @@ -530,13 +532,13 @@ syscall_dotrace: |
| b syscall_dotrace_cont |
| |
| syscall_exit_work: |
| - andi. r0,r9,_TIF_RESTOREALL |
| + andis. r0,r9,_TIF_RESTOREALL@h |
| beq+ 0f |
| REST_NVGPRS(r1) |
| b 2f |
| 0: cmplw 0,r3,r8 |
| blt+ 1f |
| - andi. r0,r9,_TIF_NOERROR |
| + andis. r0,r9,_TIF_NOERROR@h |
| bne- 1f |
| lwz r11,_CCR(r1) /* Load CR */ |
| neg r3,r3 |
| @@ -545,12 +547,12 @@ syscall_exit_work: |
| |
| 1: stw r6,RESULT(r1) /* Save result */ |
| stw r3,GPR3(r1) /* Update return value */ |
| -2: andi. r0,r9,(_TIF_PERSYSCALL_MASK) |
| +2: andis. r0,r9,(_TIF_PERSYSCALL_MASK)@h |
| beq 4f |
| |
| /* Clear per-syscall TIF flags if any are set. */ |
| |
| - li r11,_TIF_PERSYSCALL_MASK |
| + lis r11,(_TIF_PERSYSCALL_MASK)@h |
| addi r12,r2,TI_FLAGS |
| 3: lwarx r8,0,r12 |
| andc r8,r8,r11 |
| @@ -927,7 +929,14 @@ resume_kernel: |
| cmpwi 0,r0,0 /* if non-zero, just restore regs and return */ |
| bne restore_kuap |
| andi. r8,r8,_TIF_NEED_RESCHED |
| + bne+ 1f |
| + lwz r0,TI_PREEMPT_LAZY(r2) |
| + cmpwi 0,r0,0 /* if non-zero, just restore regs and return */ |
| + bne restore_kuap |
| + lwz r0,TI_FLAGS(r2) |
| + andi. r0,r0,_TIF_NEED_RESCHED_LAZY |
| beq+ restore_kuap |
| +1: |
| lwz r3,_MSR(r1) |
| andi. r0,r3,MSR_EE /* interrupts off? */ |
| beq restore_kuap /* don't schedule if so */ |
| @@ -1248,7 +1257,7 @@ global_dbcr0: |
| #endif /* !(CONFIG_4xx || CONFIG_BOOKE) */ |
| |
| do_work: /* r10 contains MSR_KERNEL here */ |
| - andi. r0,r9,_TIF_NEED_RESCHED |
| + andi. r0,r9,_TIF_NEED_RESCHED_MASK |
| beq do_user_signal |
| |
| do_resched: /* r10 contains MSR_KERNEL here */ |
| @@ -1267,7 +1276,7 @@ recheck: |
| LOAD_REG_IMMEDIATE(r10,MSR_KERNEL) |
| mtmsr r10 /* disable interrupts */ |
| lwz r9,TI_FLAGS(r2) |
| - andi. r0,r9,_TIF_NEED_RESCHED |
| + andi. r0,r9,_TIF_NEED_RESCHED_MASK |
| bne- do_resched |
| andi. r0,r9,_TIF_USER_WORK_MASK |
| beq restore_user |
| diff --git a/arch/powerpc/kernel/exceptions-64e.S b/arch/powerpc/kernel/exceptions-64e.S |
| index f579ce46eef2d..715ff292a8f8a 100644 |
| --- a/arch/powerpc/kernel/exceptions-64e.S |
| +++ b/arch/powerpc/kernel/exceptions-64e.S |
| @@ -1080,7 +1080,7 @@ _GLOBAL(ret_from_except_lite) |
| li r10, -1 |
| mtspr SPRN_DBSR,r10 |
| b restore |
| -1: andi. r0,r4,_TIF_NEED_RESCHED |
| +1: andi. r0,r4,_TIF_NEED_RESCHED_MASK |
| beq 2f |
| bl restore_interrupts |
| SCHEDULE_USER |
| @@ -1132,12 +1132,20 @@ resume_kernel: |
| bne- 0b |
| 1: |
| |
| -#ifdef CONFIG_PREEMPT |
| +#ifdef CONFIG_PREEMPTION |
| /* Check if we need to preempt */ |
| + lwz r8,TI_PREEMPT(r9) |
| + cmpwi 0,r8,0 /* if non-zero, just restore regs and return */ |
| + bne restore |
| andi. r0,r4,_TIF_NEED_RESCHED |
| + bne+ check_count |
| + |
| + andi. r0,r4,_TIF_NEED_RESCHED_LAZY |
| beq+ restore |
| + lwz r8,TI_PREEMPT_LAZY(r9) |
| + |
| /* Check that preempt_count() == 0 and interrupts are enabled */ |
| - lwz r8,TI_PREEMPT(r9) |
| +check_count: |
| cmpwi cr0,r8,0 |
| bne restore |
| ld r0,SOFTE(r1) |
| @@ -1158,7 +1166,7 @@ resume_kernel: |
| * interrupted after loading SRR0/1. |
| */ |
| wrteei 0 |
| -#endif /* CONFIG_PREEMPT */ |
| +#endif /* CONFIG_PREEMPTION */ |
| |
| restore: |
| /* |
| diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c |
| index e8a548447dd68..5ad4f27cba104 100644 |
| --- a/arch/powerpc/kernel/irq.c |
| +++ b/arch/powerpc/kernel/irq.c |
| @@ -753,10 +753,12 @@ void *mcheckirq_ctx[NR_CPUS] __read_mostly; |
| void *softirq_ctx[NR_CPUS] __read_mostly; |
| void *hardirq_ctx[NR_CPUS] __read_mostly; |
| |
| +#ifndef CONFIG_PREEMPT_RT |
| void do_softirq_own_stack(void) |
| { |
| call_do_softirq(softirq_ctx[smp_processor_id()]); |
| } |
| +#endif |
| |
| irq_hw_number_t virq_to_hw(unsigned int virq) |
| { |
| diff --git a/arch/powerpc/kernel/misc_32.S b/arch/powerpc/kernel/misc_32.S |
| index 717e658b90fd8..08ee95ad6593a 100644 |
| --- a/arch/powerpc/kernel/misc_32.S |
| +++ b/arch/powerpc/kernel/misc_32.S |
| @@ -31,6 +31,7 @@ |
| * We store the saved ksp_limit in the unused part |
| * of the STACK_FRAME_OVERHEAD |
| */ |
| +#ifndef CONFIG_PREEMPT_RT |
| _GLOBAL(call_do_softirq) |
| mflr r0 |
| stw r0,4(r1) |
| @@ -46,6 +47,7 @@ _GLOBAL(call_do_softirq) |
| stw r10,THREAD+KSP_LIMIT(r2) |
| mtlr r0 |
| blr |
| +#endif |
| |
| /* |
| * void call_do_irq(struct pt_regs *regs, void *sp); |
| diff --git a/arch/powerpc/kernel/misc_64.S b/arch/powerpc/kernel/misc_64.S |
| index 070465825c210..a6b33f7b32646 100644 |
| --- a/arch/powerpc/kernel/misc_64.S |
| +++ b/arch/powerpc/kernel/misc_64.S |
| @@ -27,6 +27,7 @@ |
| |
| .text |
| |
| +#ifndef CONFIG_PREEMPT_RT |
| _GLOBAL(call_do_softirq) |
| mflr r0 |
| std r0,16(r1) |
| @@ -37,6 +38,7 @@ _GLOBAL(call_do_softirq) |
| ld r0,16(r1) |
| mtlr r0 |
| blr |
| +#endif |
| |
| _GLOBAL(call_do_irq) |
| mflr r0 |
| diff --git a/arch/powerpc/kernel/nvram_64.c b/arch/powerpc/kernel/nvram_64.c |
| index 532f226377831..1ef55f4b389a2 100644 |
| --- a/arch/powerpc/kernel/nvram_64.c |
| +++ b/arch/powerpc/kernel/nvram_64.c |
| @@ -73,7 +73,8 @@ static const char *nvram_os_partitions[] = { |
| }; |
| |
| static void oops_to_nvram(struct kmsg_dumper *dumper, |
| - enum kmsg_dump_reason reason); |
| + enum kmsg_dump_reason reason, |
| + struct kmsg_dumper_iter *iter); |
| |
| static struct kmsg_dumper nvram_kmsg_dumper = { |
| .dump = oops_to_nvram |
| @@ -643,7 +644,8 @@ void __init nvram_init_oops_partition(int rtas_partition_exists) |
| * partition. If that's too much, go back and capture uncompressed text. |
| */ |
| static void oops_to_nvram(struct kmsg_dumper *dumper, |
| - enum kmsg_dump_reason reason) |
| + enum kmsg_dump_reason reason, |
| + struct kmsg_dumper_iter *iter) |
| { |
| struct oops_log_info *oops_hdr = (struct oops_log_info *)oops_buf; |
| static unsigned int oops_count = 0; |
| @@ -681,13 +683,13 @@ static void oops_to_nvram(struct kmsg_dumper *dumper, |
| return; |
| |
| if (big_oops_buf) { |
| - kmsg_dump_get_buffer(dumper, false, |
| + kmsg_dump_get_buffer(iter, false, |
| big_oops_buf, big_oops_buf_sz, &text_len); |
| rc = zip_oops(text_len); |
| } |
| if (rc != 0) { |
| - kmsg_dump_rewind(dumper); |
| - kmsg_dump_get_buffer(dumper, false, |
| + kmsg_dump_rewind(iter); |
| + kmsg_dump_get_buffer(iter, false, |
| oops_data, oops_data_sz, &text_len); |
| err_type = ERR_TYPE_KERNEL_PANIC; |
| oops_hdr->version = cpu_to_be16(OOPS_HDR_VERSION); |
| diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/syscall_64.c |
| index 310bcd768cd5a..ae3212dcf5627 100644 |
| --- a/arch/powerpc/kernel/syscall_64.c |
| +++ b/arch/powerpc/kernel/syscall_64.c |
| @@ -193,7 +193,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3, |
| ti_flags = READ_ONCE(*ti_flagsp); |
| while (unlikely(ti_flags & (_TIF_USER_WORK_MASK & ~_TIF_RESTORE_TM))) { |
| local_irq_enable(); |
| - if (ti_flags & _TIF_NEED_RESCHED) { |
| + if (ti_flags & _TIF_NEED_RESCHED_MASK) { |
| schedule(); |
| } else { |
| /* |
| @@ -277,7 +277,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned |
| ti_flags = READ_ONCE(*ti_flagsp); |
| while (unlikely(ti_flags & (_TIF_USER_WORK_MASK & ~_TIF_RESTORE_TM))) { |
| local_irq_enable(); /* returning to user: may enable */ |
| - if (ti_flags & _TIF_NEED_RESCHED) { |
| + if (ti_flags & _TIF_NEED_RESCHED_MASK) { |
| schedule(); |
| } else { |
| if (ti_flags & _TIF_SIGPENDING) |
| @@ -361,11 +361,15 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign |
| /* Returning to a kernel context with local irqs enabled. */ |
| WARN_ON_ONCE(!(regs->msr & MSR_EE)); |
| again: |
| - if (IS_ENABLED(CONFIG_PREEMPT)) { |
| + if (IS_ENABLED(CONFIG_PREEMPTION)) { |
| /* Return to preemptible kernel context */ |
| if (unlikely(*ti_flagsp & _TIF_NEED_RESCHED)) { |
| if (preempt_count() == 0) |
| preempt_schedule_irq(); |
| + } else if (unlikely(*ti_flagsp & _TIF_NEED_RESCHED_LAZY)) { |
| + if ((preempt_count() == 0) && |
| + (current_thread_info()->preempt_lazy_count == 0)) |
| + preempt_schedule_irq(); |
| } |
| } |
| |
| diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c |
| index 1d20f0f77a920..7e0a497a36ee3 100644 |
| --- a/arch/powerpc/kernel/time.c |
| +++ b/arch/powerpc/kernel/time.c |
| @@ -312,12 +312,11 @@ static unsigned long vtime_delta_scaled(struct cpu_accounting_data *acct, |
| return stime_scaled; |
| } |
| |
| -static unsigned long vtime_delta(struct task_struct *tsk, |
| +static unsigned long vtime_delta(struct cpu_accounting_data *acct, |
| unsigned long *stime_scaled, |
| unsigned long *steal_time) |
| { |
| unsigned long now, stime; |
| - struct cpu_accounting_data *acct = get_accounting(tsk); |
| |
| WARN_ON_ONCE(!irqs_disabled()); |
| |
| @@ -332,29 +331,30 @@ static unsigned long vtime_delta(struct task_struct *tsk, |
| return stime; |
| } |
| |
| +static void vtime_delta_kernel(struct cpu_accounting_data *acct, |
| + unsigned long *stime, unsigned long *stime_scaled) |
| +{ |
| + unsigned long steal_time; |
| + |
| + *stime = vtime_delta(acct, stime_scaled, &steal_time); |
| + *stime -= min(*stime, steal_time); |
| + acct->steal_time += steal_time; |
| +} |
| + |
| void vtime_account_kernel(struct task_struct *tsk) |
| { |
| - unsigned long stime, stime_scaled, steal_time; |
| struct cpu_accounting_data *acct = get_accounting(tsk); |
| + unsigned long stime, stime_scaled; |
| |
| - stime = vtime_delta(tsk, &stime_scaled, &steal_time); |
| - |
| - stime -= min(stime, steal_time); |
| - acct->steal_time += steal_time; |
| + vtime_delta_kernel(acct, &stime, &stime_scaled); |
| |
| - if ((tsk->flags & PF_VCPU) && !irq_count()) { |
| + if (tsk->flags & PF_VCPU) { |
| acct->gtime += stime; |
| #ifdef CONFIG_ARCH_HAS_SCALED_CPUTIME |
| acct->utime_scaled += stime_scaled; |
| #endif |
| } else { |
| - if (hardirq_count()) |
| - acct->hardirq_time += stime; |
| - else if (in_serving_softirq()) |
| - acct->softirq_time += stime; |
| - else |
| - acct->stime += stime; |
| - |
| + acct->stime += stime; |
| #ifdef CONFIG_ARCH_HAS_SCALED_CPUTIME |
| acct->stime_scaled += stime_scaled; |
| #endif |
| @@ -367,10 +367,34 @@ void vtime_account_idle(struct task_struct *tsk) |
| unsigned long stime, stime_scaled, steal_time; |
| struct cpu_accounting_data *acct = get_accounting(tsk); |
| |
| - stime = vtime_delta(tsk, &stime_scaled, &steal_time); |
| + stime = vtime_delta(acct, &stime_scaled, &steal_time); |
| acct->idle_time += stime + steal_time; |
| } |
| |
| +static void vtime_account_irq_field(struct cpu_accounting_data *acct, |
| + unsigned long *field) |
| +{ |
| + unsigned long stime, stime_scaled; |
| + |
| + vtime_delta_kernel(acct, &stime, &stime_scaled); |
| + *field += stime; |
| +#ifdef CONFIG_ARCH_HAS_SCALED_CPUTIME |
| + acct->stime_scaled += stime_scaled; |
| +#endif |
| +} |
| + |
| +void vtime_account_softirq(struct task_struct *tsk) |
| +{ |
| + struct cpu_accounting_data *acct = get_accounting(tsk); |
| + vtime_account_irq_field(acct, &acct->softirq_time); |
| +} |
| + |
| +void vtime_account_hardirq(struct task_struct *tsk) |
| +{ |
| + struct cpu_accounting_data *acct = get_accounting(tsk); |
| + vtime_account_irq_field(acct, &acct->hardirq_time); |
| +} |
| + |
| static void vtime_flush_scaled(struct task_struct *tsk, |
| struct cpu_accounting_data *acct) |
| { |
| diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c |
| index 77dffea3d5373..34861e39d03a4 100644 |
| --- a/arch/powerpc/kernel/traps.c |
| +++ b/arch/powerpc/kernel/traps.c |
| @@ -170,7 +170,6 @@ extern void panic_flush_kmsg_start(void) |
| |
| extern void panic_flush_kmsg_end(void) |
| { |
| - printk_safe_flush_on_panic(); |
| kmsg_dump(KMSG_DUMP_PANIC); |
| bust_spinlocks(0); |
| debug_locks_off(); |
| @@ -260,12 +259,17 @@ static char *get_mmu_str(void) |
| |
| static int __die(const char *str, struct pt_regs *regs, long err) |
| { |
| + const char *pr = ""; |
| + |
| printk("Oops: %s, sig: %ld [#%d]\n", str, err, ++die_counter); |
| |
| + if (IS_ENABLED(CONFIG_PREEMPTION)) |
| + pr = IS_ENABLED(CONFIG_PREEMPT_RT) ? " PREEMPT_RT" : " PREEMPT"; |
| + |
| printk("%s PAGE_SIZE=%luK%s%s%s%s%s%s %s\n", |
| IS_ENABLED(CONFIG_CPU_LITTLE_ENDIAN) ? "LE" : "BE", |
| PAGE_SIZE / 1024, get_mmu_str(), |
| - IS_ENABLED(CONFIG_PREEMPT) ? " PREEMPT" : "", |
| + pr, |
| IS_ENABLED(CONFIG_SMP) ? " SMP" : "", |
| IS_ENABLED(CONFIG_SMP) ? (" NR_CPUS=" __stringify(NR_CPUS)) : "", |
| debug_pagealloc_enabled() ? " DEBUG_PAGEALLOC" : "", |
| diff --git a/arch/powerpc/kernel/watchdog.c b/arch/powerpc/kernel/watchdog.c |
| index af3c15a1d41eb..8ae46c5945d07 100644 |
| --- a/arch/powerpc/kernel/watchdog.c |
| +++ b/arch/powerpc/kernel/watchdog.c |
| @@ -181,11 +181,6 @@ static void watchdog_smp_panic(int cpu, u64 tb) |
| |
| wd_smp_unlock(&flags); |
| |
| - printk_safe_flush(); |
| - /* |
| - * printk_safe_flush() seems to require another print |
| - * before anything actually goes out to console. |
| - */ |
| if (sysctl_hardlockup_all_cpu_backtrace) |
| trigger_allbutself_cpu_backtrace(); |
| |
| diff --git a/arch/powerpc/kexec/crash.c b/arch/powerpc/kexec/crash.c |
| index c9a889880214e..d488311efab1f 100644 |
| --- a/arch/powerpc/kexec/crash.c |
| +++ b/arch/powerpc/kexec/crash.c |
| @@ -311,9 +311,6 @@ void default_machine_crash_shutdown(struct pt_regs *regs) |
| unsigned int i; |
| int (*old_handler)(struct pt_regs *regs); |
| |
| - /* Avoid hardlocking with irresponsive CPU holding logbuf_lock */ |
| - printk_nmi_enter(); |
| - |
| /* |
| * This function is only called after the system |
| * has panicked or is otherwise in a critical state. |
| diff --git a/arch/powerpc/kvm/Kconfig b/arch/powerpc/kvm/Kconfig |
| index 549591d9aaa2c..efb5bfe93f705 100644 |
| --- a/arch/powerpc/kvm/Kconfig |
| +++ b/arch/powerpc/kvm/Kconfig |
| @@ -178,6 +178,7 @@ config KVM_E500MC |
| config KVM_MPIC |
| bool "KVM in-kernel MPIC emulation" |
| depends on KVM && E500 |
| + depends on !PREEMPT_RT |
| select HAVE_KVM_IRQCHIP |
| select HAVE_KVM_IRQFD |
| select HAVE_KVM_IRQ_ROUTING |
| diff --git a/arch/powerpc/mm/Makefile b/arch/powerpc/mm/Makefile |
| index 55b4a8bd408ab..3b4e9e4e25eaa 100644 |
| --- a/arch/powerpc/mm/Makefile |
| +++ b/arch/powerpc/mm/Makefile |
| @@ -16,7 +16,6 @@ obj-$(CONFIG_NEED_MULTIPLE_NODES) += numa.o |
| obj-$(CONFIG_PPC_MM_SLICES) += slice.o |
| obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o |
| obj-$(CONFIG_NOT_COHERENT_CACHE) += dma-noncoherent.o |
| -obj-$(CONFIG_HIGHMEM) += highmem.o |
| obj-$(CONFIG_PPC_COPRO_BASE) += copro_fault.o |
| obj-$(CONFIG_PPC_PTDUMP) += ptdump/ |
| obj-$(CONFIG_KASAN) += kasan/ |
| diff --git a/arch/powerpc/mm/highmem.c b/arch/powerpc/mm/highmem.c |
| deleted file mode 100644 |
| index 624b4438aff9d..0000000000000 |
| --- a/arch/powerpc/mm/highmem.c |
| +++ /dev/null |
| @@ -1,67 +0,0 @@ |
| -// SPDX-License-Identifier: GPL-2.0 |
| -/* |
| - * highmem.c: virtual kernel memory mappings for high memory |
| - * |
| - * PowerPC version, stolen from the i386 version. |
| - * |
| - * Used in CONFIG_HIGHMEM systems for memory pages which |
| - * are not addressable by direct kernel virtual addresses. |
| - * |
| - * Copyright (C) 1999 Gerhard Wichert, Siemens AG |
| - * Gerhard.Wichert@pdb.siemens.de |
| - * |
| - * |
| - * Redesigned the x86 32-bit VM architecture to deal with |
| - * up to 16 Terrabyte physical memory. With current x86 CPUs |
| - * we now support up to 64 Gigabytes physical RAM. |
| - * |
| - * Copyright (C) 1999 Ingo Molnar <mingo@redhat.com> |
| - * |
| - * Reworked for PowerPC by various contributors. Moved from |
| - * highmem.h by Benjamin Herrenschmidt (c) 2009 IBM Corp. |
| - */ |
| - |
| -#include <linux/highmem.h> |
| -#include <linux/module.h> |
| - |
| -void *kmap_atomic_high_prot(struct page *page, pgprot_t prot) |
| -{ |
| - unsigned long vaddr; |
| - int idx, type; |
| - |
| - type = kmap_atomic_idx_push(); |
| - idx = type + KM_TYPE_NR*smp_processor_id(); |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| - WARN_ON(IS_ENABLED(CONFIG_DEBUG_HIGHMEM) && !pte_none(*(kmap_pte - idx))); |
| - __set_pte_at(&init_mm, vaddr, kmap_pte-idx, mk_pte(page, prot), 1); |
| - local_flush_tlb_page(NULL, vaddr); |
| - |
| - return (void*) vaddr; |
| -} |
| -EXPORT_SYMBOL(kmap_atomic_high_prot); |
| - |
| -void kunmap_atomic_high(void *kvaddr) |
| -{ |
| - unsigned long vaddr = (unsigned long) kvaddr & PAGE_MASK; |
| - |
| - if (vaddr < __fix_to_virt(FIX_KMAP_END)) |
| - return; |
| - |
| - if (IS_ENABLED(CONFIG_DEBUG_HIGHMEM)) { |
| - int type = kmap_atomic_idx(); |
| - unsigned int idx; |
| - |
| - idx = type + KM_TYPE_NR * smp_processor_id(); |
| - WARN_ON(vaddr != __fix_to_virt(FIX_KMAP_BEGIN + idx)); |
| - |
| - /* |
| - * force other mappings to Oops if they'll try to access |
| - * this pte without first remap it |
| - */ |
| - pte_clear(&init_mm, vaddr, kmap_pte-idx); |
| - local_flush_tlb_page(NULL, vaddr); |
| - } |
| - |
| - kmap_atomic_idx_pop(); |
| -} |
| -EXPORT_SYMBOL(kunmap_atomic_high); |
| diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c |
| index 22eb1c718e622..1b74565b3e168 100644 |
| --- a/arch/powerpc/mm/mem.c |
| +++ b/arch/powerpc/mm/mem.c |
| @@ -62,11 +62,6 @@ |
| unsigned long long memory_limit; |
| bool init_mem_is_free; |
| |
| -#ifdef CONFIG_HIGHMEM |
| -pte_t *kmap_pte; |
| -EXPORT_SYMBOL(kmap_pte); |
| -#endif |
| - |
| pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, |
| unsigned long size, pgprot_t vma_prot) |
| { |
| @@ -236,8 +231,6 @@ void __init paging_init(void) |
| |
| map_kernel_page(PKMAP_BASE, 0, __pgprot(0)); /* XXX gross */ |
| pkmap_page_table = virt_to_kpte(PKMAP_BASE); |
| - |
| - kmap_pte = virt_to_kpte(__fix_to_virt(FIX_KMAP_BEGIN)); |
| #endif /* CONFIG_HIGHMEM */ |
| |
| printk(KERN_DEBUG "Top of RAM: 0x%llx, Total RAM: 0x%llx\n", |
| diff --git a/arch/powerpc/platforms/powernv/opal-kmsg.c b/arch/powerpc/platforms/powernv/opal-kmsg.c |
| index 6c3bc4b4da983..ec862846bc82c 100644 |
| --- a/arch/powerpc/platforms/powernv/opal-kmsg.c |
| +++ b/arch/powerpc/platforms/powernv/opal-kmsg.c |
| @@ -20,7 +20,8 @@ |
| * message, it just ensures that OPAL completely flushes the console buffer. |
| */ |
| static void kmsg_dump_opal_console_flush(struct kmsg_dumper *dumper, |
| - enum kmsg_dump_reason reason) |
| + enum kmsg_dump_reason reason, |
| + struct kmsg_dumper_iter *iter) |
| { |
| /* |
| * Outside of a panic context the pollers will continue to run, |
| diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c |
| index 245f1f8df6563..f05555dde8e7f 100644 |
| --- a/arch/powerpc/platforms/pseries/iommu.c |
| +++ b/arch/powerpc/platforms/pseries/iommu.c |
| @@ -24,6 +24,7 @@ |
| #include <linux/of.h> |
| #include <linux/iommu.h> |
| #include <linux/rculist.h> |
| +#include <linux/local_lock.h> |
| #include <asm/io.h> |
| #include <asm/prom.h> |
| #include <asm/rtas.h> |
| @@ -190,7 +191,13 @@ static int tce_build_pSeriesLP(unsigned long liobn, long tcenum, long tceshift, |
| return ret; |
| } |
| |
| -static DEFINE_PER_CPU(__be64 *, tce_page); |
| +struct tce_page { |
| + __be64 * page; |
| + local_lock_t lock; |
| +}; |
| +static DEFINE_PER_CPU(struct tce_page, tce_page) = { |
| + .lock = INIT_LOCAL_LOCK(lock), |
| +}; |
| |
| static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, |
| long npages, unsigned long uaddr, |
| @@ -212,9 +219,10 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, |
| direction, attrs); |
| } |
| |
| - local_irq_save(flags); /* to protect tcep and the page behind it */ |
| + /* to protect tcep and the page behind it */ |
| + local_lock_irqsave(&tce_page.lock, flags); |
| |
| - tcep = __this_cpu_read(tce_page); |
| + tcep = __this_cpu_read(tce_page.page); |
| |
| /* This is safe to do since interrupts are off when we're called |
| * from iommu_alloc{,_sg}() |
| @@ -223,12 +231,12 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, |
| tcep = (__be64 *)__get_free_page(GFP_ATOMIC); |
| /* If allocation fails, fall back to the loop implementation */ |
| if (!tcep) { |
| - local_irq_restore(flags); |
| + local_unlock_irqrestore(&tce_page.lock, flags); |
| return tce_build_pSeriesLP(tbl->it_index, tcenum, |
| tbl->it_page_shift, |
| npages, uaddr, direction, attrs); |
| } |
| - __this_cpu_write(tce_page, tcep); |
| + __this_cpu_write(tce_page.page, tcep); |
| } |
| |
| rpn = __pa(uaddr) >> TCE_SHIFT; |
| @@ -258,7 +266,7 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum, |
| tcenum += limit; |
| } while (npages > 0 && !rc); |
| |
| - local_irq_restore(flags); |
| + local_unlock_irqrestore(&tce_page.lock, flags); |
| |
| if (unlikely(rc == H_NOT_ENOUGH_RESOURCES)) { |
| ret = (int)rc; |
| @@ -429,16 +437,17 @@ static int tce_setrange_multi_pSeriesLP(unsigned long start_pfn, |
| DMA_BIDIRECTIONAL, 0); |
| } |
| |
| - local_irq_disable(); /* to protect tcep and the page behind it */ |
| - tcep = __this_cpu_read(tce_page); |
| + /* to protect tcep and the page behind it */ |
| + local_lock_irq(&tce_page.lock); |
| + tcep = __this_cpu_read(tce_page.page); |
| |
| if (!tcep) { |
| tcep = (__be64 *)__get_free_page(GFP_ATOMIC); |
| if (!tcep) { |
| - local_irq_enable(); |
| + local_unlock_irq(&tce_page.lock); |
| return -ENOMEM; |
| } |
| - __this_cpu_write(tce_page, tcep); |
| + __this_cpu_write(tce_page.page, tcep); |
| } |
| |
| proto_tce = TCE_PCI_READ | TCE_PCI_WRITE; |
| @@ -481,7 +490,7 @@ static int tce_setrange_multi_pSeriesLP(unsigned long start_pfn, |
| |
| /* error cleanup: caller will clear whole range */ |
| |
| - local_irq_enable(); |
| + local_unlock_irq(&tce_page.lock); |
| return rc; |
| } |
| |
| diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c |
| index 5559edf36756c..d62b8e053d4c8 100644 |
| --- a/arch/powerpc/xmon/xmon.c |
| +++ b/arch/powerpc/xmon/xmon.c |
| @@ -3005,7 +3005,7 @@ print_address(unsigned long addr) |
| static void |
| dump_log_buf(void) |
| { |
| - struct kmsg_dumper dumper = { .active = 1 }; |
| + struct kmsg_dumper_iter iter = { .active = 1 }; |
| unsigned char buf[128]; |
| size_t len; |
| |
| @@ -3017,9 +3017,9 @@ dump_log_buf(void) |
| catch_memory_errors = 1; |
| sync(); |
| |
| - kmsg_dump_rewind_nolock(&dumper); |
| + kmsg_dump_rewind(&iter); |
| xmon_start_pagination(); |
| - while (kmsg_dump_get_line_nolock(&dumper, false, buf, sizeof(buf), &len)) { |
| + while (kmsg_dump_get_line(&iter, false, buf, sizeof(buf), &len)) { |
| buf[len] = '\0'; |
| printf("%s", buf); |
| } |
| diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig |
| index 896b68e541b2e..54a4e77f2d2a3 100644 |
| --- a/arch/s390/Kconfig |
| +++ b/arch/s390/Kconfig |
| @@ -183,6 +183,7 @@ config S390 |
| select HAVE_RSEQ |
| select HAVE_SYSCALL_TRACEPOINTS |
| select HAVE_VIRT_CPU_ACCOUNTING |
| + select HAVE_VIRT_CPU_ACCOUNTING_IDLE |
| select IOMMU_HELPER if PCI |
| select IOMMU_SUPPORT if PCI |
| select MODULES_USE_ELF_RELA |
| diff --git a/arch/s390/include/asm/spinlock_types.h b/arch/s390/include/asm/spinlock_types.h |
| index cfed272e4fd59..8e28e8176ec88 100644 |
| --- a/arch/s390/include/asm/spinlock_types.h |
| +++ b/arch/s390/include/asm/spinlock_types.h |
| @@ -2,10 +2,6 @@ |
| #ifndef __ASM_SPINLOCK_TYPES_H |
| #define __ASM_SPINLOCK_TYPES_H |
| |
| -#ifndef __LINUX_SPINLOCK_TYPES_H |
| -# error "please don't include this file directly" |
| -#endif |
| - |
| typedef struct { |
| int lock; |
| } __attribute__ ((aligned (4))) arch_spinlock_t; |
| diff --git a/arch/s390/include/asm/vtime.h b/arch/s390/include/asm/vtime.h |
| index 3622d4ebc73aa..fac6a67988eb5 100644 |
| --- a/arch/s390/include/asm/vtime.h |
| +++ b/arch/s390/include/asm/vtime.h |
| @@ -2,7 +2,6 @@ |
| #ifndef _S390_VTIME_H |
| #define _S390_VTIME_H |
| |
| -#define __ARCH_HAS_VTIME_ACCOUNT |
| #define __ARCH_HAS_VTIME_TASK_SWITCH |
| |
| #endif /* _S390_VTIME_H */ |
| diff --git a/arch/s390/kernel/vtime.c b/arch/s390/kernel/vtime.c |
| index 579ec3a8c816f..9b3c5978b6683 100644 |
| --- a/arch/s390/kernel/vtime.c |
| +++ b/arch/s390/kernel/vtime.c |
| @@ -223,35 +223,50 @@ void vtime_flush(struct task_struct *tsk) |
| S390_lowcore.avg_steal_timer = avg_steal; |
| } |
| |
| +static u64 vtime_delta(void) |
| +{ |
| + u64 timer = S390_lowcore.last_update_timer; |
| + |
| + S390_lowcore.last_update_timer = get_vtimer(); |
| + |
| + return timer - S390_lowcore.last_update_timer; |
| +} |
| + |
| /* |
| * Update process times based on virtual cpu times stored by entry.S |
| * to the lowcore fields user_timer, system_timer & steal_clock. |
| */ |
| -void vtime_account_irq_enter(struct task_struct *tsk) |
| +void vtime_account_kernel(struct task_struct *tsk) |
| { |
| - u64 timer; |
| - |
| - timer = S390_lowcore.last_update_timer; |
| - S390_lowcore.last_update_timer = get_vtimer(); |
| - timer -= S390_lowcore.last_update_timer; |
| + u64 delta = vtime_delta(); |
| |
| - if ((tsk->flags & PF_VCPU) && (irq_count() == 0)) |
| - S390_lowcore.guest_timer += timer; |
| - else if (hardirq_count()) |
| - S390_lowcore.hardirq_timer += timer; |
| - else if (in_serving_softirq()) |
| - S390_lowcore.softirq_timer += timer; |
| + if (tsk->flags & PF_VCPU) |
| + S390_lowcore.guest_timer += delta; |
| else |
| - S390_lowcore.system_timer += timer; |
| + S390_lowcore.system_timer += delta; |
| |
| - virt_timer_forward(timer); |
| + virt_timer_forward(delta); |
| } |
| -EXPORT_SYMBOL_GPL(vtime_account_irq_enter); |
| - |
| -void vtime_account_kernel(struct task_struct *tsk) |
| -__attribute__((alias("vtime_account_irq_enter"))); |
| EXPORT_SYMBOL_GPL(vtime_account_kernel); |
| |
| +void vtime_account_softirq(struct task_struct *tsk) |
| +{ |
| + u64 delta = vtime_delta(); |
| + |
| + S390_lowcore.softirq_timer += delta; |
| + |
| + virt_timer_forward(delta); |
| +} |
| + |
| +void vtime_account_hardirq(struct task_struct *tsk) |
| +{ |
| + u64 delta = vtime_delta(); |
| + |
| + S390_lowcore.hardirq_timer += delta; |
| + |
| + virt_timer_forward(delta); |
| +} |
| + |
| /* |
| * Sorted add to a list. List is linear searched until first bigger |
| * element is found. |
| diff --git a/arch/sh/include/asm/fixmap.h b/arch/sh/include/asm/fixmap.h |
| index f38adc189b832..b07fbc7f7bc6a 100644 |
| --- a/arch/sh/include/asm/fixmap.h |
| +++ b/arch/sh/include/asm/fixmap.h |
| @@ -13,9 +13,6 @@ |
| #include <linux/kernel.h> |
| #include <linux/threads.h> |
| #include <asm/page.h> |
| -#ifdef CONFIG_HIGHMEM |
| -#include <asm/kmap_types.h> |
| -#endif |
| |
| /* |
| * Here we define all the compile-time 'special' virtual |
| @@ -53,11 +50,6 @@ enum fixed_addresses { |
| FIX_CMAP_BEGIN, |
| FIX_CMAP_END = FIX_CMAP_BEGIN + (FIX_N_COLOURS * NR_CPUS) - 1, |
| |
| -#ifdef CONFIG_HIGHMEM |
| - FIX_KMAP_BEGIN, /* reserved pte's for temporary kernel mappings */ |
| - FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_TYPE_NR * NR_CPUS) - 1, |
| -#endif |
| - |
| #ifdef CONFIG_IOREMAP_FIXED |
| /* |
| * FIX_IOREMAP entries are useful for mapping physical address |
| diff --git a/arch/sh/include/asm/hardirq.h b/arch/sh/include/asm/hardirq.h |
| index edaea3559a232..9fe4495a8e90b 100644 |
| --- a/arch/sh/include/asm/hardirq.h |
| +++ b/arch/sh/include/asm/hardirq.h |
| @@ -2,16 +2,10 @@ |
| #ifndef __ASM_SH_HARDIRQ_H |
| #define __ASM_SH_HARDIRQ_H |
| |
| -#include <linux/threads.h> |
| -#include <linux/irq.h> |
| - |
| -typedef struct { |
| - unsigned int __softirq_pending; |
| - unsigned int __nmi_count; /* arch dependent */ |
| -} ____cacheline_aligned irq_cpustat_t; |
| - |
| -#include <linux/irq_cpustat.h> /* Standard mappings for irq_cpustat_t above */ |
| - |
| extern void ack_bad_irq(unsigned int irq); |
| +#define ack_bad_irq ack_bad_irq |
| +#define ARCH_WANTS_NMI_IRQSTAT |
| + |
| +#include <asm-generic/hardirq.h> |
| |
| #endif /* __ASM_SH_HARDIRQ_H */ |
| diff --git a/arch/sh/include/asm/kmap_types.h b/arch/sh/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index b78107f923dd3..0000000000000 |
| --- a/arch/sh/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,15 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0 */ |
| -#ifndef __SH_KMAP_TYPES_H |
| -#define __SH_KMAP_TYPES_H |
| - |
| -/* Dummy header just to define km_type. */ |
| - |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| -#define __WITH_KM_FENCE |
| -#endif |
| - |
| -#include <asm-generic/kmap_types.h> |
| - |
| -#undef __WITH_KM_FENCE |
| - |
| -#endif |
| diff --git a/arch/sh/include/asm/spinlock_types.h b/arch/sh/include/asm/spinlock_types.h |
| index e82369f286a20..22ca9a98bbb8b 100644 |
| --- a/arch/sh/include/asm/spinlock_types.h |
| +++ b/arch/sh/include/asm/spinlock_types.h |
| @@ -2,10 +2,6 @@ |
| #ifndef __ASM_SH_SPINLOCK_TYPES_H |
| #define __ASM_SH_SPINLOCK_TYPES_H |
| |
| -#ifndef __LINUX_SPINLOCK_TYPES_H |
| -# error "please don't include this file directly" |
| -#endif |
| - |
| typedef struct { |
| volatile unsigned int lock; |
| } arch_spinlock_t; |
| diff --git a/arch/sh/kernel/irq.c b/arch/sh/kernel/irq.c |
| index 5717c7cbdd97a..5db7af565decc 100644 |
| --- a/arch/sh/kernel/irq.c |
| +++ b/arch/sh/kernel/irq.c |
| @@ -44,7 +44,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) |
| |
| seq_printf(p, "%*s: ", prec, "NMI"); |
| for_each_online_cpu(j) |
| - seq_printf(p, "%10u ", nmi_count(j)); |
| + seq_printf(p, "%10u ", per_cpu(irq_stat.__nmi_count, j)); |
| seq_printf(p, " Non-maskable interrupts\n"); |
| |
| seq_printf(p, "%*s: %10u\n", prec, "ERR", atomic_read(&irq_err_count)); |
| @@ -148,6 +148,7 @@ void irq_ctx_exit(int cpu) |
| hardirq_ctx[cpu] = NULL; |
| } |
| |
| +#ifndef CONFIG_PREEMPT_RT |
| void do_softirq_own_stack(void) |
| { |
| struct thread_info *curctx; |
| @@ -175,6 +176,7 @@ void do_softirq_own_stack(void) |
| "r5", "r6", "r7", "r8", "r9", "r15", "t", "pr" |
| ); |
| } |
| +#endif |
| #else |
| static inline void handle_one_irq(unsigned int irq) |
| { |
| diff --git a/arch/sh/kernel/traps.c b/arch/sh/kernel/traps.c |
| index 9c3d32b80038a..f5beecdac6938 100644 |
| --- a/arch/sh/kernel/traps.c |
| +++ b/arch/sh/kernel/traps.c |
| @@ -186,7 +186,7 @@ BUILD_TRAP_HANDLER(nmi) |
| arch_ftrace_nmi_enter(); |
| |
| nmi_enter(); |
| - nmi_count(cpu)++; |
| + this_cpu_inc(irq_stat.__nmi_count); |
| |
| switch (notify_die(DIE_NMI, "NMI", regs, 0, vec & 0xff, SIGINT)) { |
| case NOTIFY_OK: |
| diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c |
| index 3348e0c4d7698..0db6919af8d32 100644 |
| --- a/arch/sh/mm/init.c |
| +++ b/arch/sh/mm/init.c |
| @@ -362,9 +362,6 @@ void __init mem_init(void) |
| mem_init_print_info(NULL); |
| pr_info("virtual kernel memory layout:\n" |
| " fixmap : 0x%08lx - 0x%08lx (%4ld kB)\n" |
| -#ifdef CONFIG_HIGHMEM |
| - " pkmap : 0x%08lx - 0x%08lx (%4ld kB)\n" |
| -#endif |
| " vmalloc : 0x%08lx - 0x%08lx (%4ld MB)\n" |
| " lowmem : 0x%08lx - 0x%08lx (%4ld MB) (cached)\n" |
| #ifdef CONFIG_UNCACHED_MAPPING |
| @@ -376,11 +373,6 @@ void __init mem_init(void) |
| FIXADDR_START, FIXADDR_TOP, |
| (FIXADDR_TOP - FIXADDR_START) >> 10, |
| |
| -#ifdef CONFIG_HIGHMEM |
| - PKMAP_BASE, PKMAP_BASE+LAST_PKMAP*PAGE_SIZE, |
| - (LAST_PKMAP*PAGE_SIZE) >> 10, |
| -#endif |
| - |
| (unsigned long)VMALLOC_START, VMALLOC_END, |
| (VMALLOC_END - VMALLOC_START) >> 20, |
| |
| diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig |
| index 530b7ec5d3ca9..a38d00d8b7839 100644 |
| --- a/arch/sparc/Kconfig |
| +++ b/arch/sparc/Kconfig |
| @@ -139,6 +139,7 @@ config MMU |
| config HIGHMEM |
| bool |
| default y if SPARC32 |
| + select KMAP_LOCAL |
| |
| config ZONE_DMA |
| bool |
| diff --git a/arch/sparc/include/asm/highmem.h b/arch/sparc/include/asm/highmem.h |
| index 6c35f0d27ee1a..875116209ec14 100644 |
| --- a/arch/sparc/include/asm/highmem.h |
| +++ b/arch/sparc/include/asm/highmem.h |
| @@ -24,7 +24,6 @@ |
| #include <linux/interrupt.h> |
| #include <linux/pgtable.h> |
| #include <asm/vaddrs.h> |
| -#include <asm/kmap_types.h> |
| #include <asm/pgtsrmmu.h> |
| |
| /* declarations for highmem.c */ |
| @@ -33,8 +32,6 @@ extern unsigned long highstart_pfn, highend_pfn; |
| #define kmap_prot __pgprot(SRMMU_ET_PTE | SRMMU_PRIV | SRMMU_CACHE) |
| extern pte_t *pkmap_page_table; |
| |
| -void kmap_init(void) __init; |
| - |
| /* |
| * Right now we initialize only a single pte table. It can be extended |
| * easily, subsequent pte tables have to be allocated in one physical |
| @@ -53,6 +50,11 @@ void kmap_init(void) __init; |
| |
| #define flush_cache_kmaps() flush_cache_all() |
| |
| +/* FIXME: Use __flush_tlb_one(vaddr) instead of flush_cache_all() -- Anton */ |
| +#define arch_kmap_local_post_map(vaddr, pteval) flush_cache_all() |
| +#define arch_kmap_local_post_unmap(vaddr) flush_cache_all() |
| + |
| + |
| #endif /* __KERNEL__ */ |
| |
| #endif /* _ASM_HIGHMEM_H */ |
| diff --git a/arch/sparc/include/asm/kmap_types.h b/arch/sparc/include/asm/kmap_types.h |
| deleted file mode 100644 |
| index 55a99b6bd91e0..0000000000000 |
| --- a/arch/sparc/include/asm/kmap_types.h |
| +++ /dev/null |
| @@ -1,11 +0,0 @@ |
| -/* SPDX-License-Identifier: GPL-2.0 */ |
| -#ifndef _ASM_KMAP_TYPES_H |
| -#define _ASM_KMAP_TYPES_H |
| - |
| -/* Dummy header just to define km_type. None of this |
| - * is actually used on sparc. -DaveM |
| - */ |
| - |
| -#include <asm-generic/kmap_types.h> |
| - |
| -#endif |
| diff --git a/arch/sparc/include/asm/vaddrs.h b/arch/sparc/include/asm/vaddrs.h |
| index 84d054b07a6ff..4fec0341e2a81 100644 |
| --- a/arch/sparc/include/asm/vaddrs.h |
| +++ b/arch/sparc/include/asm/vaddrs.h |
| @@ -32,13 +32,13 @@ |
| #define SRMMU_NOCACHE_ALCRATIO 64 /* 256 pages per 64MB of system RAM */ |
| |
| #ifndef __ASSEMBLY__ |
| -#include <asm/kmap_types.h> |
| +#include <asm/kmap_size.h> |
| |
| enum fixed_addresses { |
| FIX_HOLE, |
| #ifdef CONFIG_HIGHMEM |
| FIX_KMAP_BEGIN, |
| - FIX_KMAP_END = (KM_TYPE_NR * NR_CPUS), |
| + FIX_KMAP_END = (KM_MAX_IDX * NR_CPUS), |
| #endif |
| __end_of_fixed_addresses |
| }; |
| diff --git a/arch/sparc/kernel/irq_64.c b/arch/sparc/kernel/irq_64.c |
| index 3ec9f1402aad3..eb21682abfcb1 100644 |
| --- a/arch/sparc/kernel/irq_64.c |
| +++ b/arch/sparc/kernel/irq_64.c |
| @@ -854,6 +854,7 @@ void __irq_entry handler_irq(int pil, struct pt_regs *regs) |
| set_irq_regs(old_regs); |
| } |
| |
| +#ifndef CONFIG_PREEMPT_RT |
| void do_softirq_own_stack(void) |
| { |
| void *orig_sp, *sp = softirq_stack[smp_processor_id()]; |
| @@ -868,6 +869,7 @@ void do_softirq_own_stack(void) |
| __asm__ __volatile__("mov %0, %%sp" |
| : : "r" (orig_sp)); |
| } |
| +#endif |
| |
| #ifdef CONFIG_HOTPLUG_CPU |
| void fixup_irqs(void) |
| diff --git a/arch/sparc/mm/Makefile b/arch/sparc/mm/Makefile |
| index b078205b70e0b..68db1f859b028 100644 |
| --- a/arch/sparc/mm/Makefile |
| +++ b/arch/sparc/mm/Makefile |
| @@ -15,6 +15,3 @@ obj-$(CONFIG_SPARC32) += leon_mm.o |
| |
| # Only used by sparc64 |
| obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o |
| - |
| -# Only used by sparc32 |
| -obj-$(CONFIG_HIGHMEM) += highmem.o |
| diff --git a/arch/sparc/mm/highmem.c b/arch/sparc/mm/highmem.c |
| deleted file mode 100644 |
| index 8f2a2afb048a2..0000000000000 |
| --- a/arch/sparc/mm/highmem.c |
| +++ /dev/null |
| @@ -1,115 +0,0 @@ |
| -// SPDX-License-Identifier: GPL-2.0 |
| -/* |
| - * highmem.c: virtual kernel memory mappings for high memory |
| - * |
| - * Provides kernel-static versions of atomic kmap functions originally |
| - * found as inlines in include/asm-sparc/highmem.h. These became |
| - * needed as kmap_atomic() and kunmap_atomic() started getting |
| - * called from within modules. |
| - * -- Tomas Szepe <szepe@pinerecords.com>, September 2002 |
| - * |
| - * But kmap_atomic() and kunmap_atomic() cannot be inlined in |
| - * modules because they are loaded with btfixup-ped functions. |
| - */ |
| - |
| -/* |
| - * The use of kmap_atomic/kunmap_atomic is discouraged - kmap/kunmap |
| - * gives a more generic (and caching) interface. But kmap_atomic can |
| - * be used in IRQ contexts, so in some (very limited) cases we need it. |
| - * |
| - * XXX This is an old text. Actually, it's good to use atomic kmaps, |
| - * provided you remember that they are atomic and not try to sleep |
| - * with a kmap taken, much like a spinlock. Non-atomic kmaps are |
| - * shared by CPUs, and so precious, and establishing them requires IPI. |
| - * Atomic kmaps are lightweight and we may have NCPUS more of them. |
| - */ |
| -#include <linux/highmem.h> |
| -#include <linux/export.h> |
| -#include <linux/mm.h> |
| - |
| -#include <asm/cacheflush.h> |
| -#include <asm/tlbflush.h> |
| -#include <asm/vaddrs.h> |
| - |
| -static pte_t *kmap_pte; |
| - |
| -void __init kmap_init(void) |
| -{ |
| - unsigned long address = __fix_to_virt(FIX_KMAP_BEGIN); |
| - |
| - /* cache the first kmap pte */ |
| - kmap_pte = virt_to_kpte(address); |
| -} |
| - |
| -void *kmap_atomic_high_prot(struct page *page, pgprot_t prot) |
| -{ |
| - unsigned long vaddr; |
| - long idx, type; |
| - |
| - type = kmap_atomic_idx_push(); |
| - idx = type + KM_TYPE_NR*smp_processor_id(); |
| - vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| - |
| -/* XXX Fix - Anton */ |
| -#if 0 |
| - __flush_cache_one(vaddr); |
| -#else |
| - flush_cache_all(); |
| -#endif |
| - |
| -#ifdef CONFIG_DEBUG_HIGHMEM |
| - BUG_ON(!pte_none(*(kmap_pte-idx))); |
| -#endif |
|