forge/linux-pinenote
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip

Browse source 
* 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip: (311 commits)
  perf tools: Add mode to build without newt support
  perf symbols: symbol inconsistency message should be done only at verbose=1
  perf tui: Add explicit -lslang option
  perf options: Type check all the remaining OPT_ variants
  perf options: Type check OPT_BOOLEAN and fix the offenders
  perf options: Check v type in OPT_U?INTEGER
  perf options: Introduce OPT_UINTEGER
  perf tui: Add workaround for slang < 2.1.4
  perf record: Fix bug mismatch with -c option definition
  perf options: Introduce OPT_U64
  perf tui: Add help window to show key associations
  perf tui: Make <- exit menus too
  perf newt: Add single key shortcuts for zoom into DSO and threads
  perf newt: Exit browser unconditionally when CTRL+C, q or Q is pressed
  perf newt: Fix the 'A'/'a' shortcut for annotate
  perf newt: Make <- exit the ui_browser
  x86, perf: P4 PMU - fix counters management logic
  perf newt: Make <- zoom out filters
  perf report: Report number of events, not samples
  perf hist: Clarify events_stats fields usage
  ...

Fix up trivial conflicts in kernel/fork.c and tools/perf/builtin-record.c
This commit is contained in:
Linus Torvalds 2010-05-18 08:19:03 -07:00
commit 4d7b4ac22f
207 changed files with 14703 additions and 8164 deletions
Download patch file Download diff file Expand all files Collapse all files

10
Documentation/kprobes.txt
View file

@ -165,8 +165,8 @@ the user entry_handler invocation is also skipped.
1.4 How Does Jump Optimization Work? 1.4 How Does Jump Optimization Work?
If you configured your kernel with CONFIG_OPTPROBES=y (currently If your kernel is built with CONFIG_OPTPROBES=y (currently this flag
this option is supported on x86/x86-64, non-preemptive kernel) and is automatically set 'y' on x86/x86-64, non-preemptive kernel) and
the "debug.kprobes_optimization" kernel parameter is set to 1 (see the "debug.kprobes_optimization" kernel parameter is set to 1 (see
sysctl(8)), Kprobes tries to reduce probe-hit overhead by using a jump sysctl(8)), Kprobes tries to reduce probe-hit overhead by using a jump
instruction instead of a breakpoint instruction at each probepoint. instruction instead of a breakpoint instruction at each probepoint.
@ -271,8 +271,6 @@ tweak the kernel's execution path, you need to suppress optimization,
using one of the following techniques: using one of the following techniques:
- Specify an empty function for the kprobe's post_handler or break_handler. - Specify an empty function for the kprobe's post_handler or break_handler.
or or
- Config CONFIG_OPTPROBES=n.
or
- Execute 'sysctl -w debug.kprobes_optimization=n' - Execute 'sysctl -w debug.kprobes_optimization=n'
2. Architectures Supported 2. Architectures Supported
@ -307,10 +305,6 @@ it useful to "Compile the kernel with debug info" (CONFIG_DEBUG_INFO),
so you can use "objdump -d -l vmlinux" to see the source-to-object so you can use "objdump -d -l vmlinux" to see the source-to-object
code mapping. code mapping.
If you want to reduce probing overhead, set "Kprobes jump optimization
support" (CONFIG_OPTPROBES) to "y". You can find this option under the
"Kprobes" line.
4. API Reference 4. API Reference
The Kprobes API includes a "register" function and an "unregister" The Kprobes API includes a "register" function and an "unregister"

2
Documentation/trace/kprobetrace.txt
View file

@ -41,6 +41,8 @@ Synopsis of kprobe_events
$retval : Fetch return value.(*) $retval : Fetch return value.(*)
+|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(**) +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(**)
NAME=FETCHARG : Set NAME as the argument name of FETCHARG. NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
(u8/u16/u32/u64/s8/s16/s32/s64) are supported.
(*) only for return probe. (*) only for return probe.
(**) this is useful for fetching a field of data structures. (**) this is useful for fetching a field of data structures.

10
MAINTAINERS
View file

@ -4354,13 +4354,13 @@ M: Paul Mackerras <paulus@samba.org>
M: Ingo Molnar <mingo@elte.hu> M: Ingo Molnar <mingo@elte.hu>
M: Arnaldo Carvalho de Melo <acme@redhat.com> M: Arnaldo Carvalho de Melo <acme@redhat.com>
S: Supported S: Supported
F: kernel/perf_event.c F: kernel/perf_event*.c
F: include/linux/perf_event.h F: include/linux/perf_event.h
F: arch/*/kernel/perf_event.c F: arch/*/kernel/perf_event*.c
F: arch/*/kernel/*/perf_event.c F: arch/*/kernel/*/perf_event*.c
F: arch/*/kernel/*/*/perf_event.c F: arch/*/kernel/*/*/perf_event*.c
F: arch/*/include/asm/perf_event.h F: arch/*/include/asm/perf_event.h
F: arch/*/lib/perf_event.c F: arch/*/lib/perf_event*.c
F: arch/*/kernel/perf_callchain.c F: arch/*/kernel/perf_callchain.c
F: tools/perf/ F: tools/perf/

20
arch/Kconfig
View file

@ -42,15 +42,10 @@ config KPROBES
If in doubt, say "N". If in doubt, say "N".
config OPTPROBES config OPTPROBES
bool "Kprobes jump optimization support (EXPERIMENTAL)" def_bool y
default y depends on KPROBES && HAVE_OPTPROBES
depends on KPROBES
depends on !PREEMPT depends on !PREEMPT
depends on HAVE_OPTPROBES
select KALLSYMS_ALL select KALLSYMS_ALL
help
This option will allow kprobes to optimize breakpoint to
a jump for reducing its overhead.
config HAVE_EFFICIENT_UNALIGNED_ACCESS config HAVE_EFFICIENT_UNALIGNED_ACCESS
bool bool
@ -142,6 +137,17 @@ config HAVE_HW_BREAKPOINT
bool bool
depends on PERF_EVENTS depends on PERF_EVENTS
config HAVE_MIXED_BREAKPOINTS_REGS
bool
depends on HAVE_HW_BREAKPOINT
help
Depending on the arch implementation of hardware breakpoints,
some of them have separate registers for data and instruction
breakpoints addresses, others have mixed registers to store
them but define the access type in a control register.
Select this option if your arch implements breakpoints under the
latter fashion.
config HAVE_USER_RETURN_NOTIFIER config HAVE_USER_RETURN_NOTIFIER
bool bool

129
arch/powerpc/kernel/perf_event.c
View file

@ -35,6 +35,9 @@ struct cpu_hw_events {
u64 alternatives[MAX_HWEVENTS][MAX_EVENT_ALTERNATIVES]; u64 alternatives[MAX_HWEVENTS][MAX_EVENT_ALTERNATIVES];
unsigned long amasks[MAX_HWEVENTS][MAX_EVENT_ALTERNATIVES]; unsigned long amasks[MAX_HWEVENTS][MAX_EVENT_ALTERNATIVES];
unsigned long avalues[MAX_HWEVENTS][MAX_EVENT_ALTERNATIVES]; unsigned long avalues[MAX_HWEVENTS][MAX_EVENT_ALTERNATIVES];
unsigned int group_flag;
int n_txn_start;
}; };
DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events); DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events);
@ -718,66 +721,6 @@ static int collect_events(struct perf_event *group, int max_count,
return n; return n;
} }
static void event_sched_in(struct perf_event *event)
{
event->state = PERF_EVENT_STATE_ACTIVE;
event->oncpu = smp_processor_id();
event->tstamp_running += event->ctx->time - event->tstamp_stopped;
if (is_software_event(event))
event->pmu->enable(event);
}
/*
* Called to enable a whole group of events.
* Returns 1 if the group was enabled, or -EAGAIN if it could not be.
* Assumes the caller has disabled interrupts and has
* frozen the PMU with hw_perf_save_disable.
*/
int hw_perf_group_sched_in(struct perf_event *group_leader,
struct perf_cpu_context *cpuctx,
struct perf_event_context *ctx)
{
struct cpu_hw_events *cpuhw;
long i, n, n0;
struct perf_event *sub;
if (!ppmu)
return 0;
cpuhw = &__get_cpu_var(cpu_hw_events);
n0 = cpuhw->n_events;
n = collect_events(group_leader, ppmu->n_counter - n0,
&cpuhw->event[n0], &cpuhw->events[n0],
&cpuhw->flags[n0]);
if (n < 0)
return -EAGAIN;
if (check_excludes(cpuhw->event, cpuhw->flags, n0, n))
return -EAGAIN;
i = power_check_constraints(cpuhw, cpuhw->events, cpuhw->flags, n + n0);
if (i < 0)
return -EAGAIN;
cpuhw->n_events = n0 + n;
cpuhw->n_added += n;
/*
* OK, this group can go on; update event states etc.,
* and enable any software events
*/
for (i = n0; i < n0 + n; ++i)
cpuhw->event[i]->hw.config = cpuhw->events[i];
cpuctx->active_oncpu += n;
n = 1;
event_sched_in(group_leader);
list_for_each_entry(sub, &group_leader->sibling_list, group_entry) {
if (sub->state != PERF_EVENT_STATE_OFF) {
event_sched_in(sub);
++n;
}
}
ctx->nr_active += n;
return 1;
}
/* /*
* Add a event to the PMU. * Add a event to the PMU.
* If all events are not already frozen, then we disable and * If all events are not already frozen, then we disable and
@ -805,12 +748,22 @@ static int power_pmu_enable(struct perf_event *event)
cpuhw->event[n0] = event; cpuhw->event[n0] = event;
cpuhw->events[n0] = event->hw.config; cpuhw->events[n0] = event->hw.config;
cpuhw->flags[n0] = event->hw.event_base; cpuhw->flags[n0] = event->hw.event_base;
/*
* If group events scheduling transaction was started,
* skip the schedulability test here, it will be peformed
* at commit time(->commit_txn) as a whole
*/
if (cpuhw->group_flag & PERF_EVENT_TXN_STARTED)
goto nocheck;
if (check_excludes(cpuhw->event, cpuhw->flags, n0, 1)) if (check_excludes(cpuhw->event, cpuhw->flags, n0, 1))
goto out; goto out;
if (power_check_constraints(cpuhw, cpuhw->events, cpuhw->flags, n0 + 1)) if (power_check_constraints(cpuhw, cpuhw->events, cpuhw->flags, n0 + 1))
goto out; goto out;
event->hw.config = cpuhw->events[n0]; event->hw.config = cpuhw->events[n0];
nocheck:
++cpuhw->n_events; ++cpuhw->n_events;
++cpuhw->n_added; ++cpuhw->n_added;
@ -896,11 +849,65 @@ static void power_pmu_unthrottle(struct perf_event *event)
local_irq_restore(flags); local_irq_restore(flags);
} }
/*
* Start group events scheduling transaction
* Set the flag to make pmu::enable() not perform the
* schedulability test, it will be performed at commit time
*/
void power_pmu_start_txn(const struct pmu *pmu)
{
struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
cpuhw->group_flag |= PERF_EVENT_TXN_STARTED;
cpuhw->n_txn_start = cpuhw->n_events;
}
/*
* Stop group events scheduling transaction
* Clear the flag and pmu::enable() will perform the
* schedulability test.
*/
void power_pmu_cancel_txn(const struct pmu *pmu)
{
struct cpu_hw_events *cpuhw = &__get_cpu_var(cpu_hw_events);
cpuhw->group_flag &= ~PERF_EVENT_TXN_STARTED;
}
/*
* Commit group events scheduling transaction
* Perform the group schedulability test as a whole
* Return 0 if success
*/
int power_pmu_commit_txn(const struct pmu *pmu)
{
struct cpu_hw_events *cpuhw;
long i, n;
if (!ppmu)
return -EAGAIN;
cpuhw = &__get_cpu_var(cpu_hw_events);
n = cpuhw->n_events;
if (check_excludes(cpuhw->event, cpuhw->flags, 0, n))
return -EAGAIN;
i = power_check_constraints(cpuhw, cpuhw->events, cpuhw->flags, n);
if (i < 0)
return -EAGAIN;
for (i = cpuhw->n_txn_start; i < n; ++i)
cpuhw->event[i]->hw.config = cpuhw->events[i];
return 0;
}
struct pmu power_pmu = { struct pmu power_pmu = {
.enable = power_pmu_enable, .enable = power_pmu_enable,
.disable = power_pmu_disable, .disable = power_pmu_disable,
.read = power_pmu_read, .read = power_pmu_read,
.unthrottle = power_pmu_unthrottle, .unthrottle = power_pmu_unthrottle,
.start_txn = power_pmu_start_txn,
.cancel_txn = power_pmu_cancel_txn,
.commit_txn = power_pmu_commit_txn,
}; };
/* /*

1
arch/sh/Kconfig
View file

@ -44,6 +44,7 @@ config SUPERH32
select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_GRAPH_TRACER
select HAVE_ARCH_KGDB select HAVE_ARCH_KGDB
select HAVE_HW_BREAKPOINT select HAVE_HW_BREAKPOINT
select HAVE_MIXED_BREAKPOINTS_REGS
select PERF_EVENTS if HAVE_HW_BREAKPOINT select PERF_EVENTS if HAVE_HW_BREAKPOINT
select ARCH_HIBERNATION_POSSIBLE if MMU select ARCH_HIBERNATION_POSSIBLE if MMU

10
arch/sh/include/asm/hw_breakpoint.h
View file

@ -46,10 +46,14 @@ struct pmu;
/* Maximum number of UBC channels */ /* Maximum number of UBC channels */
#define HBP_NUM 2 #define HBP_NUM 2
static inline int hw_breakpoint_slots(int type)
{
return HBP_NUM;
}
/* arch/sh/kernel/hw_breakpoint.c */ /* arch/sh/kernel/hw_breakpoint.c */
extern int arch_check_va_in_userspace(unsigned long va, u16 hbp_len); extern int arch_check_bp_in_kernelspace(struct perf_event *bp);
extern int arch_validate_hwbkpt_settings(struct perf_event *bp, extern int arch_validate_hwbkpt_settings(struct perf_event *bp);
struct task_struct *tsk);
extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused, extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused,
unsigned long val, void *data); unsigned long val, void *data);

34
arch/sh/kernel/hw_breakpoint.c
View file

@ -119,26 +119,17 @@ static int get_hbp_len(u16 hbp_len)
return len_in_bytes; return len_in_bytes;
} }
/*
* Check for virtual address in user space.
*/
int arch_check_va_in_userspace(unsigned long va, u16 hbp_len)
{
unsigned int len;
len = get_hbp_len(hbp_len);
return (va <= TASK_SIZE - len);
}
/* /*
* Check for virtual address in kernel space. * Check for virtual address in kernel space.
*/ */
static int arch_check_va_in_kernelspace(unsigned long va, u8 hbp_len) int arch_check_bp_in_kernelspace(struct perf_event *bp)
{ {
unsigned int len; unsigned int len;
unsigned long va;
struct arch_hw_breakpoint *info = counter_arch_bp(bp);
len = get_hbp_len(hbp_len); va = info->address;
len = get_hbp_len(info->len);
return (va >= TASK_SIZE) && ((va + len - 1) >= TASK_SIZE); return (va >= TASK_SIZE) && ((va + len - 1) >= TASK_SIZE);
} }
@ -226,8 +217,7 @@ static int arch_build_bp_info(struct perf_event *bp)
/* /*
* Validate the arch-specific HW Breakpoint register settings * Validate the arch-specific HW Breakpoint register settings
*/ */
int arch_validate_hwbkpt_settings(struct perf_event *bp, int arch_validate_hwbkpt_settings(struct perf_event *bp)
struct task_struct *tsk)
{ {
struct arch_hw_breakpoint *info = counter_arch_bp(bp); struct arch_hw_breakpoint *info = counter_arch_bp(bp);
unsigned int align; unsigned int align;
@ -270,15 +260,6 @@ int arch_validate_hwbkpt_settings(struct perf_event *bp,
if (info->address & align) if (info->address & align)
return -EINVAL; return -EINVAL;
/* Check that the virtual address is in the proper range */
if (tsk) {
if (!arch_check_va_in_userspace(info->address, info->len))
return -EFAULT;
} else {
if (!arch_check_va_in_kernelspace(info->address, info->len))
return -EFAULT;
}
return 0; return 0;
} }
@ -363,8 +344,7 @@ static int __kprobes hw_breakpoint_handler(struct die_args *args)
perf_bp_event(bp, args->regs); perf_bp_event(bp, args->regs);
/* Deliver the signal to userspace */ /* Deliver the signal to userspace */
if (arch_check_va_in_userspace(bp->attr.bp_addr, if (!arch_check_bp_in_kernelspace(bp)) {
bp->attr.bp_len)) {
siginfo_t info; siginfo_t info;
info.si_signo = args->signr; info.si_signo = args->signr;

2
arch/sh/kernel/ptrace_32.c
View file

@ -85,7 +85,7 @@ static int set_single_step(struct task_struct *tsk, unsigned long addr)
bp = thread->ptrace_bps[0]; bp = thread->ptrace_bps[0];
if (!bp) { if (!bp) {
hw_breakpoint_init(&attr); ptrace_breakpoint_init(&attr);
attr.bp_addr = addr; attr.bp_addr = addr;
attr.bp_len = HW_BREAKPOINT_LEN_2; attr.bp_len = HW_BREAKPOINT_LEN_2;

4
arch/x86/Kconfig
View file

@ -53,11 +53,15 @@ config X86
select HAVE_KERNEL_LZMA select HAVE_KERNEL_LZMA
select HAVE_KERNEL_LZO select HAVE_KERNEL_LZO
select HAVE_HW_BREAKPOINT select HAVE_HW_BREAKPOINT
select HAVE_MIXED_BREAKPOINTS_REGS
select PERF_EVENTS select PERF_EVENTS
select ANON_INODES select ANON_INODES
select HAVE_ARCH_KMEMCHECK select HAVE_ARCH_KMEMCHECK
select HAVE_USER_RETURN_NOTIFIER select HAVE_USER_RETURN_NOTIFIER
config INSTRUCTION_DECODER
def_bool (KPROBES || PERF_EVENTS)
config OUTPUT_FORMAT config OUTPUT_FORMAT
string string
default "elf32-i386" if X86_32 default "elf32-i386" if X86_32

20
arch/x86/Kconfig.cpu
View file

@ -502,23 +502,3 @@ config CPU_SUP_UMC_32
CPU might render the kernel unbootable. CPU might render the kernel unbootable.
If unsure, say N. If unsure, say N.
config X86_DS
def_bool X86_PTRACE_BTS
depends on X86_DEBUGCTLMSR
select HAVE_HW_BRANCH_TRACER
config X86_PTRACE_BTS
bool "Branch Trace Store"
default y
depends on X86_DEBUGCTLMSR
depends on BROKEN
---help---
This adds a ptrace interface to the hardware's branch trace store.
Debuggers may use it to collect an execution trace of the debugged
application in order to answer the question 'how did I get here?'.
Debuggers may trace user mode as well as kernel mode.
Say Y unless there is no application development on this machine
and you want to save a small amount of code size.

9
arch/x86/Kconfig.debug
View file

@ -174,15 +174,6 @@ config IOMMU_LEAK
Add a simple leak tracer to the IOMMU code. This is useful when you Add a simple leak tracer to the IOMMU code. This is useful when you
are debugging a buggy device driver that leaks IOMMU mappings. are debugging a buggy device driver that leaks IOMMU mappings.
config X86_DS_SELFTEST
bool "DS selftest"
default y
depends on DEBUG_KERNEL
depends on X86_DS
---help---
Perform Debug Store selftests at boot time.
If in doubt, say "N".
config HAVE_MMIOTRACE_SUPPORT config HAVE_MMIOTRACE_SUPPORT
def_bool y def_bool y

13
arch/x86/include/asm/apic.h
View file

@ -373,6 +373,7 @@ extern atomic_t init_deasserted;
extern int wakeup_secondary_cpu_via_nmi(int apicid, unsigned long start_eip); extern int wakeup_secondary_cpu_via_nmi(int apicid, unsigned long start_eip);
#endif #endif
#ifdef CONFIG_X86_LOCAL_APIC
static inline u32 apic_read(u32 reg) static inline u32 apic_read(u32 reg)
{ {
return apic->read(reg); return apic->read(reg);
@ -403,10 +404,19 @@ static inline u32 safe_apic_wait_icr_idle(void)
return apic->safe_wait_icr_idle(); return apic->safe_wait_icr_idle();
} }
#else /* CONFIG_X86_LOCAL_APIC */
static inline u32 apic_read(u32 reg) { return 0; }
static inline void apic_write(u32 reg, u32 val) { }
static inline u64 apic_icr_read(void) { return 0; }
static inline void apic_icr_write(u32 low, u32 high) { }
static inline void apic_wait_icr_idle(void) { }
static inline u32 safe_apic_wait_icr_idle(void) { return 0; }
#endif /* CONFIG_X86_LOCAL_APIC */
static inline void ack_APIC_irq(void) static inline void ack_APIC_irq(void)
{ {
#ifdef CONFIG_X86_LOCAL_APIC
/* /*
* ack_APIC_irq() actually gets compiled as a single instruction * ack_APIC_irq() actually gets compiled as a single instruction
* ... yummie. * ... yummie.
@ -414,7 +424,6 @@ static inline void ack_APIC_irq(void)
/* Docs say use 0 for future compatibility */ /* Docs say use 0 for future compatibility */
apic_write(APIC_EOI, 0); apic_write(APIC_EOI, 0);
#endif
} }
static inline unsigned default_get_apic_id(unsigned long x) static inline unsigned default_get_apic_id(unsigned long x)

302
arch/x86/include/asm/ds.h
View file

@ -1,302 +0,0 @@
/*
* Debug Store (DS) support
*
* This provides a low-level interface to the hardware's Debug Store
* feature that is used for branch trace store (BTS) and
* precise-event based sampling (PEBS).
*
* It manages:
* - DS and BTS hardware configuration
* - buffer overflow handling (to be done)
* - buffer access
*
* It does not do:
* - security checking (is the caller allowed to trace the task)
* - buffer allocation (memory accounting)
*
*
* Copyright (C) 2007-2009 Intel Corporation.
* Markus Metzger <markus.t.metzger@intel.com>, 2007-2009
*/
#ifndef _ASM_X86_DS_H
#define _ASM_X86_DS_H
#include <linux/types.h>
#include <linux/init.h>
#include <linux/err.h>
#ifdef CONFIG_X86_DS
struct task_struct;
struct ds_context;
struct ds_tracer;
struct bts_tracer;
struct pebs_tracer;
typedef void (*bts_ovfl_callback_t)(struct bts_tracer *);
typedef void (*pebs_ovfl_callback_t)(struct pebs_tracer *);
/*
* A list of features plus corresponding macros to talk about them in
* the ds_request function's flags parameter.
*
* We use the enum to index an array of corresponding control bits;
* we use the macro to index a flags bit-vector.
*/
enum ds_feature {
dsf_bts = 0,
dsf_bts_kernel,
#define BTS_KERNEL (1 << dsf_bts_kernel)
/* trace kernel-mode branches */
dsf_bts_user,
#define BTS_USER (1 << dsf_bts_user)
/* trace user-mode branches */
dsf_bts_overflow,
dsf_bts_max,
dsf_pebs = dsf_bts_max,
dsf_pebs_max,
dsf_ctl_max = dsf_pebs_max,
dsf_bts_timestamps = dsf_ctl_max,
#define BTS_TIMESTAMPS (1 << dsf_bts_timestamps)
/* add timestamps into BTS trace */
#define BTS_USER_FLAGS (BTS_KERNEL | BTS_USER | BTS_TIMESTAMPS)
};
/*
* Request BTS or PEBS
*
* Due to alignement constraints, the actual buffer may be slightly
* smaller than the requested or provided buffer.
*
* Returns a pointer to a tracer structure on success, or
* ERR_PTR(errcode) on failure.
*
* The interrupt threshold is independent from the overflow callback
* to allow users to use their own overflow interrupt handling mechanism.
*
* The function might sleep.
*
* task: the task to request recording for
* cpu: the cpu to request recording for
* base: the base pointer for the (non-pageable) buffer;
* size: the size of the provided buffer in bytes
* ovfl: pointer to a function to be called on buffer overflow;
* NULL if cyclic buffer requested
* th: the interrupt threshold in records from the end of the buffer;
* -1 if no interrupt threshold is requested.
* flags: a bit-mask of the above flags
*/
extern struct bts_tracer *ds_request_bts_task(struct task_struct *task,
void *base, size_t size,
bts_ovfl_callback_t ovfl,
size_t th, unsigned int flags);
extern struct bts_tracer *ds_request_bts_cpu(int cpu, void *base, size_t size,
bts_ovfl_callback_t ovfl,
size_t th, unsigned int flags);
extern struct pebs_tracer *ds_request_pebs_task(struct task_struct *task,
void *base, size_t size,
pebs_ovfl_callback_t ovfl,
size_t th, unsigned int flags);
extern struct pebs_tracer *ds_request_pebs_cpu(int cpu,
void *base, size_t size,
pebs_ovfl_callback_t ovfl,
size_t th, unsigned int flags);
/*
* Release BTS or PEBS resources
* Suspend and resume BTS or PEBS tracing
*
* Must be called with irq's enabled.
*
* tracer: the tracer handle returned from ds_request_~()
*/
extern void ds_release_bts(struct bts_tracer *tracer);
extern void ds_suspend_bts(struct bts_tracer *tracer);
extern void ds_resume_bts(struct bts_tracer *tracer);
extern void ds_release_pebs(struct pebs_tracer *tracer);
extern void ds_suspend_pebs(struct pebs_tracer *tracer);
extern void ds_resume_pebs(struct pebs_tracer *tracer);
/*
* Release BTS or PEBS resources
* Suspend and resume BTS or PEBS tracing
*
* Cpu tracers must call this on the traced cpu.
* Task tracers must call ds_release_~_noirq() for themselves.
*
* May be called with irq's disabled.
*
* Returns 0 if successful;
* -EPERM if the cpu tracer does not trace the current cpu.
* -EPERM if the task tracer does not trace itself.
*
* tracer: the tracer handle returned from ds_request_~()
*/
extern int ds_release_bts_noirq(struct bts_tracer *tracer);
extern int ds_suspend_bts_noirq(struct bts_tracer *tracer);
extern int ds_resume_bts_noirq(struct bts_tracer *tracer);
extern int ds_release_pebs_noirq(struct pebs_tracer *tracer);
extern int ds_suspend_pebs_noirq(struct pebs_tracer *tracer);
extern int ds_resume_pebs_noirq(struct pebs_tracer *tracer);
/*
* The raw DS buffer state as it is used for BTS and PEBS recording.
*
* This is the low-level, arch-dependent interface for working
* directly on the raw trace data.
*/
struct ds_trace {
/* the number of bts/pebs records */
size_t n;
/* the size of a bts/pebs record in bytes */
size_t size;
/* pointers into the raw buffer:
- to the first entry */
void *begin;
/* - one beyond the last entry */
void *end;
/* - one beyond the newest entry */
void *top;
/* - the interrupt threshold */
void *ith;
/* flags given on ds_request() */
unsigned int flags;
};
/*
* An arch-independent view on branch trace data.
*/
enum bts_qualifier {
bts_invalid,
#define BTS_INVALID bts_invalid
bts_branch,
#define BTS_BRANCH bts_branch
bts_task_arrives,
#define BTS_TASK_ARRIVES bts_task_arrives
bts_task_departs,
#define BTS_TASK_DEPARTS bts_task_departs
bts_qual_bit_size = 4,
bts_qual_max = (1 << bts_qual_bit_size),
};
struct bts_struct {
__u64 qualifier;
union {
/* BTS_BRANCH */
struct {
__u64 from;
__u64 to;
} lbr;
/* BTS_TASK_ARRIVES or BTS_TASK_DEPARTS */
struct {
__u64 clock;
pid_t pid;
} event;
} variant;
};
/*
* The BTS state.
*
* This gives access to the raw DS state and adds functions to provide
* an arch-independent view of the BTS data.
*/
struct bts_trace {
struct ds_trace ds;
int (*read)(struct bts_tracer *tracer, const void *at,
struct bts_struct *out);
int (*write)(struct bts_tracer *tracer, const struct bts_struct *in);
};
/*
* The PEBS state.
*
* This gives access to the raw DS state and the PEBS-specific counter
* reset value.
*/
struct pebs_trace {
struct ds_trace ds;
/* the number of valid counters in the below array */
unsigned int counters;
#define MAX_PEBS_COUNTERS 4
/* the counter reset value */
unsigned long long counter_reset[MAX_PEBS_COUNTERS];
};
/*
* Read the BTS or PEBS trace.
*
* Returns a view on the trace collected for the parameter tracer.
*
* The view remains valid as long as the traced task is not running or
* the tracer is suspended.
* Writes into the trace buffer are not reflected.
*
* tracer: the tracer handle returned from ds_request_~()
*/
extern const struct bts_trace *ds_read_bts(struct bts_tracer *tracer);
extern const struct pebs_trace *ds_read_pebs(struct pebs_tracer *tracer);
/*
* Reset the write pointer of the BTS/PEBS buffer.
*
* Returns 0 on success; -Eerrno on error
*
* tracer: the tracer handle returned from ds_request_~()
*/
extern int ds_reset_bts(struct bts_tracer *tracer);
extern int ds_reset_pebs(struct pebs_tracer *tracer);
/*
* Set the PEBS counter reset value.
*
* Returns 0 on success; -Eerrno on error
*
* tracer: the tracer handle returned from ds_request_pebs()
* counter: the index of the counter
* value: the new counter reset value
*/
extern int ds_set_pebs_reset(struct pebs_tracer *tracer,
unsigned int counter, u64 value);
/*
* Initialization
*/
struct cpuinfo_x86;
extern void __cpuinit ds_init_intel(struct cpuinfo_x86 *);
/*
* Context switch work
*/
extern void ds_switch_to(struct task_struct *prev, struct task_struct *next);
#else /* CONFIG_X86_DS */
struct cpuinfo_x86;
static inline void __cpuinit ds_init_intel(struct cpuinfo_x86 *ignored) {}
static inline void ds_switch_to(struct task_struct *prev,
struct task_struct *next) {}
#endif /* CONFIG_X86_DS */
#endif /* _ASM_X86_DS_H */

10
arch/x86/include/asm/hw_breakpoint.h
View file

@ -41,12 +41,16 @@ struct arch_hw_breakpoint {
/* Total number of available HW breakpoint registers */ /* Total number of available HW breakpoint registers */
#define HBP_NUM 4 #define HBP_NUM 4
static inline int hw_breakpoint_slots(int type)
{
return HBP_NUM;
}
struct perf_event; struct perf_event;
struct pmu; struct pmu;
extern int arch_check_va_in_userspace(unsigned long va, u8 hbp_len); extern int arch_check_bp_in_kernelspace(struct perf_event *bp);
extern int arch_validate_hwbkpt_settings(struct perf_event *bp, extern int arch_validate_hwbkpt_settings(struct perf_event *bp);
struct task_struct *tsk);
extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused, extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused,
unsigned long val, void *data); unsigned long val, void *data);

2
arch/x86/include/asm/insn.h
View file

@ -68,6 +68,8 @@ struct insn {
const insn_byte_t *next_byte; const insn_byte_t *next_byte;
}; };
#define MAX_INSN_SIZE 16
#define X86_MODRM_MOD(modrm) (((modrm) & 0xc0) >> 6) #define X86_MODRM_MOD(modrm) (((modrm) & 0xc0) >> 6)
#define X86_MODRM_REG(modrm) (((modrm) & 0x38) >> 3) #define X86_MODRM_REG(modrm) (((modrm) & 0x38) >> 3)
#define X86_MODRM_RM(modrm) ((modrm) & 0x07) #define X86_MODRM_RM(modrm) ((modrm) & 0x07)

2
arch/x86/include/asm/kprobes.h
View file

@ -24,6 +24,7 @@
#include <linux/types.h> #include <linux/types.h>
#include <linux/ptrace.h> #include <linux/ptrace.h>
#include <linux/percpu.h> #include <linux/percpu.h>
#include <asm/insn.h>
#define __ARCH_WANT_KPROBES_INSN_SLOT #define __ARCH_WANT_KPROBES_INSN_SLOT
@ -36,7 +37,6 @@ typedef u8 kprobe_opcode_t;
#define RELATIVEJUMP_SIZE 5 #define RELATIVEJUMP_SIZE 5
#define RELATIVECALL_OPCODE 0xe8 #define RELATIVECALL_OPCODE 0xe8
#define RELATIVE_ADDR_SIZE 4 #define RELATIVE_ADDR_SIZE 4
#define MAX_INSN_SIZE 16
#define MAX_STACK_SIZE 64 #define MAX_STACK_SIZE 64
#define MIN_STACK_SIZE(ADDR) \ #define MIN_STACK_SIZE(ADDR) \
(((MAX_STACK_SIZE) < (((unsigned long)current_thread_info()) + \ (((MAX_STACK_SIZE) < (((unsigned long)current_thread_info()) + \

15
arch/x86/include/asm/msr-index.h
View file

@ -71,11 +71,14 @@
#define MSR_IA32_LASTINTTOIP 0x000001de #define MSR_IA32_LASTINTTOIP 0x000001de
/* DEBUGCTLMSR bits (others vary by model): */ /* DEBUGCTLMSR bits (others vary by model): */
#define _DEBUGCTLMSR_LBR 0 /* last branch recording */ #define DEBUGCTLMSR_LBR (1UL << 0) /* last branch recording */
#define _DEBUGCTLMSR_BTF 1 /* single-step on branches */ #define DEBUGCTLMSR_BTF (1UL << 1) /* single-step on branches */
#define DEBUGCTLMSR_TR (1UL << 6)
#define DEBUGCTLMSR_LBR (1UL << _DEBUGCTLMSR_LBR) #define DEBUGCTLMSR_BTS (1UL << 7)
#define DEBUGCTLMSR_BTF (1UL << _DEBUGCTLMSR_BTF) #define DEBUGCTLMSR_BTINT (1UL << 8)
#define DEBUGCTLMSR_BTS_OFF_OS (1UL << 9)
#define DEBUGCTLMSR_BTS_OFF_USR (1UL << 10)
#define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI (1UL << 11)
#define MSR_IA32_MC0_CTL 0x00000400 #define MSR_IA32_MC0_CTL 0x00000400
#define MSR_IA32_MC0_STATUS 0x00000401 #define MSR_IA32_MC0_STATUS 0x00000401
@ -359,6 +362,8 @@
#define MSR_P4_U2L_ESCR0 0x000003b0 #define MSR_P4_U2L_ESCR0 0x000003b0
#define MSR_P4_U2L_ESCR1 0x000003b1 #define MSR_P4_U2L_ESCR1 0x000003b1
#define MSR_P4_PEBS_MATRIX_VERT 0x000003f2
/* Intel Core-based CPU performance counters */ /* Intel Core-based CPU performance counters */
#define MSR_CORE_PERF_FIXED_CTR0 0x00000309 #define MSR_CORE_PERF_FIXED_CTR0 0x00000309
#define MSR_CORE_PERF_FIXED_CTR1 0x0000030a #define MSR_CORE_PERF_FIXED_CTR1 0x0000030a

72
arch/x86/include/asm/perf_event.h
View file

@ -5,7 +5,7 @@
* Performance event hw details: * Performance event hw details:
*/ */
#define X86_PMC_MAX_GENERIC 8 #define X86_PMC_MAX_GENERIC 32
#define X86_PMC_MAX_FIXED 3 #define X86_PMC_MAX_FIXED 3
#define X86_PMC_IDX_GENERIC 0 #define X86_PMC_IDX_GENERIC 0
@ -18,39 +18,31 @@
#define MSR_ARCH_PERFMON_EVENTSEL0 0x186 #define MSR_ARCH_PERFMON_EVENTSEL0 0x186
#define MSR_ARCH_PERFMON_EVENTSEL1 0x187 #define MSR_ARCH_PERFMON_EVENTSEL1 0x187
#define ARCH_PERFMON_EVENTSEL_ENABLE (1 << 22) #define ARCH_PERFMON_EVENTSEL_EVENT 0x000000FFULL
#define ARCH_PERFMON_EVENTSEL_ANY (1 << 21) #define ARCH_PERFMON_EVENTSEL_UMASK 0x0000FF00ULL
#define ARCH_PERFMON_EVENTSEL_INT (1 << 20) #define ARCH_PERFMON_EVENTSEL_USR (1ULL << 16)
#define ARCH_PERFMON_EVENTSEL_OS (1 << 17) #define ARCH_PERFMON_EVENTSEL_OS (1ULL << 17)
#define ARCH_PERFMON_EVENTSEL_USR (1 << 16) #define ARCH_PERFMON_EVENTSEL_EDGE (1ULL << 18)
#define ARCH_PERFMON_EVENTSEL_INT (1ULL << 20)
#define ARCH_PERFMON_EVENTSEL_ANY (1ULL << 21)
#define ARCH_PERFMON_EVENTSEL_ENABLE (1ULL << 22)
#define ARCH_PERFMON_EVENTSEL_INV (1ULL << 23)
#define ARCH_PERFMON_EVENTSEL_CMASK 0xFF000000ULL
/* #define AMD64_EVENTSEL_EVENT \
* Includes eventsel and unit mask as well: (ARCH_PERFMON_EVENTSEL_EVENT | (0x0FULL << 32))
*/ #define INTEL_ARCH_EVENT_MASK \
(ARCH_PERFMON_EVENTSEL_UMASK | ARCH_PERFMON_EVENTSEL_EVENT)
#define X86_RAW_EVENT_MASK \
#define INTEL_ARCH_EVTSEL_MASK 0x000000FFULL (ARCH_PERFMON_EVENTSEL_EVENT | \
#define INTEL_ARCH_UNIT_MASK 0x0000FF00ULL ARCH_PERFMON_EVENTSEL_UMASK | \
#define INTEL_ARCH_EDGE_MASK 0x00040000ULL ARCH_PERFMON_EVENTSEL_EDGE | \
#define INTEL_ARCH_INV_MASK 0x00800000ULL ARCH_PERFMON_EVENTSEL_INV | \
#define INTEL_ARCH_CNT_MASK 0xFF000000ULL ARCH_PERFMON_EVENTSEL_CMASK)
#define INTEL_ARCH_EVENT_MASK (INTEL_ARCH_UNIT_MASK|INTEL_ARCH_EVTSEL_MASK) #define AMD64_RAW_EVENT_MASK \
(X86_RAW_EVENT_MASK | \
/* AMD64_EVENTSEL_EVENT)
* filter mask to validate fixed counter events.
* the following filters disqualify for fixed counters:
* - inv
* - edge
* - cnt-mask
* The other filters are supported by fixed counters.
* The any-thread option is supported starting with v3.
*/
#define INTEL_ARCH_FIXED_MASK \
(INTEL_ARCH_CNT_MASK| \
INTEL_ARCH_INV_MASK| \
INTEL_ARCH_EDGE_MASK|\
INTEL_ARCH_UNIT_MASK|\
INTEL_ARCH_EVENT_MASK)
#define ARCH_PERFMON_UNHALTED_CORE_CYCLES_SEL 0x3c #define ARCH_PERFMON_UNHALTED_CORE_CYCLES_SEL 0x3c
#define ARCH_PERFMON_UNHALTED_CORE_CYCLES_UMASK (0x00 << 8) #define ARCH_PERFMON_UNHALTED_CORE_CYCLES_UMASK (0x00 << 8)
@ -67,7 +59,7 @@
union cpuid10_eax { union cpuid10_eax {
struct { struct {
unsigned int version_id:8; unsigned int version_id:8;
unsigned int num_events:8; unsigned int num_counters:8;
unsigned int bit_width:8; unsigned int bit_width:8;
unsigned int mask_length:8; unsigned int mask_length:8;
} split; } split;
@ -76,7 +68,7 @@ union cpuid10_eax {
union cpuid10_edx { union cpuid10_edx {
struct { struct {
unsigned int num_events_fixed:4; unsigned int num_counters_fixed:4;
unsigned int reserved:28; unsigned int reserved:28;
} split; } split;
unsigned int full; unsigned int full;
@ -136,6 +128,18 @@ extern void perf_events_lapic_init(void);
#define PERF_EVENT_INDEX_OFFSET 0 #define PERF_EVENT_INDEX_OFFSET 0
/*
* Abuse bit 3 of the cpu eflags register to indicate proper PEBS IP fixups.
* This flag is otherwise unused and ABI specified to be 0, so nobody should
* care what we do with it.
*/
#define PERF_EFLAGS_EXACT (1UL << 3)
struct pt_regs;
extern unsigned long perf_instruction_pointer(struct pt_regs *regs);
extern unsigned long perf_misc_flags(struct pt_regs *regs);
#define perf_misc_flags(regs) perf_misc_flags(regs)
#else #else
static inline void init_hw_perf_events(void) { } static inline void init_hw_perf_events(void) { }
static inline void perf_events_lapic_init(void) { } static inline void perf_events_lapic_init(void) { }

794
arch/x86/include/asm/perf_event_p4.h Normal file
View file

@ -0,0 +1,794 @@
/*
* Netburst Perfomance Events (P4, old Xeon)
*/
#ifndef PERF_EVENT_P4_H
#define PERF_EVENT_P4_H
#include <linux/cpu.h>
#include <linux/bitops.h>
/*
* NetBurst has perfomance MSRs shared between
* threads if HT is turned on, ie for both logical
* processors (mem: in turn in Atom with HT support
* perf-MSRs are not shared and every thread has its
* own perf-MSRs set)
*/
#define ARCH_P4_TOTAL_ESCR (46)
#define ARCH_P4_RESERVED_ESCR (2) /* IQ_ESCR(0,1) not always present */
#define ARCH_P4_MAX_ESCR (ARCH_P4_TOTAL_ESCR - ARCH_P4_RESERVED_ESCR)
#define ARCH_P4_MAX_CCCR (18)
#define ARCH_P4_MAX_COUNTER (ARCH_P4_MAX_CCCR / 2)
#define P4_ESCR_EVENT_MASK 0x7e000000U
#define P4_ESCR_EVENT_SHIFT 25
#define P4_ESCR_EVENTMASK_MASK 0x01fffe00U
#define P4_ESCR_EVENTMASK_SHIFT 9
#define P4_ESCR_TAG_MASK 0x000001e0U
#define P4_ESCR_TAG_SHIFT 5
#define P4_ESCR_TAG_ENABLE 0x00000010U
#define P4_ESCR_T0_OS 0x00000008U
#define P4_ESCR_T0_USR 0x00000004U
#define P4_ESCR_T1_OS 0x00000002U
#define P4_ESCR_T1_USR 0x00000001U
#define P4_ESCR_EVENT(v) ((v) << P4_ESCR_EVENT_SHIFT)
#define P4_ESCR_EMASK(v) ((v) << P4_ESCR_EVENTMASK_SHIFT)
#define P4_ESCR_TAG(v) ((v) << P4_ESCR_TAG_SHIFT)
/* Non HT mask */
#define P4_ESCR_MASK \
(P4_ESCR_EVENT_MASK | \
P4_ESCR_EVENTMASK_MASK | \
P4_ESCR_TAG_MASK | \
P4_ESCR_TAG_ENABLE | \
P4_ESCR_T0_OS | \
P4_ESCR_T0_USR)
/* HT mask */
#define P4_ESCR_MASK_HT \
(P4_ESCR_MASK | P4_ESCR_T1_OS | P4_ESCR_T1_USR)
#define P4_CCCR_OVF 0x80000000U
#define P4_CCCR_CASCADE 0x40000000U
#define P4_CCCR_OVF_PMI_T0 0x04000000U
#define P4_CCCR_OVF_PMI_T1 0x08000000U
#define P4_CCCR_FORCE_OVF 0x02000000U
#define P4_CCCR_EDGE 0x01000000U
#define P4_CCCR_THRESHOLD_MASK 0x00f00000U
#define P4_CCCR_THRESHOLD_SHIFT 20
#define P4_CCCR_COMPLEMENT 0x00080000U
#define P4_CCCR_COMPARE 0x00040000U
#define P4_CCCR_ESCR_SELECT_MASK 0x0000e000U
#define P4_CCCR_ESCR_SELECT_SHIFT 13
#define P4_CCCR_ENABLE 0x00001000U
#define P4_CCCR_THREAD_SINGLE 0x00010000U
#define P4_CCCR_THREAD_BOTH 0x00020000U
#define P4_CCCR_THREAD_ANY 0x00030000U
#define P4_CCCR_RESERVED 0x00000fffU
#define P4_CCCR_THRESHOLD(v) ((v) << P4_CCCR_THRESHOLD_SHIFT)
#define P4_CCCR_ESEL(v) ((v) << P4_CCCR_ESCR_SELECT_SHIFT)
/* Custom bits in reerved CCCR area */
#define P4_CCCR_CACHE_OPS_MASK 0x0000003fU
/* Non HT mask */
#define P4_CCCR_MASK \
(P4_CCCR_OVF | \
P4_CCCR_CASCADE | \
P4_CCCR_OVF_PMI_T0 | \
P4_CCCR_FORCE_OVF | \
P4_CCCR_EDGE | \
P4_CCCR_THRESHOLD_MASK | \
P4_CCCR_COMPLEMENT | \
P4_CCCR_COMPARE | \
P4_CCCR_ESCR_SELECT_MASK | \
P4_CCCR_ENABLE)
/* HT mask */
#define P4_CCCR_MASK_HT (P4_CCCR_MASK | P4_CCCR_THREAD_ANY)
#define P4_GEN_ESCR_EMASK(class, name, bit) \
class##__##name = ((1 << bit) << P4_ESCR_EVENTMASK_SHIFT)
#define P4_ESCR_EMASK_BIT(class, name) class##__##name
/*
* config field is 64bit width and consists of
* HT << 63 | ESCR << 32 | CCCR
* where HT is HyperThreading bit (since ESCR
* has it reserved we may use it for own purpose)
*
* note that this is NOT the addresses of respective
* ESCR and CCCR but rather an only packed value should
* be unpacked and written to a proper addresses
*
* the base idea is to pack as much info as
* possible
*/
#define p4_config_pack_escr(v) (((u64)(v)) << 32)
#define p4_config_pack_cccr(v) (((u64)(v)) & 0xffffffffULL)
#define p4_config_unpack_escr(v) (((u64)(v)) >> 32)
#define p4_config_unpack_cccr(v) (((u64)(v)) & 0xffffffffULL)
#define p4_config_unpack_emask(v) \
({ \
u32 t = p4_config_unpack_escr((v)); \
t = t & P4_ESCR_EVENTMASK_MASK; \
t = t >> P4_ESCR_EVENTMASK_SHIFT; \
t; \
})
#define p4_config_unpack_event(v) \
({ \
u32 t = p4_config_unpack_escr((v)); \
t = t & P4_ESCR_EVENT_MASK; \
t = t >> P4_ESCR_EVENT_SHIFT; \
t; \
})
#define p4_config_unpack_cache_event(v) (((u64)(v)) & P4_CCCR_CACHE_OPS_MASK)
#define P4_CONFIG_HT_SHIFT 63
#define P4_CONFIG_HT (1ULL << P4_CONFIG_HT_SHIFT)
static inline bool p4_is_event_cascaded(u64 config)
{
u32 cccr = p4_config_unpack_cccr(config);
return !!(cccr & P4_CCCR_CASCADE);
}
static inline int p4_ht_config_thread(u64 config)
{
return !!(config & P4_CONFIG_HT);
}
static inline u64 p4_set_ht_bit(u64 config)
{
return config | P4_CONFIG_HT;
}
static inline u64 p4_clear_ht_bit(u64 config)
{
return config & ~P4_CONFIG_HT;
}
static inline int p4_ht_active(void)
{
#ifdef CONFIG_SMP
return smp_num_siblings > 1;
#endif
return 0;
}
static inline int p4_ht_thread(int cpu)
{
#ifdef CONFIG_SMP
if (smp_num_siblings == 2)
return cpu != cpumask_first(__get_cpu_var(cpu_sibling_map));
#endif
return 0;
}
static inline int p4_should_swap_ts(u64 config, int cpu)
{
return p4_ht_config_thread(config) ^ p4_ht_thread(cpu);
}
static inline u32 p4_default_cccr_conf(int cpu)
{
/*
* Note that P4_CCCR_THREAD_ANY is "required" on
* non-HT machines (on HT machines we count TS events
* regardless the state of second logical processor
*/
u32 cccr = P4_CCCR_THREAD_ANY;
if (!p4_ht_thread(cpu))
cccr |= P4_CCCR_OVF_PMI_T0;
else
cccr |= P4_CCCR_OVF_PMI_T1;
return cccr;
}
static inline u32 p4_default_escr_conf(int cpu, int exclude_os, int exclude_usr)
{
u32 escr = 0;
if (!p4_ht_thread(cpu)) {
if (!exclude_os)
escr |= P4_ESCR_T0_OS;
if (!exclude_usr)
escr |= P4_ESCR_T0_USR;
} else {
if (!exclude_os)
escr |= P4_ESCR_T1_OS;
if (!exclude_usr)
escr |= P4_ESCR_T1_USR;
}
return escr;
}
enum P4_EVENTS {
P4_EVENT_TC_DELIVER_MODE,
P4_EVENT_BPU_FETCH_REQUEST,
P4_EVENT_ITLB_REFERENCE,
P4_EVENT_MEMORY_CANCEL,
P4_EVENT_MEMORY_COMPLETE,
P4_EVENT_LOAD_PORT_REPLAY,
P4_EVENT_STORE_PORT_REPLAY,
P4_EVENT_MOB_LOAD_REPLAY,
P4_EVENT_PAGE_WALK_TYPE,
P4_EVENT_BSQ_CACHE_REFERENCE,
P4_EVENT_IOQ_ALLOCATION,
P4_EVENT_IOQ_ACTIVE_ENTRIES,
P4_EVENT_FSB_DATA_ACTIVITY,
P4_EVENT_BSQ_ALLOCATION,
P4_EVENT_BSQ_ACTIVE_ENTRIES,
P4_EVENT_SSE_INPUT_ASSIST,
P4_EVENT_PACKED_SP_UOP,
P4_EVENT_PACKED_DP_UOP,
P4_EVENT_SCALAR_SP_UOP,
P4_EVENT_SCALAR_DP_UOP,
P4_EVENT_64BIT_MMX_UOP,
P4_EVENT_128BIT_MMX_UOP,
P4_EVENT_X87_FP_UOP,
P4_EVENT_TC_MISC,
P4_EVENT_GLOBAL_POWER_EVENTS,
P4_EVENT_TC_MS_XFER,
P4_EVENT_UOP_QUEUE_WRITES,
P4_EVENT_RETIRED_MISPRED_BRANCH_TYPE,
P4_EVENT_RETIRED_BRANCH_TYPE,
P4_EVENT_RESOURCE_STALL,
P4_EVENT_WC_BUFFER,
P4_EVENT_B2B_CYCLES,
P4_EVENT_BNR,
P4_EVENT_SNOOP,
P4_EVENT_RESPONSE,
P4_EVENT_FRONT_END_EVENT,
P4_EVENT_EXECUTION_EVENT,
P4_EVENT_REPLAY_EVENT,
P4_EVENT_INSTR_RETIRED,
P4_EVENT_UOPS_RETIRED,
P4_EVENT_UOP_TYPE,
P4_EVENT_BRANCH_RETIRED,
P4_EVENT_MISPRED_BRANCH_RETIRED,
P4_EVENT_X87_ASSIST,
P4_EVENT_MACHINE_CLEAR,
P4_EVENT_INSTR_COMPLETED,
};
#define P4_OPCODE(event) event##_OPCODE
#define P4_OPCODE_ESEL(opcode) ((opcode & 0x00ff) >> 0)
#define P4_OPCODE_EVNT(opcode) ((opcode & 0xff00) >> 8)
#define P4_OPCODE_PACK(event, sel) (((event) << 8) | sel)
/*
* Comments below the event represent ESCR restriction
* for this event and counter index per ESCR
*
* MSR_P4_IQ_ESCR0 and MSR_P4_IQ_ESCR1 are available only on early
* processor builds (family 0FH, models 01H-02H). These MSRs
* are not available on later versions, so that we don't use
* them completely
*
* Also note that CCCR1 do not have P4_CCCR_ENABLE bit properly
* working so that we should not use this CCCR and respective
* counter as result
*/
enum P4_EVENT_OPCODES {
P4_OPCODE(P4_EVENT_TC_DELIVER_MODE) = P4_OPCODE_PACK(0x01, 0x01),
/*
* MSR_P4_TC_ESCR0: 4, 5
* MSR_P4_TC_ESCR1: 6, 7
*/
P4_OPCODE(P4_EVENT_BPU_FETCH_REQUEST) = P4_OPCODE_PACK(0x03, 0x00),
/*
* MSR_P4_BPU_ESCR0: 0, 1
* MSR_P4_BPU_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_ITLB_REFERENCE) = P4_OPCODE_PACK(0x18, 0x03),
/*
* MSR_P4_ITLB_ESCR0: 0, 1
* MSR_P4_ITLB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_MEMORY_CANCEL) = P4_OPCODE_PACK(0x02, 0x05),
/*
* MSR_P4_DAC_ESCR0: 8, 9
* MSR_P4_DAC_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_MEMORY_COMPLETE) = P4_OPCODE_PACK(0x08, 0x02),
/*
* MSR_P4_SAAT_ESCR0: 8, 9
* MSR_P4_SAAT_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_LOAD_PORT_REPLAY) = P4_OPCODE_PACK(0x04, 0x02),
/*
* MSR_P4_SAAT_ESCR0: 8, 9
* MSR_P4_SAAT_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_STORE_PORT_REPLAY) = P4_OPCODE_PACK(0x05, 0x02),
/*
* MSR_P4_SAAT_ESCR0: 8, 9
* MSR_P4_SAAT_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_MOB_LOAD_REPLAY) = P4_OPCODE_PACK(0x03, 0x02),
/*
* MSR_P4_MOB_ESCR0: 0, 1
* MSR_P4_MOB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_PAGE_WALK_TYPE) = P4_OPCODE_PACK(0x01, 0x04),
/*
* MSR_P4_PMH_ESCR0: 0, 1
* MSR_P4_PMH_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_BSQ_CACHE_REFERENCE) = P4_OPCODE_PACK(0x0c, 0x07),
/*
* MSR_P4_BSU_ESCR0: 0, 1
* MSR_P4_BSU_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_IOQ_ALLOCATION) = P4_OPCODE_PACK(0x03, 0x06),
/*
* MSR_P4_FSB_ESCR0: 0, 1
* MSR_P4_FSB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_IOQ_ACTIVE_ENTRIES) = P4_OPCODE_PACK(0x1a, 0x06),
/*
* MSR_P4_FSB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_FSB_DATA_ACTIVITY) = P4_OPCODE_PACK(0x17, 0x06),
/*
* MSR_P4_FSB_ESCR0: 0, 1
* MSR_P4_FSB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_BSQ_ALLOCATION) = P4_OPCODE_PACK(0x05, 0x07),
/*
* MSR_P4_BSU_ESCR0: 0, 1
*/
P4_OPCODE(P4_EVENT_BSQ_ACTIVE_ENTRIES) = P4_OPCODE_PACK(0x06, 0x07),
/*
* NOTE: no ESCR name in docs, it's guessed
* MSR_P4_BSU_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_SSE_INPUT_ASSIST) = P4_OPCODE_PACK(0x34, 0x01),
/*
* MSR_P4_FIRM_ESCR0: 8, 9
* MSR_P4_FIRM_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_PACKED_SP_UOP) = P4_OPCODE_PACK(0x08, 0x01),
/*
* MSR_P4_FIRM_ESCR0: 8, 9
* MSR_P4_FIRM_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_PACKED_DP_UOP) = P4_OPCODE_PACK(0x0c, 0x01),
/*
* MSR_P4_FIRM_ESCR0: 8, 9
* MSR_P4_FIRM_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_SCALAR_SP_UOP) = P4_OPCODE_PACK(0x0a, 0x01),
/*
* MSR_P4_FIRM_ESCR0: 8, 9
* MSR_P4_FIRM_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_SCALAR_DP_UOP) = P4_OPCODE_PACK(0x0e, 0x01),
/*
* MSR_P4_FIRM_ESCR0: 8, 9
* MSR_P4_FIRM_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_64BIT_MMX_UOP) = P4_OPCODE_PACK(0x02, 0x01),
/*
* MSR_P4_FIRM_ESCR0: 8, 9
* MSR_P4_FIRM_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_128BIT_MMX_UOP) = P4_OPCODE_PACK(0x1a, 0x01),
/*
* MSR_P4_FIRM_ESCR0: 8, 9
* MSR_P4_FIRM_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_X87_FP_UOP) = P4_OPCODE_PACK(0x04, 0x01),
/*
* MSR_P4_FIRM_ESCR0: 8, 9
* MSR_P4_FIRM_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_TC_MISC) = P4_OPCODE_PACK(0x06, 0x01),
/*
* MSR_P4_TC_ESCR0: 4, 5
* MSR_P4_TC_ESCR1: 6, 7
*/
P4_OPCODE(P4_EVENT_GLOBAL_POWER_EVENTS) = P4_OPCODE_PACK(0x13, 0x06),
/*
* MSR_P4_FSB_ESCR0: 0, 1
* MSR_P4_FSB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_TC_MS_XFER) = P4_OPCODE_PACK(0x05, 0x00),
/*
* MSR_P4_MS_ESCR0: 4, 5
* MSR_P4_MS_ESCR1: 6, 7
*/
P4_OPCODE(P4_EVENT_UOP_QUEUE_WRITES) = P4_OPCODE_PACK(0x09, 0x00),
/*
* MSR_P4_MS_ESCR0: 4, 5
* MSR_P4_MS_ESCR1: 6, 7
*/
P4_OPCODE(P4_EVENT_RETIRED_MISPRED_BRANCH_TYPE) = P4_OPCODE_PACK(0x05, 0x02),
/*
* MSR_P4_TBPU_ESCR0: 4, 5
* MSR_P4_TBPU_ESCR1: 6, 7
*/
P4_OPCODE(P4_EVENT_RETIRED_BRANCH_TYPE) = P4_OPCODE_PACK(0x04, 0x02),
/*
* MSR_P4_TBPU_ESCR0: 4, 5
* MSR_P4_TBPU_ESCR1: 6, 7
*/
P4_OPCODE(P4_EVENT_RESOURCE_STALL) = P4_OPCODE_PACK(0x01, 0x01),
/*
* MSR_P4_ALF_ESCR0: 12, 13, 16
* MSR_P4_ALF_ESCR1: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_WC_BUFFER) = P4_OPCODE_PACK(0x05, 0x05),
/*
* MSR_P4_DAC_ESCR0: 8, 9
* MSR_P4_DAC_ESCR1: 10, 11
*/
P4_OPCODE(P4_EVENT_B2B_CYCLES) = P4_OPCODE_PACK(0x16, 0x03),
/*
* MSR_P4_FSB_ESCR0: 0, 1
* MSR_P4_FSB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_BNR) = P4_OPCODE_PACK(0x08, 0x03),
/*
* MSR_P4_FSB_ESCR0: 0, 1
* MSR_P4_FSB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_SNOOP) = P4_OPCODE_PACK(0x06, 0x03),
/*
* MSR_P4_FSB_ESCR0: 0, 1
* MSR_P4_FSB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_RESPONSE) = P4_OPCODE_PACK(0x04, 0x03),
/*
* MSR_P4_FSB_ESCR0: 0, 1
* MSR_P4_FSB_ESCR1: 2, 3
*/
P4_OPCODE(P4_EVENT_FRONT_END_EVENT) = P4_OPCODE_PACK(0x08, 0x05),
/*
* MSR_P4_CRU_ESCR2: 12, 13, 16
* MSR_P4_CRU_ESCR3: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_EXECUTION_EVENT) = P4_OPCODE_PACK(0x0c, 0x05),
/*
* MSR_P4_CRU_ESCR2: 12, 13, 16
* MSR_P4_CRU_ESCR3: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_REPLAY_EVENT) = P4_OPCODE_PACK(0x09, 0x05),
/*
* MSR_P4_CRU_ESCR2: 12, 13, 16
* MSR_P4_CRU_ESCR3: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_INSTR_RETIRED) = P4_OPCODE_PACK(0x02, 0x04),
/*
* MSR_P4_CRU_ESCR0: 12, 13, 16
* MSR_P4_CRU_ESCR1: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_UOPS_RETIRED) = P4_OPCODE_PACK(0x01, 0x04),
/*
* MSR_P4_CRU_ESCR0: 12, 13, 16
* MSR_P4_CRU_ESCR1: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_UOP_TYPE) = P4_OPCODE_PACK(0x02, 0x02),
/*
* MSR_P4_RAT_ESCR0: 12, 13, 16
* MSR_P4_RAT_ESCR1: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_BRANCH_RETIRED) = P4_OPCODE_PACK(0x06, 0x05),
/*
* MSR_P4_CRU_ESCR2: 12, 13, 16
* MSR_P4_CRU_ESCR3: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_MISPRED_BRANCH_RETIRED) = P4_OPCODE_PACK(0x03, 0x04),
/*
* MSR_P4_CRU_ESCR0: 12, 13, 16
* MSR_P4_CRU_ESCR1: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_X87_ASSIST) = P4_OPCODE_PACK(0x03, 0x05),
/*
* MSR_P4_CRU_ESCR2: 12, 13, 16
* MSR_P4_CRU_ESCR3: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_MACHINE_CLEAR) = P4_OPCODE_PACK(0x02, 0x05),
/*
* MSR_P4_CRU_ESCR2: 12, 13, 16
* MSR_P4_CRU_ESCR3: 14, 15, 17
*/
P4_OPCODE(P4_EVENT_INSTR_COMPLETED) = P4_OPCODE_PACK(0x07, 0x04),
/*
* MSR_P4_CRU_ESCR0: 12, 13, 16
* MSR_P4_CRU_ESCR1: 14, 15, 17
*/
};
/*
* a caller should use P4_ESCR_EMASK_NAME helper to
* pick the EventMask needed, for example
*
* P4_ESCR_EMASK_NAME(P4_EVENT_TC_DELIVER_MODE, DD)
*/
enum P4_ESCR_EMASKS {
P4_GEN_ESCR_EMASK(P4_EVENT_TC_DELIVER_MODE, DD, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_TC_DELIVER_MODE, DB, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_TC_DELIVER_MODE, DI, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_TC_DELIVER_MODE, BD, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_TC_DELIVER_MODE, BB, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_TC_DELIVER_MODE, BI, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_TC_DELIVER_MODE, ID, 6),
P4_GEN_ESCR_EMASK(P4_EVENT_BPU_FETCH_REQUEST, TCMISS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_ITLB_REFERENCE, HIT, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_ITLB_REFERENCE, MISS, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_ITLB_REFERENCE, HIT_UK, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_MEMORY_CANCEL, ST_RB_FULL, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_MEMORY_CANCEL, 64K_CONF, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_MEMORY_COMPLETE, LSC, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_MEMORY_COMPLETE, SSC, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_LOAD_PORT_REPLAY, SPLIT_LD, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_STORE_PORT_REPLAY, SPLIT_ST, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_MOB_LOAD_REPLAY, NO_STA, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_MOB_LOAD_REPLAY, NO_STD, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_MOB_LOAD_REPLAY, PARTIAL_DATA, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_MOB_LOAD_REPLAY, UNALGN_ADDR, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_PAGE_WALK_TYPE, DTMISS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_PAGE_WALK_TYPE, ITMISS, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, RD_2ndL_HITS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, RD_2ndL_HITE, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, RD_2ndL_HITM, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, RD_3rdL_HITS, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, RD_3rdL_HITE, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, RD_3rdL_HITM, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, RD_2ndL_MISS, 8),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, RD_3rdL_MISS, 9),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_CACHE_REFERENCE, WR_2ndL_MISS, 10),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, DEFAULT, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, ALL_READ, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, ALL_WRITE, 6),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, MEM_UC, 7),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, MEM_WC, 8),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, MEM_WT, 9),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, MEM_WP, 10),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, MEM_WB, 11),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, OWN, 13),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, OTHER, 14),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ALLOCATION, PREFETCH, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, DEFAULT, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, ALL_READ, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, ALL_WRITE, 6),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, MEM_UC, 7),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, MEM_WC, 8),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, MEM_WT, 9),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, MEM_WP, 10),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, MEM_WB, 11),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, OWN, 13),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, OTHER, 14),
P4_GEN_ESCR_EMASK(P4_EVENT_IOQ_ACTIVE_ENTRIES, PREFETCH, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_FSB_DATA_ACTIVITY, DRDY_DRV, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_FSB_DATA_ACTIVITY, DRDY_OWN, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_FSB_DATA_ACTIVITY, DRDY_OTHER, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_FSB_DATA_ACTIVITY, DBSY_DRV, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_FSB_DATA_ACTIVITY, DBSY_OWN, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_FSB_DATA_ACTIVITY, DBSY_OTHER, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_TYPE0, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_TYPE1, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_LEN0, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_LEN1, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_IO_TYPE, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_LOCK_TYPE, 6),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_CACHE_TYPE, 7),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_SPLIT_TYPE, 8),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_DEM_TYPE, 9),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, REQ_ORD_TYPE, 10),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, MEM_TYPE0, 11),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, MEM_TYPE1, 12),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ALLOCATION, MEM_TYPE2, 13),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_TYPE0, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_TYPE1, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_LEN0, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_LEN1, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_IO_TYPE, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_LOCK_TYPE, 6),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_CACHE_TYPE, 7),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_SPLIT_TYPE, 8),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_DEM_TYPE, 9),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, REQ_ORD_TYPE, 10),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, MEM_TYPE0, 11),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, MEM_TYPE1, 12),
P4_GEN_ESCR_EMASK(P4_EVENT_BSQ_ACTIVE_ENTRIES, MEM_TYPE2, 13),
P4_GEN_ESCR_EMASK(P4_EVENT_SSE_INPUT_ASSIST, ALL, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_PACKED_SP_UOP, ALL, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_PACKED_DP_UOP, ALL, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_SCALAR_SP_UOP, ALL, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_SCALAR_DP_UOP, ALL, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_64BIT_MMX_UOP, ALL, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_128BIT_MMX_UOP, ALL, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_X87_FP_UOP, ALL, 15),
P4_GEN_ESCR_EMASK(P4_EVENT_TC_MISC, FLUSH, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_GLOBAL_POWER_EVENTS, RUNNING, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_TC_MS_XFER, CISC, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_UOP_QUEUE_WRITES, FROM_TC_BUILD, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_UOP_QUEUE_WRITES, FROM_TC_DELIVER, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_UOP_QUEUE_WRITES, FROM_ROM, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_RETIRED_MISPRED_BRANCH_TYPE, CONDITIONAL, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_RETIRED_MISPRED_BRANCH_TYPE, CALL, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_RETIRED_MISPRED_BRANCH_TYPE, RETURN, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_RETIRED_MISPRED_BRANCH_TYPE, INDIRECT, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_RETIRED_BRANCH_TYPE, CONDITIONAL, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_RETIRED_BRANCH_TYPE, CALL, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_RETIRED_BRANCH_TYPE, RETURN, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_RETIRED_BRANCH_TYPE, INDIRECT, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_RESOURCE_STALL, SBFULL, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_WC_BUFFER, WCB_EVICTS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_WC_BUFFER, WCB_FULL_EVICTS, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_FRONT_END_EVENT, NBOGUS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_FRONT_END_EVENT, BOGUS, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_EXECUTION_EVENT, NBOGUS0, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_EXECUTION_EVENT, NBOGUS1, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_EXECUTION_EVENT, NBOGUS2, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_EXECUTION_EVENT, NBOGUS3, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_EXECUTION_EVENT, BOGUS0, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_EXECUTION_EVENT, BOGUS1, 5),
P4_GEN_ESCR_EMASK(P4_EVENT_EXECUTION_EVENT, BOGUS2, 6),
P4_GEN_ESCR_EMASK(P4_EVENT_EXECUTION_EVENT, BOGUS3, 7),
P4_GEN_ESCR_EMASK(P4_EVENT_REPLAY_EVENT, NBOGUS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_REPLAY_EVENT, BOGUS, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_INSTR_RETIRED, NBOGUSNTAG, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_INSTR_RETIRED, NBOGUSTAG, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_INSTR_RETIRED, BOGUSNTAG, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_INSTR_RETIRED, BOGUSTAG, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_UOPS_RETIRED, NBOGUS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_UOPS_RETIRED, BOGUS, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_UOP_TYPE, TAGLOADS, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_UOP_TYPE, TAGSTORES, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_BRANCH_RETIRED, MMNP, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_BRANCH_RETIRED, MMNM, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_BRANCH_RETIRED, MMTP, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_BRANCH_RETIRED, MMTM, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_MISPRED_BRANCH_RETIRED, NBOGUS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_X87_ASSIST, FPSU, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_X87_ASSIST, FPSO, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_X87_ASSIST, POAO, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_X87_ASSIST, POAU, 3),
P4_GEN_ESCR_EMASK(P4_EVENT_X87_ASSIST, PREA, 4),
P4_GEN_ESCR_EMASK(P4_EVENT_MACHINE_CLEAR, CLEAR, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_MACHINE_CLEAR, MOCLEAR, 1),
P4_GEN_ESCR_EMASK(P4_EVENT_MACHINE_CLEAR, SMCLEAR, 2),
P4_GEN_ESCR_EMASK(P4_EVENT_INSTR_COMPLETED, NBOGUS, 0),
P4_GEN_ESCR_EMASK(P4_EVENT_INSTR_COMPLETED, BOGUS, 1),
};
/* P4 PEBS: stale for a while */
#define P4_PEBS_METRIC_MASK 0x00001fffU
#define P4_PEBS_UOB_TAG 0x01000000U
#define P4_PEBS_ENABLE 0x02000000U
/* Replay metrics for MSR_IA32_PEBS_ENABLE and MSR_P4_PEBS_MATRIX_VERT */
#define P4_PEBS__1stl_cache_load_miss_retired 0x3000001
#define P4_PEBS__2ndl_cache_load_miss_retired 0x3000002
#define P4_PEBS__dtlb_load_miss_retired 0x3000004
#define P4_PEBS__dtlb_store_miss_retired 0x3000004
#define P4_PEBS__dtlb_all_miss_retired 0x3000004
#define P4_PEBS__tagged_mispred_branch 0x3018000
#define P4_PEBS__mob_load_replay_retired 0x3000200
#define P4_PEBS__split_load_retired 0x3000400
#define P4_PEBS__split_store_retired 0x3000400
#define P4_VERT__1stl_cache_load_miss_retired 0x0000001
#define P4_VERT__2ndl_cache_load_miss_retired 0x0000001
#define P4_VERT__dtlb_load_miss_retired 0x0000001
#define P4_VERT__dtlb_store_miss_retired 0x0000002
#define P4_VERT__dtlb_all_miss_retired 0x0000003
#define P4_VERT__tagged_mispred_branch 0x0000010
#define P4_VERT__mob_load_replay_retired 0x0000001
#define P4_VERT__split_load_retired 0x0000001
#define P4_VERT__split_store_retired 0x0000002
enum P4_CACHE_EVENTS {
P4_CACHE__NONE,
P4_CACHE__1stl_cache_load_miss_retired,
P4_CACHE__2ndl_cache_load_miss_retired,
P4_CACHE__dtlb_load_miss_retired,
P4_CACHE__dtlb_store_miss_retired,
P4_CACHE__itlb_reference_hit,
P4_CACHE__itlb_reference_miss,
P4_CACHE__MAX
};
#endif /* PERF_EVENT_P4_H */

33
arch/x86/include/asm/processor.h
View file

@ -21,7 +21,6 @@ struct mm_struct;
#include <asm/msr.h> #include <asm/msr.h>
#include <asm/desc_defs.h> #include <asm/desc_defs.h>
#include <asm/nops.h> #include <asm/nops.h>
#include <asm/ds.h>
#include <linux/personality.h> #include <linux/personality.h>
#include <linux/cpumask.h> #include <linux/cpumask.h>
@ -29,6 +28,7 @@ struct mm_struct;
#include <linux/threads.h> #include <linux/threads.h>
#include <linux/math64.h> #include <linux/math64.h>
#include <linux/init.h> #include <linux/init.h>
#include <linux/err.h>
#define HBP_NUM 4 #define HBP_NUM 4
/* /*
@ -473,10 +473,6 @@ struct thread_struct {
unsigned long iopl; unsigned long iopl;
/* Max allowed port in the bitmap, in bytes: */ /* Max allowed port in the bitmap, in bytes: */
unsigned io_bitmap_max; unsigned io_bitmap_max;
/* MSR_IA32_DEBUGCTLMSR value to switch in if TIF_DEBUGCTLMSR is set. */
unsigned long debugctlmsr;
/* Debug Store context; see asm/ds.h */
struct ds_context *ds_ctx;
}; };
static inline unsigned long native_get_debugreg(int regno) static inline unsigned long native_get_debugreg(int regno)
@ -814,21 +810,6 @@ static inline unsigned long get_debugctlmsr(void)
return debugctlmsr; return debugctlmsr;
} }
static inline unsigned long get_debugctlmsr_on_cpu(int cpu)
{
u64 debugctlmsr = 0;
u32 val1, val2;
#ifndef CONFIG_X86_DEBUGCTLMSR
if (boot_cpu_data.x86 < 6)
return 0;
#endif
rdmsr_on_cpu(cpu, MSR_IA32_DEBUGCTLMSR, &val1, &val2);
debugctlmsr = val1 | ((u64)val2 << 32);
return debugctlmsr;
}
static inline void update_debugctlmsr(unsigned long debugctlmsr) static inline void update_debugctlmsr(unsigned long debugctlmsr)
{ {
#ifndef CONFIG_X86_DEBUGCTLMSR #ifndef CONFIG_X86_DEBUGCTLMSR
@ -838,18 +819,6 @@ static inline void update_debugctlmsr(unsigned long debugctlmsr)
wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctlmsr); wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctlmsr);
} }
static inline void update_debugctlmsr_on_cpu(int cpu,
unsigned long debugctlmsr)
{
#ifndef CONFIG_X86_DEBUGCTLMSR
if (boot_cpu_data.x86 < 6)
return;
#endif
wrmsr_on_cpu(cpu, MSR_IA32_DEBUGCTLMSR,
(u32)((u64)debugctlmsr),
(u32)((u64)debugctlmsr >> 32));
}
/* /*
* from system description table in BIOS. Mostly for MCA use, but * from system description table in BIOS. Mostly for MCA use, but
* others may find it useful: * others may find it useful:

57
arch/x86/include/asm/ptrace-abi.h
View file

@ -82,61 +82,6 @@
#ifndef __ASSEMBLY__ #ifndef __ASSEMBLY__
#include <linux/types.h> #include <linux/types.h>
#endif
/* configuration/status structure used in PTRACE_BTS_CONFIG and
PTRACE_BTS_STATUS commands.
*/
struct ptrace_bts_config {
/* requested or actual size of BTS buffer in bytes */
__u32 size;
/* bitmask of below flags */
__u32 flags;
/* buffer overflow signal */
__u32 signal;
/* actual size of bts_struct in bytes */
__u32 bts_size;
};
#endif /* __ASSEMBLY__ */
#define PTRACE_BTS_O_TRACE 0x1 /* branch trace */
#define PTRACE_BTS_O_SCHED 0x2 /* scheduling events w/ jiffies */
#define PTRACE_BTS_O_SIGNAL 0x4 /* send SIG<signal> on buffer overflow
instead of wrapping around */
#define PTRACE_BTS_O_ALLOC 0x8 /* (re)allocate buffer */
#define PTRACE_BTS_CONFIG 40
/* Configure branch trace recording.
ADDR points to a struct ptrace_bts_config.
DATA gives the size of that buffer.
A new buffer is allocated, if requested in the flags.
An overflow signal may only be requested for new buffers.
Returns the number of bytes read.
*/
#define PTRACE_BTS_STATUS 41
/* Return the current configuration in a struct ptrace_bts_config
pointed to by ADDR; DATA gives the size of that buffer.
Returns the number of bytes written.
*/
#define PTRACE_BTS_SIZE 42
/* Return the number of available BTS records for draining.
DATA and ADDR are ignored.
*/
#define PTRACE_BTS_GET 43
/* Get a single BTS record.
DATA defines the index into the BTS array, where 0 is the newest
entry, and higher indices refer to older entries.
ADDR is pointing to struct bts_struct (see asm/ds.h).
*/
#define PTRACE_BTS_CLEAR 44
/* Clear the BTS buffer.
DATA and ADDR are ignored.
*/
#define PTRACE_BTS_DRAIN 45
/* Read all available BTS records and clear the buffer.
ADDR points to an array of struct bts_struct.
DATA gives the size of that buffer.
BTS records are read from oldest to newest.
Returns number of BTS records drained.
*/
#endif /* _ASM_X86_PTRACE_ABI_H */ #endif /* _ASM_X86_PTRACE_ABI_H */

6
arch/x86/include/asm/ptrace.h
View file

@ -289,12 +289,6 @@ extern int do_get_thread_area(struct task_struct *p, int idx,
extern int do_set_thread_area(struct task_struct *p, int idx, extern int do_set_thread_area(struct task_struct *p, int idx,
struct user_desc __user *info, int can_allocate); struct user_desc __user *info, int can_allocate);
#ifdef CONFIG_X86_PTRACE_BTS
extern void ptrace_bts_untrace(struct task_struct *tsk);
#define arch_ptrace_untrace(tsk) ptrace_bts_untrace(tsk)
#endif /* CONFIG_X86_PTRACE_BTS */
#endif /* __KERNEL__ */ #endif /* __KERNEL__ */
#endif /* !__ASSEMBLY__ */ #endif /* !__ASSEMBLY__ */

8
arch/x86/include/asm/thread_info.h
View file

@ -92,8 +92,7 @@ struct thread_info {
#define TIF_IO_BITMAP 22 /* uses I/O bitmap */ #define TIF_IO_BITMAP 22 /* uses I/O bitmap */
#define TIF_FREEZE 23 /* is freezing for suspend */ #define TIF_FREEZE 23 /* is freezing for suspend */
#define TIF_FORCED_TF 24 /* true if TF in eflags artificially */ #define TIF_FORCED_TF 24 /* true if TF in eflags artificially */
#define TIF_DEBUGCTLMSR 25 /* uses thread_struct.debugctlmsr */ #define TIF_BLOCKSTEP 25 /* set when we want DEBUGCTLMSR_BTF */
#define TIF_DS_AREA_MSR 26 /* uses thread_struct.ds_area_msr */
#define TIF_LAZY_MMU_UPDATES 27 /* task is updating the mmu lazily */ #define TIF_LAZY_MMU_UPDATES 27 /* task is updating the mmu lazily */
#define TIF_SYSCALL_TRACEPOINT 28 /* syscall tracepoint instrumentation */ #define TIF_SYSCALL_TRACEPOINT 28 /* syscall tracepoint instrumentation */
@ -115,8 +114,7 @@ struct thread_info {
#define _TIF_IO_BITMAP (1 << TIF_IO_BITMAP) #define _TIF_IO_BITMAP (1 << TIF_IO_BITMAP)
#define _TIF_FREEZE (1 << TIF_FREEZE) #define _TIF_FREEZE (1 << TIF_FREEZE)
#define _TIF_FORCED_TF (1 << TIF_FORCED_TF) #define _TIF_FORCED_TF (1 << TIF_FORCED_TF)
#define _TIF_DEBUGCTLMSR (1 << TIF_DEBUGCTLMSR) #define _TIF_BLOCKSTEP (1 << TIF_BLOCKSTEP)
#define _TIF_DS_AREA_MSR (1 << TIF_DS_AREA_MSR)
#define _TIF_LAZY_MMU_UPDATES (1 << TIF_LAZY_MMU_UPDATES) #define _TIF_LAZY_MMU_UPDATES (1 << TIF_LAZY_MMU_UPDATES)
#define _TIF_SYSCALL_TRACEPOINT (1 << TIF_SYSCALL_TRACEPOINT) #define _TIF_SYSCALL_TRACEPOINT (1 << TIF_SYSCALL_TRACEPOINT)
@ -147,7 +145,7 @@ struct thread_info {
/* flags to check in __switch_to() */ /* flags to check in __switch_to() */
#define _TIF_WORK_CTXSW \ #define _TIF_WORK_CTXSW \
(_TIF_IO_BITMAP|_TIF_DEBUGCTLMSR|_TIF_DS_AREA_MSR|_TIF_NOTSC) (_TIF_IO_BITMAP|_TIF_NOTSC|_TIF_BLOCKSTEP)
#define _TIF_WORK_CTXSW_PREV (_TIF_WORK_CTXSW|_TIF_USER_RETURN_NOTIFY) #define _TIF_WORK_CTXSW_PREV (_TIF_WORK_CTXSW|_TIF_USER_RETURN_NOTIFY)
#define _TIF_WORK_CTXSW_NEXT (_TIF_WORK_CTXSW|_TIF_DEBUG) #define _TIF_WORK_CTXSW_NEXT (_TIF_WORK_CTXSW|_TIF_DEBUG)

2
arch/x86/kernel/Makefile
View file

@ -47,8 +47,6 @@ obj-$(CONFIG_X86_TRAMPOLINE) += trampoline.o
obj-y += process.o obj-y += process.o
obj-y += i387.o xsave.o obj-y += i387.o xsave.o
obj-y += ptrace.o obj-y += ptrace.o
obj-$(CONFIG_X86_DS) += ds.o
obj-$(CONFIG_X86_DS_SELFTEST) += ds_selftest.o
obj-$(CONFIG_X86_32) += tls.o obj-$(CONFIG_X86_32) += tls.o
obj-$(CONFIG_IA32_EMULATION) += tls.o obj-$(CONFIG_IA32_EMULATION) += tls.o
obj-y += step.o obj-y += step.o

2
arch/x86/kernel/cpu/intel.c
View file

@ -12,7 +12,6 @@
#include <asm/processor.h> #include <asm/processor.h>
#include <asm/pgtable.h> #include <asm/pgtable.h>
#include <asm/msr.h> #include <asm/msr.h>
#include <asm/ds.h>
#include <asm/bugs.h> #include <asm/bugs.h>
#include <asm/cpu.h> #include <asm/cpu.h>
@ -388,7 +387,6 @@ static void __cpuinit init_intel(struct cpuinfo_x86 *c)
set_cpu_cap(c, X86_FEATURE_BTS); set_cpu_cap(c, X86_FEATURE_BTS);
if (!(l1 & (1<<12))) if (!(l1 & (1<<12)))
set_cpu_cap(c, X86_FEATURE_PEBS); set_cpu_cap(c, X86_FEATURE_PEBS);
ds_init_intel(c);
} }
if (c->x86 == 6 && c->x86_model == 29 && cpu_has_clflush) if (c->x86 == 6 && c->x86_model == 29 && cpu_has_clflush)

823
arch/x86/kernel/cpu/perf_event.c
View file

File diff suppressed because it is too large Load diff

42
arch/x86/kernel/cpu/perf_event_amd.c
View file

@ -2,7 +2,7 @@
static DEFINE_RAW_SPINLOCK(amd_nb_lock); static DEFINE_RAW_SPINLOCK(amd_nb_lock);
static __initconst u64 amd_hw_cache_event_ids static __initconst const u64 amd_hw_cache_event_ids
[PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_MAX]
[PERF_COUNT_HW_CACHE_OP_MAX] [PERF_COUNT_HW_CACHE_OP_MAX]
[PERF_COUNT_HW_CACHE_RESULT_MAX] = [PERF_COUNT_HW_CACHE_RESULT_MAX] =
@ -111,22 +111,19 @@ static u64 amd_pmu_event_map(int hw_event)
return amd_perfmon_event_map[hw_event]; return amd_perfmon_event_map[hw_event];
} }
static u64 amd_pmu_raw_event(u64 hw_event) static int amd_pmu_hw_config(struct perf_event *event)
{ {
#define K7_EVNTSEL_EVENT_MASK 0xF000000FFULL int ret = x86_pmu_hw_config(event);
#define K7_EVNTSEL_UNIT_MASK 0x00000FF00ULL
#define K7_EVNTSEL_EDGE_MASK 0x000040000ULL
#define K7_EVNTSEL_INV_MASK 0x000800000ULL
#define K7_EVNTSEL_REG_MASK 0x0FF000000ULL
#define K7_EVNTSEL_MASK \ if (ret)
(K7_EVNTSEL_EVENT_MASK | \ return ret;
K7_EVNTSEL_UNIT_MASK | \
K7_EVNTSEL_EDGE_MASK | \
K7_EVNTSEL_INV_MASK | \
K7_EVNTSEL_REG_MASK)
return hw_event & K7_EVNTSEL_MASK; if (event->attr.type != PERF_TYPE_RAW)
return 0;
event->hw.config |= event->attr.config & AMD64_RAW_EVENT_MASK;
return 0;
} }
/* /*
@ -165,7 +162,7 @@ static void amd_put_event_constraints(struct cpu_hw_events *cpuc,
* be removed on one CPU at a time AND PMU is disabled * be removed on one CPU at a time AND PMU is disabled
* when we come here * when we come here
*/ */
for (i = 0; i < x86_pmu.num_events; i++) { for (i = 0; i < x86_pmu.num_counters; i++) {
if (nb->owners[i] == event) { if (nb->owners[i] == event) {
cmpxchg(nb->owners+i, event, NULL); cmpxchg(nb->owners+i, event, NULL);
break; break;
@ -215,7 +212,7 @@ amd_get_event_constraints(struct cpu_hw_events *cpuc, struct perf_event *event)
struct hw_perf_event *hwc = &event->hw; struct hw_perf_event *hwc = &event->hw;
struct amd_nb *nb = cpuc->amd_nb; struct amd_nb *nb = cpuc->amd_nb;
struct perf_event *old = NULL; struct perf_event *old = NULL;
int max = x86_pmu.num_events; int max = x86_pmu.num_counters;
int i, j, k = -1; int i, j, k = -1;
/* /*
@ -293,7 +290,7 @@ static struct amd_nb *amd_alloc_nb(int cpu, int nb_id)
/* /*
* initialize all possible NB constraints * initialize all possible NB constraints
*/ */
for (i = 0; i < x86_pmu.num_events; i++) { for (i = 0; i < x86_pmu.num_counters; i++) {
__set_bit(i, nb->event_constraints[i].idxmsk); __set_bit(i, nb->event_constraints[i].idxmsk);
nb->event_constraints[i].weight = 1; nb->event_constraints[i].weight = 1;
} }
@ -371,21 +368,22 @@ static void amd_pmu_cpu_dead(int cpu)
raw_spin_unlock(&amd_nb_lock); raw_spin_unlock(&amd_nb_lock);
} }
static __initconst struct x86_pmu amd_pmu = { static __initconst const struct x86_pmu amd_pmu = {
.name = "AMD", .name = "AMD",
.handle_irq = x86_pmu_handle_irq, .handle_irq = x86_pmu_handle_irq,
.disable_all = x86_pmu_disable_all, .disable_all = x86_pmu_disable_all,
.enable_all = x86_pmu_enable_all, .enable_all = x86_pmu_enable_all,
.enable = x86_pmu_enable_event, .enable = x86_pmu_enable_event,
.disable = x86_pmu_disable_event, .disable = x86_pmu_disable_event,
.hw_config = amd_pmu_hw_config,
.schedule_events = x86_schedule_events,
.eventsel = MSR_K7_EVNTSEL0, .eventsel = MSR_K7_EVNTSEL0,
.perfctr = MSR_K7_PERFCTR0, .perfctr = MSR_K7_PERFCTR0,
.event_map = amd_pmu_event_map, .event_map = amd_pmu_event_map,
.raw_event = amd_pmu_raw_event,
.max_events = ARRAY_SIZE(amd_perfmon_event_map), .max_events = ARRAY_SIZE(amd_perfmon_event_map),
.num_events = 4, .num_counters = 4,
.event_bits = 48, .cntval_bits = 48,
.event_mask = (1ULL << 48) - 1, .cntval_mask = (1ULL << 48) - 1,
.apic = 1, .apic = 1,
/* use highest bit to detect overflow */ /* use highest bit to detect overflow */
.max_period = (1ULL << 47) - 1, .max_period = (1ULL << 47) - 1,

355
arch/x86/kernel/cpu/perf_event_intel.c
View file

@ -88,7 +88,7 @@ static u64 intel_pmu_event_map(int hw_event)
return intel_perfmon_event_map[hw_event]; return intel_perfmon_event_map[hw_event];
} }
static __initconst u64 westmere_hw_cache_event_ids static __initconst const u64 westmere_hw_cache_event_ids
[PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_MAX]
[PERF_COUNT_HW_CACHE_OP_MAX] [PERF_COUNT_HW_CACHE_OP_MAX]
[PERF_COUNT_HW_CACHE_RESULT_MAX] = [PERF_COUNT_HW_CACHE_RESULT_MAX] =
@ -179,7 +179,7 @@ static __initconst u64 westmere_hw_cache_event_ids
}, },
}; };
static __initconst u64 nehalem_hw_cache_event_ids static __initconst const u64 nehalem_hw_cache_event_ids
[PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_MAX]
[PERF_COUNT_HW_CACHE_OP_MAX] [PERF_COUNT_HW_CACHE_OP_MAX]
[PERF_COUNT_HW_CACHE_RESULT_MAX] = [PERF_COUNT_HW_CACHE_RESULT_MAX] =
@ -270,7 +270,7 @@ static __initconst u64 nehalem_hw_cache_event_ids
}, },
}; };
static __initconst u64 core2_hw_cache_event_ids static __initconst const u64 core2_hw_cache_event_ids
[PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_MAX]
[PERF_COUNT_HW_CACHE_OP_MAX] [PERF_COUNT_HW_CACHE_OP_MAX]
[PERF_COUNT_HW_CACHE_RESULT_MAX] = [PERF_COUNT_HW_CACHE_RESULT_MAX] =
@ -361,7 +361,7 @@ static __initconst u64 core2_hw_cache_event_ids
}, },
}; };
static __initconst u64 atom_hw_cache_event_ids static __initconst const u64 atom_hw_cache_event_ids
[PERF_COUNT_HW_CACHE_MAX] [PERF_COUNT_HW_CACHE_MAX]
[PERF_COUNT_HW_CACHE_OP_MAX] [PERF_COUNT_HW_CACHE_OP_MAX]
[PERF_COUNT_HW_CACHE_RESULT_MAX] = [PERF_COUNT_HW_CACHE_RESULT_MAX] =
@ -452,60 +452,6 @@ static __initconst u64 atom_hw_cache_event_ids
}, },
}; };
static u64 intel_pmu_raw_event(u64 hw_event)
{
#define CORE_EVNTSEL_EVENT_MASK 0x000000FFULL
#define CORE_EVNTSEL_UNIT_MASK 0x0000FF00ULL
#define CORE_EVNTSEL_EDGE_MASK 0x00040000ULL
#define CORE_EVNTSEL_INV_MASK 0x00800000ULL
#define CORE_EVNTSEL_REG_MASK 0xFF000000ULL
#define CORE_EVNTSEL_MASK \
(INTEL_ARCH_EVTSEL_MASK | \
INTEL_ARCH_UNIT_MASK | \
INTEL_ARCH_EDGE_MASK | \
INTEL_ARCH_INV_MASK | \
INTEL_ARCH_CNT_MASK)
return hw_event & CORE_EVNTSEL_MASK;
}
static void intel_pmu_enable_bts(u64 config)
{
unsigned long debugctlmsr;
debugctlmsr = get_debugctlmsr();
debugctlmsr |= X86_DEBUGCTL_TR;
debugctlmsr |= X86_DEBUGCTL_BTS;
debugctlmsr |= X86_DEBUGCTL_BTINT;
if (!(config & ARCH_PERFMON_EVENTSEL_OS))
debugctlmsr |= X86_DEBUGCTL_BTS_OFF_OS;
if (!(config & ARCH_PERFMON_EVENTSEL_USR))
debugctlmsr |= X86_DEBUGCTL_BTS_OFF_USR;
update_debugctlmsr(debugctlmsr);
}
static void intel_pmu_disable_bts(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
unsigned long debugctlmsr;
if (!cpuc->ds)
return;
debugctlmsr = get_debugctlmsr();
debugctlmsr &=
~(X86_DEBUGCTL_TR | X86_DEBUGCTL_BTS | X86_DEBUGCTL_BTINT |
X86_DEBUGCTL_BTS_OFF_OS | X86_DEBUGCTL_BTS_OFF_USR);
update_debugctlmsr(debugctlmsr);
}
static void intel_pmu_disable_all(void) static void intel_pmu_disable_all(void)
{ {
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events); struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
@ -514,12 +460,17 @@ static void intel_pmu_disable_all(void)
if (test_bit(X86_PMC_IDX_FIXED_BTS, cpuc->active_mask)) if (test_bit(X86_PMC_IDX_FIXED_BTS, cpuc->active_mask))
intel_pmu_disable_bts(); intel_pmu_disable_bts();
intel_pmu_pebs_disable_all();
intel_pmu_lbr_disable_all();
} }
static void intel_pmu_enable_all(void) static void intel_pmu_enable_all(int added)
{ {
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events); struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
intel_pmu_pebs_enable_all();
intel_pmu_lbr_enable_all();
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, x86_pmu.intel_ctrl); wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, x86_pmu.intel_ctrl);
if (test_bit(X86_PMC_IDX_FIXED_BTS, cpuc->active_mask)) { if (test_bit(X86_PMC_IDX_FIXED_BTS, cpuc->active_mask)) {
@ -533,6 +484,42 @@ static void intel_pmu_enable_all(void)
} }
} }
/*
* Workaround for:
* Intel Errata AAK100 (model 26)
* Intel Errata AAP53 (model 30)
* Intel Errata BD53 (model 44)
*
* These chips need to be 'reset' when adding counters by programming
* the magic three (non counting) events 0x4300D2, 0x4300B1 and 0x4300B5
* either in sequence on the same PMC or on different PMCs.
*/
static void intel_pmu_nhm_enable_all(int added)
{
if (added) {
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
int i;
wrmsrl(MSR_ARCH_PERFMON_EVENTSEL0 + 0, 0x4300D2);
wrmsrl(MSR_ARCH_PERFMON_EVENTSEL0 + 1, 0x4300B1);
wrmsrl(MSR_ARCH_PERFMON_EVENTSEL0 + 2, 0x4300B5);
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, 0x3);
wrmsrl(MSR_CORE_PERF_GLOBAL_CTRL, 0x0);
for (i = 0; i < 3; i++) {
struct perf_event *event = cpuc->events[i];
if (!event)
continue;
__x86_pmu_enable_event(&event->hw,
ARCH_PERFMON_EVENTSEL_ENABLE);
}
}
intel_pmu_enable_all(added);
}
static inline u64 intel_pmu_get_status(void) static inline u64 intel_pmu_get_status(void)
{ {
u64 status; u64 status;
@ -547,8 +534,7 @@ static inline void intel_pmu_ack_status(u64 ack)
wrmsrl(MSR_CORE_PERF_GLOBAL_OVF_CTRL, ack); wrmsrl(MSR_CORE_PERF_GLOBAL_OVF_CTRL, ack);
} }
static inline void static void intel_pmu_disable_fixed(struct hw_perf_event *hwc)
intel_pmu_disable_fixed(struct hw_perf_event *hwc)
{ {
int idx = hwc->idx - X86_PMC_IDX_FIXED; int idx = hwc->idx - X86_PMC_IDX_FIXED;
u64 ctrl_val, mask; u64 ctrl_val, mask;
@ -557,71 +543,10 @@ intel_pmu_disable_fixed(struct hw_perf_event *hwc)
rdmsrl(hwc->config_base, ctrl_val); rdmsrl(hwc->config_base, ctrl_val);
ctrl_val &= ~mask; ctrl_val &= ~mask;
(void)checking_wrmsrl(hwc->config_base, ctrl_val); wrmsrl(hwc->config_base, ctrl_val);
} }
static void intel_pmu_drain_bts_buffer(void) static void intel_pmu_disable_event(struct perf_event *event)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct bts_record {
u64 from;
u64 to;
u64 flags;
};
struct perf_event *event = cpuc->events[X86_PMC_IDX_FIXED_BTS];
struct bts_record *at, *top;
struct perf_output_handle handle;
struct perf_event_header header;
struct perf_sample_data data;
struct pt_regs regs;
if (!event)
return;
if (!ds)
return;
at = (struct bts_record *)(unsigned long)ds->bts_buffer_base;
top = (struct bts_record *)(unsigned long)ds->bts_index;
if (top <= at)
return;
ds->bts_index = ds->bts_buffer_base;
perf_sample_data_init(&data, 0);
data.period = event->hw.last_period;
regs.ip = 0;
/*
* Prepare a generic sample, i.e. fill in the invariant fields.
* We will overwrite the from and to address before we output
* the sample.
*/
perf_prepare_sample(&header, &data, event, &regs);
if (perf_output_begin(&handle, event,
header.size * (top - at), 1, 1))
return;
for (; at < top; at++) {
data.ip = at->from;
data.addr = at->to;
perf_output_sample(&handle, &header, &data, event);
}
perf_output_end(&handle);
/* There's new data available. */
event->hw.interrupts++;
event->pending_kill = POLL_IN;
}
static inline void
intel_pmu_disable_event(struct perf_event *event)
{ {
struct hw_perf_event *hwc = &event->hw; struct hw_perf_event *hwc = &event->hw;
@ -637,14 +562,15 @@ intel_pmu_disable_event(struct perf_event *event)
} }
x86_pmu_disable_event(event); x86_pmu_disable_event(event);
if (unlikely(event->attr.precise_ip))
intel_pmu_pebs_disable(event);
} }
static inline void static void intel_pmu_enable_fixed(struct hw_perf_event *hwc)
intel_pmu_enable_fixed(struct hw_perf_event *hwc)
{ {
int idx = hwc->idx - X86_PMC_IDX_FIXED; int idx = hwc->idx - X86_PMC_IDX_FIXED;
u64 ctrl_val, bits, mask; u64 ctrl_val, bits, mask;
int err;
/* /*
* Enable IRQ generation (0x8), * Enable IRQ generation (0x8),
@ -669,7 +595,7 @@ intel_pmu_enable_fixed(struct hw_perf_event *hwc)
rdmsrl(hwc->config_base, ctrl_val); rdmsrl(hwc->config_base, ctrl_val);
ctrl_val &= ~mask; ctrl_val &= ~mask;
ctrl_val |= bits; ctrl_val |= bits;
err = checking_wrmsrl(hwc->config_base, ctrl_val); wrmsrl(hwc->config_base, ctrl_val);
} }
static void intel_pmu_enable_event(struct perf_event *event) static void intel_pmu_enable_event(struct perf_event *event)
@ -689,7 +615,10 @@ static void intel_pmu_enable_event(struct perf_event *event)
return; return;
} }
__x86_pmu_enable_event(hwc); if (unlikely(event->attr.precise_ip))
intel_pmu_pebs_enable(event);
__x86_pmu_enable_event(hwc, ARCH_PERFMON_EVENTSEL_ENABLE);
} }
/* /*
@ -708,20 +637,20 @@ static void intel_pmu_reset(void)
unsigned long flags; unsigned long flags;
int idx; int idx;
if (!x86_pmu.num_events) if (!x86_pmu.num_counters)
return; return;
local_irq_save(flags); local_irq_save(flags);
printk("clearing PMU state on CPU#%d\n", smp_processor_id()); printk("clearing PMU state on CPU#%d\n", smp_processor_id());
for (idx = 0; idx < x86_pmu.num_events; idx++) { for (idx = 0; idx < x86_pmu.num_counters; idx++) {
checking_wrmsrl(x86_pmu.eventsel + idx, 0ull); checking_wrmsrl(x86_pmu.eventsel + idx, 0ull);
checking_wrmsrl(x86_pmu.perfctr + idx, 0ull); checking_wrmsrl(x86_pmu.perfctr + idx, 0ull);
} }
for (idx = 0; idx < x86_pmu.num_events_fixed; idx++) { for (idx = 0; idx < x86_pmu.num_counters_fixed; idx++)
checking_wrmsrl(MSR_ARCH_PERFMON_FIXED_CTR0 + idx, 0ull); checking_wrmsrl(MSR_ARCH_PERFMON_FIXED_CTR0 + idx, 0ull);
}
if (ds) if (ds)
ds->bts_index = ds->bts_buffer_base; ds->bts_index = ds->bts_buffer_base;
@ -747,7 +676,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
intel_pmu_drain_bts_buffer(); intel_pmu_drain_bts_buffer();
status = intel_pmu_get_status(); status = intel_pmu_get_status();
if (!status) { if (!status) {
intel_pmu_enable_all(); intel_pmu_enable_all(0);
return 0; return 0;
} }
@ -762,6 +691,15 @@ again:
inc_irq_stat(apic_perf_irqs); inc_irq_stat(apic_perf_irqs);
ack = status; ack = status;
intel_pmu_lbr_read();
/*
* PEBS overflow sets bit 62 in the global status register
*/
if (__test_and_clear_bit(62, (unsigned long *)&status))
x86_pmu.drain_pebs(regs);
for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) { for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
struct perf_event *event = cpuc->events[bit]; struct perf_event *event = cpuc->events[bit];
@ -787,26 +725,22 @@ again:
goto again; goto again;
done: done:
intel_pmu_enable_all(); intel_pmu_enable_all(0);
return 1; return 1;
} }
static struct event_constraint bts_constraint =
EVENT_CONSTRAINT(0, 1ULL << X86_PMC_IDX_FIXED_BTS, 0);
static struct event_constraint * static struct event_constraint *
intel_special_constraints(struct perf_event *event) intel_bts_constraints(struct perf_event *event)
{ {
unsigned int hw_event; struct hw_perf_event *hwc = &event->hw;
unsigned int hw_event, bts_event;
hw_event = event->hw.config & INTEL_ARCH_EVENT_MASK; hw_event = hwc->config & INTEL_ARCH_EVENT_MASK;
bts_event = x86_pmu.event_map(PERF_COUNT_HW_BRANCH_INSTRUCTIONS);
if (unlikely((hw_event ==
x86_pmu.event_map(PERF_COUNT_HW_BRANCH_INSTRUCTIONS)) &&
(event->hw.sample_period == 1))) {
if (unlikely(hw_event == bts_event && hwc->sample_period == 1))
return &bts_constraint; return &bts_constraint;
}
return NULL; return NULL;
} }
@ -815,24 +749,53 @@ intel_get_event_constraints(struct cpu_hw_events *cpuc, struct perf_event *event
{ {
struct event_constraint *c; struct event_constraint *c;
c = intel_special_constraints(event); c = intel_bts_constraints(event);
if (c)
return c;
c = intel_pebs_constraints(event);
if (c) if (c)
return c; return c;
return x86_get_event_constraints(cpuc, event); return x86_get_event_constraints(cpuc, event);
} }
static __initconst struct x86_pmu core_pmu = { static int intel_pmu_hw_config(struct perf_event *event)
{
int ret = x86_pmu_hw_config(event);
if (ret)
return ret;
if (event->attr.type != PERF_TYPE_RAW)
return 0;
if (!(event->attr.config & ARCH_PERFMON_EVENTSEL_ANY))
return 0;
if (x86_pmu.version < 3)
return -EINVAL;
if (perf_paranoid_cpu() && !capable(CAP_SYS_ADMIN))
return -EACCES;
event->hw.config |= ARCH_PERFMON_EVENTSEL_ANY;
return 0;
}
static __initconst const struct x86_pmu core_pmu = {
.name = "core", .name = "core",
.handle_irq = x86_pmu_handle_irq, .handle_irq = x86_pmu_handle_irq,
.disable_all = x86_pmu_disable_all, .disable_all = x86_pmu_disable_all,
.enable_all = x86_pmu_enable_all, .enable_all = x86_pmu_enable_all,
.enable = x86_pmu_enable_event, .enable = x86_pmu_enable_event,
.disable = x86_pmu_disable_event, .disable = x86_pmu_disable_event,
.hw_config = x86_pmu_hw_config,
.schedule_events = x86_schedule_events,
.eventsel = MSR_ARCH_PERFMON_EVENTSEL0, .eventsel = MSR_ARCH_PERFMON_EVENTSEL0,
.perfctr = MSR_ARCH_PERFMON_PERFCTR0, .perfctr = MSR_ARCH_PERFMON_PERFCTR0,
.event_map = intel_pmu_event_map, .event_map = intel_pmu_event_map,
.raw_event = intel_pmu_raw_event,
.max_events = ARRAY_SIZE(intel_perfmon_event_map), .max_events = ARRAY_SIZE(intel_perfmon_event_map),
.apic = 1, .apic = 1,
/* /*
@ -845,17 +808,32 @@ static __initconst struct x86_pmu core_pmu = {
.event_constraints = intel_core_event_constraints, .event_constraints = intel_core_event_constraints,
}; };
static __initconst struct x86_pmu intel_pmu = { static void intel_pmu_cpu_starting(int cpu)
{
init_debug_store_on_cpu(cpu);
/*
* Deal with CPUs that don't clear their LBRs on power-up.
*/
intel_pmu_lbr_reset();
}
static void intel_pmu_cpu_dying(int cpu)
{
fini_debug_store_on_cpu(cpu);
}
static __initconst const struct x86_pmu intel_pmu = {
.name = "Intel", .name = "Intel",
.handle_irq = intel_pmu_handle_irq, .handle_irq = intel_pmu_handle_irq,
.disable_all = intel_pmu_disable_all, .disable_all = intel_pmu_disable_all,
.enable_all = intel_pmu_enable_all, .enable_all = intel_pmu_enable_all,
.enable = intel_pmu_enable_event, .enable = intel_pmu_enable_event,
.disable = intel_pmu_disable_event, .disable = intel_pmu_disable_event,
.hw_config = intel_pmu_hw_config,
.schedule_events = x86_schedule_events,
.eventsel = MSR_ARCH_PERFMON_EVENTSEL0, .eventsel = MSR_ARCH_PERFMON_EVENTSEL0,
.perfctr = MSR_ARCH_PERFMON_PERFCTR0, .perfctr = MSR_ARCH_PERFMON_PERFCTR0,
.event_map = intel_pmu_event_map, .event_map = intel_pmu_event_map,
.raw_event = intel_pmu_raw_event,
.max_events = ARRAY_SIZE(intel_perfmon_event_map), .max_events = ARRAY_SIZE(intel_perfmon_event_map),
.apic = 1, .apic = 1,
/* /*
@ -864,14 +842,38 @@ static __initconst struct x86_pmu intel_pmu = {
* the generic event period: * the generic event period:
*/ */
.max_period = (1ULL << 31) - 1, .max_period = (1ULL << 31) - 1,
.enable_bts = intel_pmu_enable_bts,
.disable_bts = intel_pmu_disable_bts,
.get_event_constraints = intel_get_event_constraints, .get_event_constraints = intel_get_event_constraints,
.cpu_starting = init_debug_store_on_cpu, .cpu_starting = intel_pmu_cpu_starting,
.cpu_dying = fini_debug_store_on_cpu, .cpu_dying = intel_pmu_cpu_dying,
}; };
static void intel_clovertown_quirks(void)
{
/*
* PEBS is unreliable due to:
*
* AJ67 - PEBS may experience CPL leaks
* AJ68 - PEBS PMI may be delayed by one event
* AJ69 - GLOBAL_STATUS[62] will only be set when DEBUGCTL[12]
* AJ106 - FREEZE_LBRS_ON_PMI doesn't work in combination with PEBS
*
* AJ67 could be worked around by restricting the OS/USR flags.
* AJ69 could be worked around by setting PMU_FREEZE_ON_PMI.
*
* AJ106 could possibly be worked around by not allowing LBR
* usage from PEBS, including the fixup.
* AJ68 could possibly be worked around by always programming
* a pebs_event_reset[0] value and coping with the lost events.
*
* But taken together it might just make sense to not enable PEBS on
* these chips.
*/
printk(KERN_WARNING "PEBS disabled due to CPU errata.\n");
x86_pmu.pebs = 0;
x86_pmu.pebs_constraints = NULL;
}
static __init int intel_pmu_init(void) static __init int intel_pmu_init(void)
{ {
union cpuid10_edx edx; union cpuid10_edx edx;
@ -881,12 +883,13 @@ static __init int intel_pmu_init(void)
int version; int version;
if (!cpu_has(&boot_cpu_data, X86_FEATURE_ARCH_PERFMON)) { if (!cpu_has(&boot_cpu_data, X86_FEATURE_ARCH_PERFMON)) {
/* check for P6 processor family */ switch (boot_cpu_data.x86) {
if (boot_cpu_data.x86 == 6) { case 0x6:
return p6_pmu_init(); return p6_pmu_init();
} else { case 0xf:
return -ENODEV; return p4_pmu_init();
} }
return -ENODEV;
} }
/* /*
@ -904,16 +907,28 @@ static __init int intel_pmu_init(void)
x86_pmu = intel_pmu; x86_pmu = intel_pmu;
x86_pmu.version = version; x86_pmu.version = version;
x86_pmu.num_events = eax.split.num_events; x86_pmu.num_counters = eax.split.num_counters;
x86_pmu.event_bits = eax.split.bit_width; x86_pmu.cntval_bits = eax.split.bit_width;
x86_pmu.event_mask = (1ULL << eax.split.bit_width) - 1; x86_pmu.cntval_mask = (1ULL << eax.split.bit_width) - 1;
/* /*
* Quirk: v2 perfmon does not report fixed-purpose events, so * Quirk: v2 perfmon does not report fixed-purpose events, so
* assume at least 3 events: * assume at least 3 events:
*/ */
if (version > 1) if (version > 1)
x86_pmu.num_events_fixed = max((int)edx.split.num_events_fixed, 3); x86_pmu.num_counters_fixed = max((int)edx.split.num_counters_fixed, 3);
/*
* v2 and above have a perf capabilities MSR
*/
if (version > 1) {
u64 capabilities;
rdmsrl(MSR_IA32_PERF_CAPABILITIES, capabilities);
x86_pmu.intel_cap.capabilities = capabilities;
}
intel_ds_init();
/* /*
* Install the hw-cache-events table: * Install the hw-cache-events table:
@ -924,12 +939,15 @@ static __init int intel_pmu_init(void)
break; break;
case 15: /* original 65 nm celeron/pentium/core2/xeon, "Merom"/"Conroe" */ case 15: /* original 65 nm celeron/pentium/core2/xeon, "Merom"/"Conroe" */
x86_pmu.quirks = intel_clovertown_quirks;
case 22: /* single-core 65 nm celeron/core2solo "Merom-L"/"Conroe-L" */ case 22: /* single-core 65 nm celeron/core2solo "Merom-L"/"Conroe-L" */
case 23: /* current 45 nm celeron/core2/xeon "Penryn"/"Wolfdale" */ case 23: /* current 45 nm celeron/core2/xeon "Penryn"/"Wolfdale" */
case 29: /* six-core 45 nm xeon "Dunnington" */ case 29: /* six-core 45 nm xeon "Dunnington" */
memcpy(hw_cache_event_ids, core2_hw_cache_event_ids, memcpy(hw_cache_event_ids, core2_hw_cache_event_ids,
sizeof(hw_cache_event_ids)); sizeof(hw_cache_event_ids));
intel_pmu_lbr_init_core();
x86_pmu.event_constraints = intel_core2_event_constraints; x86_pmu.event_constraints = intel_core2_event_constraints;
pr_cont("Core2 events, "); pr_cont("Core2 events, ");
break; break;
@ -940,13 +958,19 @@ static __init int intel_pmu_init(void)
memcpy(hw_cache_event_ids, nehalem_hw_cache_event_ids, memcpy(hw_cache_event_ids, nehalem_hw_cache_event_ids,
sizeof(hw_cache_event_ids)); sizeof(hw_cache_event_ids));
intel_pmu_lbr_init_nhm();
x86_pmu.event_constraints = intel_nehalem_event_constraints; x86_pmu.event_constraints = intel_nehalem_event_constraints;
pr_cont("Nehalem/Corei7 events, "); x86_pmu.enable_all = intel_pmu_nhm_enable_all;
pr_cont("Nehalem events, ");
break; break;
case 28: /* Atom */ case 28: /* Atom */
memcpy(hw_cache_event_ids, atom_hw_cache_event_ids, memcpy(hw_cache_event_ids, atom_hw_cache_event_ids,
sizeof(hw_cache_event_ids)); sizeof(hw_cache_event_ids));
intel_pmu_lbr_init_atom();
x86_pmu.event_constraints = intel_gen_event_constraints; x86_pmu.event_constraints = intel_gen_event_constraints;
pr_cont("Atom events, "); pr_cont("Atom events, ");
break; break;
@ -956,7 +980,10 @@ static __init int intel_pmu_init(void)
memcpy(hw_cache_event_ids, westmere_hw_cache_event_ids, memcpy(hw_cache_event_ids, westmere_hw_cache_event_ids,
sizeof(hw_cache_event_ids)); sizeof(hw_cache_event_ids));
intel_pmu_lbr_init_nhm();
x86_pmu.event_constraints = intel_westmere_event_constraints; x86_pmu.event_constraints = intel_westmere_event_constraints;
x86_pmu.enable_all = intel_pmu_nhm_enable_all;
pr_cont("Westmere events, "); pr_cont("Westmere events, ");
break; break;

641
arch/x86/kernel/cpu/perf_event_intel_ds.c Normal file
View file

@ -0,0 +1,641 @@
#ifdef CONFIG_CPU_SUP_INTEL
/* The maximal number of PEBS events: */
#define MAX_PEBS_EVENTS 4
/* The size of a BTS record in bytes: */
#define BTS_RECORD_SIZE 24
#define BTS_BUFFER_SIZE (PAGE_SIZE << 4)
#define PEBS_BUFFER_SIZE PAGE_SIZE
/*
* pebs_record_32 for p4 and core not supported
struct pebs_record_32 {
u32 flags, ip;
u32 ax, bc, cx, dx;
u32 si, di, bp, sp;
};
*/
struct pebs_record_core {
u64 flags, ip;
u64 ax, bx, cx, dx;
u64 si, di, bp, sp;
u64 r8, r9, r10, r11;
u64 r12, r13, r14, r15;
};
struct pebs_record_nhm {
u64 flags, ip;
u64 ax, bx, cx, dx;
u64 si, di, bp, sp;
u64 r8, r9, r10, r11;
u64 r12, r13, r14, r15;
u64 status, dla, dse, lat;
};
/*
* A debug store configuration.
*
* We only support architectures that use 64bit fields.
*/
struct debug_store {
u64 bts_buffer_base;
u64 bts_index;
u64 bts_absolute_maximum;
u64 bts_interrupt_threshold;
u64 pebs_buffer_base;
u64 pebs_index;
u64 pebs_absolute_maximum;
u64 pebs_interrupt_threshold;
u64 pebs_event_reset[MAX_PEBS_EVENTS];
};
static void init_debug_store_on_cpu(int cpu)
{
struct debug_store *ds = per_cpu(cpu_hw_events, cpu).ds;
if (!ds)
return;
wrmsr_on_cpu(cpu, MSR_IA32_DS_AREA,
(u32)((u64)(unsigned long)ds),
(u32)((u64)(unsigned long)ds >> 32));
}
static void fini_debug_store_on_cpu(int cpu)
{
if (!per_cpu(cpu_hw_events, cpu).ds)
return;
wrmsr_on_cpu(cpu, MSR_IA32_DS_AREA, 0, 0);
}
static void release_ds_buffers(void)
{
int cpu;
if (!x86_pmu.bts && !x86_pmu.pebs)
return;
get_online_cpus();
for_each_online_cpu(cpu)
fini_debug_store_on_cpu(cpu);
for_each_possible_cpu(cpu) {
struct debug_store *ds = per_cpu(cpu_hw_events, cpu).ds;
if (!ds)
continue;
per_cpu(cpu_hw_events, cpu).ds = NULL;
kfree((void *)(unsigned long)ds->pebs_buffer_base);
kfree((void *)(unsigned long)ds->bts_buffer_base);
kfree(ds);
}
put_online_cpus();
}
static int reserve_ds_buffers(void)
{
int cpu, err = 0;
if (!x86_pmu.bts && !x86_pmu.pebs)
return 0;
get_online_cpus();
for_each_possible_cpu(cpu) {
struct debug_store *ds;
void *buffer;
int max, thresh;
err = -ENOMEM;
ds = kzalloc(sizeof(*ds), GFP_KERNEL);
if (unlikely(!ds))
break;
per_cpu(cpu_hw_events, cpu).ds = ds;
if (x86_pmu.bts) {
buffer = kzalloc(BTS_BUFFER_SIZE, GFP_KERNEL);
if (unlikely(!buffer))
break;
max = BTS_BUFFER_SIZE / BTS_RECORD_SIZE;
thresh = max / 16;
ds->bts_buffer_base = (u64)(unsigned long)buffer;
ds->bts_index = ds->bts_buffer_base;
ds->bts_absolute_maximum = ds->bts_buffer_base +
max * BTS_RECORD_SIZE;
ds->bts_interrupt_threshold = ds->bts_absolute_maximum -
thresh * BTS_RECORD_SIZE;
}
if (x86_pmu.pebs) {
buffer = kzalloc(PEBS_BUFFER_SIZE, GFP_KERNEL);
if (unlikely(!buffer))
break;
max = PEBS_BUFFER_SIZE / x86_pmu.pebs_record_size;
ds->pebs_buffer_base = (u64)(unsigned long)buffer;
ds->pebs_index = ds->pebs_buffer_base;
ds->pebs_absolute_maximum = ds->pebs_buffer_base +
max * x86_pmu.pebs_record_size;
/*
* Always use single record PEBS
*/
ds->pebs_interrupt_threshold = ds->pebs_buffer_base +
x86_pmu.pebs_record_size;
}
err = 0;
}
if (err)
release_ds_buffers();
else {
for_each_online_cpu(cpu)
init_debug_store_on_cpu(cpu);
}
put_online_cpus();
return err;
}
/*
* BTS
*/
static struct event_constraint bts_constraint =
EVENT_CONSTRAINT(0, 1ULL << X86_PMC_IDX_FIXED_BTS, 0);
static void intel_pmu_enable_bts(u64 config)
{
unsigned long debugctlmsr;
debugctlmsr = get_debugctlmsr();
debugctlmsr |= DEBUGCTLMSR_TR;
debugctlmsr |= DEBUGCTLMSR_BTS;
debugctlmsr |= DEBUGCTLMSR_BTINT;
if (!(config & ARCH_PERFMON_EVENTSEL_OS))
debugctlmsr |= DEBUGCTLMSR_BTS_OFF_OS;
if (!(config & ARCH_PERFMON_EVENTSEL_USR))
debugctlmsr |= DEBUGCTLMSR_BTS_OFF_USR;
update_debugctlmsr(debugctlmsr);
}
static void intel_pmu_disable_bts(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
unsigned long debugctlmsr;
if (!cpuc->ds)
return;
debugctlmsr = get_debugctlmsr();
debugctlmsr &=
~(DEBUGCTLMSR_TR | DEBUGCTLMSR_BTS | DEBUGCTLMSR_BTINT |
DEBUGCTLMSR_BTS_OFF_OS | DEBUGCTLMSR_BTS_OFF_USR);
update_debugctlmsr(debugctlmsr);
}
static void intel_pmu_drain_bts_buffer(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct bts_record {
u64 from;
u64 to;
u64 flags;
};
struct perf_event *event = cpuc->events[X86_PMC_IDX_FIXED_BTS];
struct bts_record *at, *top;
struct perf_output_handle handle;
struct perf_event_header header;
struct perf_sample_data data;
struct pt_regs regs;
if (!event)
return;
if (!ds)
return;
at = (struct bts_record *)(unsigned long)ds->bts_buffer_base;
top = (struct bts_record *)(unsigned long)ds->bts_index;
if (top <= at)
return;
ds->bts_index = ds->bts_buffer_base;
perf_sample_data_init(&data, 0);
data.period = event->hw.last_period;
regs.ip = 0;
/*
* Prepare a generic sample, i.e. fill in the invariant fields.
* We will overwrite the from and to address before we output
* the sample.
*/
perf_prepare_sample(&header, &data, event, &regs);
if (perf_output_begin(&handle, event, header.size * (top - at), 1, 1))
return;
for (; at < top; at++) {
data.ip = at->from;
data.addr = at->to;
perf_output_sample(&handle, &header, &data, event);
}
perf_output_end(&handle);
/* There's new data available. */
event->hw.interrupts++;
event->pending_kill = POLL_IN;
}
/*
* PEBS
*/
static struct event_constraint intel_core_pebs_events[] = {
PEBS_EVENT_CONSTRAINT(0x00c0, 0x1), /* INSTR_RETIRED.ANY */
PEBS_EVENT_CONSTRAINT(0xfec1, 0x1), /* X87_OPS_RETIRED.ANY */
PEBS_EVENT_CONSTRAINT(0x00c5, 0x1), /* BR_INST_RETIRED.MISPRED */
PEBS_EVENT_CONSTRAINT(0x1fc7, 0x1), /* SIMD_INST_RETURED.ANY */
PEBS_EVENT_CONSTRAINT(0x01cb, 0x1), /* MEM_LOAD_RETIRED.L1D_MISS */
PEBS_EVENT_CONSTRAINT(0x02cb, 0x1), /* MEM_LOAD_RETIRED.L1D_LINE_MISS */
PEBS_EVENT_CONSTRAINT(0x04cb, 0x1), /* MEM_LOAD_RETIRED.L2_MISS */
PEBS_EVENT_CONSTRAINT(0x08cb, 0x1), /* MEM_LOAD_RETIRED.L2_LINE_MISS */
PEBS_EVENT_CONSTRAINT(0x10cb, 0x1), /* MEM_LOAD_RETIRED.DTLB_MISS */
EVENT_CONSTRAINT_END
};
static struct event_constraint intel_nehalem_pebs_events[] = {
PEBS_EVENT_CONSTRAINT(0x00c0, 0xf), /* INSTR_RETIRED.ANY */
PEBS_EVENT_CONSTRAINT(0xfec1, 0xf), /* X87_OPS_RETIRED.ANY */
PEBS_EVENT_CONSTRAINT(0x00c5, 0xf), /* BR_INST_RETIRED.MISPRED */
PEBS_EVENT_CONSTRAINT(0x1fc7, 0xf), /* SIMD_INST_RETURED.ANY */
PEBS_EVENT_CONSTRAINT(0x01cb, 0xf), /* MEM_LOAD_RETIRED.L1D_MISS */
PEBS_EVENT_CONSTRAINT(0x02cb, 0xf), /* MEM_LOAD_RETIRED.L1D_LINE_MISS */
PEBS_EVENT_CONSTRAINT(0x04cb, 0xf), /* MEM_LOAD_RETIRED.L2_MISS */
PEBS_EVENT_CONSTRAINT(0x08cb, 0xf), /* MEM_LOAD_RETIRED.L2_LINE_MISS */
PEBS_EVENT_CONSTRAINT(0x10cb, 0xf), /* MEM_LOAD_RETIRED.DTLB_MISS */
EVENT_CONSTRAINT_END
};
static struct event_constraint *
intel_pebs_constraints(struct perf_event *event)
{
struct event_constraint *c;
if (!event->attr.precise_ip)
return NULL;
if (x86_pmu.pebs_constraints) {
for_each_event_constraint(c, x86_pmu.pebs_constraints) {
if ((event->hw.config & c->cmask) == c->code)
return c;
}
}
return &emptyconstraint;
}
static void intel_pmu_pebs_enable(struct perf_event *event)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
hwc->config &= ~ARCH_PERFMON_EVENTSEL_INT;
cpuc->pebs_enabled |= 1ULL << hwc->idx;
WARN_ON_ONCE(cpuc->enabled);
if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip > 1)
intel_pmu_lbr_enable(event);
}
static void intel_pmu_pebs_disable(struct perf_event *event)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
struct hw_perf_event *hwc = &event->hw;
cpuc->pebs_enabled &= ~(1ULL << hwc->idx);
if (cpuc->enabled)
wrmsrl(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled);
hwc->config |= ARCH_PERFMON_EVENTSEL_INT;
if (x86_pmu.intel_cap.pebs_trap && event->attr.precise_ip > 1)
intel_pmu_lbr_disable(event);
}
static void intel_pmu_pebs_enable_all(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
if (cpuc->pebs_enabled)
wrmsrl(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled);
}
static void intel_pmu_pebs_disable_all(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
if (cpuc->pebs_enabled)
wrmsrl(MSR_IA32_PEBS_ENABLE, 0);
}
#include <asm/insn.h>
static inline bool kernel_ip(unsigned long ip)
{
#ifdef CONFIG_X86_32
return ip > PAGE_OFFSET;
#else
return (long)ip < 0;
#endif
}
static int intel_pmu_pebs_fixup_ip(struct pt_regs *regs)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
unsigned long from = cpuc->lbr_entries[0].from;
unsigned long old_to, to = cpuc->lbr_entries[0].to;
unsigned long ip = regs->ip;
/*
* We don't need to fixup if the PEBS assist is fault like
*/
if (!x86_pmu.intel_cap.pebs_trap)
return 1;
/*
* No LBR entry, no basic block, no rewinding
*/
if (!cpuc->lbr_stack.nr || !from || !to)
return 0;
/*
* Basic blocks should never cross user/kernel boundaries
*/
if (kernel_ip(ip) != kernel_ip(to))
return 0;
/*
* unsigned math, either ip is before the start (impossible) or
* the basic block is larger than 1 page (sanity)
*/
if ((ip - to) > PAGE_SIZE)
return 0;
/*
* We sampled a branch insn, rewind using the LBR stack
*/
if (ip == to) {
regs->ip = from;
return 1;
}
do {
struct insn insn;
u8 buf[MAX_INSN_SIZE];
void *kaddr;
old_to = to;
if (!kernel_ip(ip)) {
int bytes, size = MAX_INSN_SIZE;
bytes = copy_from_user_nmi(buf, (void __user *)to, size);
if (bytes != size)
return 0;
kaddr = buf;
} else
kaddr = (void *)to;
kernel_insn_init(&insn, kaddr);
insn_get_length(&insn);
to += insn.length;
} while (to < ip);
if (to == ip) {
regs->ip = old_to;
return 1;
}
/*
* Even though we decoded the basic block, the instruction stream
* never matched the given IP, either the TO or the IP got corrupted.
*/
return 0;
}
static int intel_pmu_save_and_restart(struct perf_event *event);
static void __intel_pmu_pebs_event(struct perf_event *event,
struct pt_regs *iregs, void *__pebs)
{
/*
* We cast to pebs_record_core since that is a subset of
* both formats and we don't use the other fields in this
* routine.
*/
struct pebs_record_core *pebs = __pebs;
struct perf_sample_data data;
struct pt_regs regs;
if (!intel_pmu_save_and_restart(event))
return;
perf_sample_data_init(&data, 0);
data.period = event->hw.last_period;
/*
* We use the interrupt regs as a base because the PEBS record
* does not contain a full regs set, specifically it seems to
* lack segment descriptors, which get used by things like
* user_mode().
*
* In the simple case fix up only the IP and BP,SP regs, for
* PERF_SAMPLE_IP and PERF_SAMPLE_CALLCHAIN to function properly.
* A possible PERF_SAMPLE_REGS will have to transfer all regs.
*/
regs = *iregs;
regs.ip = pebs->ip;
regs.bp = pebs->bp;
regs.sp = pebs->sp;
if (event->attr.precise_ip > 1 && intel_pmu_pebs_fixup_ip(&regs))
regs.flags |= PERF_EFLAGS_EXACT;
else
regs.flags &= ~PERF_EFLAGS_EXACT;
if (perf_event_overflow(event, 1, &data, &regs))
x86_pmu_stop(event);
}
static void intel_pmu_drain_pebs_core(struct pt_regs *iregs)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct perf_event *event = cpuc->events[0]; /* PMC0 only */
struct pebs_record_core *at, *top;
int n;
if (!ds || !x86_pmu.pebs)
return;
at = (struct pebs_record_core *)(unsigned long)ds->pebs_buffer_base;
top = (struct pebs_record_core *)(unsigned long)ds->pebs_index;
/*
* Whatever else happens, drain the thing
*/
ds->pebs_index = ds->pebs_buffer_base;
if (!test_bit(0, cpuc->active_mask))
return;
WARN_ON_ONCE(!event);
if (!event->attr.precise_ip)
return;
n = top - at;
if (n <= 0)
return;
/*
* Should not happen, we program the threshold at 1 and do not
* set a reset value.
*/
WARN_ON_ONCE(n > 1);
at += n - 1;
__intel_pmu_pebs_event(event, iregs, at);
}
static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
struct debug_store *ds = cpuc->ds;
struct pebs_record_nhm *at, *top;
struct perf_event *event = NULL;
u64 status = 0;
int bit, n;
if (!ds || !x86_pmu.pebs)
return;
at = (struct pebs_record_nhm *)(unsigned long)ds->pebs_buffer_base;
top = (struct pebs_record_nhm *)(unsigned long)ds->pebs_index;
ds->pebs_index = ds->pebs_buffer_base;
n = top - at;
if (n <= 0)
return;
/*
* Should not happen, we program the threshold at 1 and do not
* set a reset value.
*/
WARN_ON_ONCE(n > MAX_PEBS_EVENTS);
for ( ; at < top; at++) {
for_each_set_bit(bit, (unsigned long *)&at->status, MAX_PEBS_EVENTS) {
event = cpuc->events[bit];
if (!test_bit(bit, cpuc->active_mask))
continue;
WARN_ON_ONCE(!event);
if (!event->attr.precise_ip)
continue;
if (__test_and_set_bit(bit, (unsigned long *)&status))
continue;
break;
}
if (!event || bit >= MAX_PEBS_EVENTS)
continue;
__intel_pmu_pebs_event(event, iregs, at);
}
}
/*
* BTS, PEBS probe and setup
*/
static void intel_ds_init(void)
{
/*
* No support for 32bit formats
*/
if (!boot_cpu_has(X86_FEATURE_DTES64))
return;
x86_pmu.bts = boot_cpu_has(X86_FEATURE_BTS);
x86_pmu.pebs = boot_cpu_has(X86_FEATURE_PEBS);
if (x86_pmu.pebs) {
char pebs_type = x86_pmu.intel_cap.pebs_trap ? '+' : '-';
int format = x86_pmu.intel_cap.pebs_format;
switch (format) {
case 0:
printk(KERN_CONT "PEBS fmt0%c, ", pebs_type);
x86_pmu.pebs_record_size = sizeof(struct pebs_record_core);
x86_pmu.drain_pebs = intel_pmu_drain_pebs_core;
x86_pmu.pebs_constraints = intel_core_pebs_events;
break;
case 1:
printk(KERN_CONT "PEBS fmt1%c, ", pebs_type);
x86_pmu.pebs_record_size = sizeof(struct pebs_record_nhm);
x86_pmu.drain_pebs = intel_pmu_drain_pebs_nhm;
x86_pmu.pebs_constraints = intel_nehalem_pebs_events;
break;
default:
printk(KERN_CONT "no PEBS fmt%d%c, ", format, pebs_type);
x86_pmu.pebs = 0;
break;
}
}
}
#else /* CONFIG_CPU_SUP_INTEL */
static int reserve_ds_buffers(void)
{
return 0;
}
static void release_ds_buffers(void)
{
}
#endif /* CONFIG_CPU_SUP_INTEL */

218
arch/x86/kernel/cpu/perf_event_intel_lbr.c Normal file
View file

@ -0,0 +1,218 @@
#ifdef CONFIG_CPU_SUP_INTEL
enum {
LBR_FORMAT_32 = 0x00,
LBR_FORMAT_LIP = 0x01,
LBR_FORMAT_EIP = 0x02,
LBR_FORMAT_EIP_FLAGS = 0x03,
};
/*
* We only support LBR implementations that have FREEZE_LBRS_ON_PMI
* otherwise it becomes near impossible to get a reliable stack.
*/
static void __intel_pmu_lbr_enable(void)
{
u64 debugctl;
rdmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
debugctl |= (DEBUGCTLMSR_LBR | DEBUGCTLMSR_FREEZE_LBRS_ON_PMI);
wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
}
static void __intel_pmu_lbr_disable(void)
{
u64 debugctl;
rdmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
debugctl &= ~(DEBUGCTLMSR_LBR | DEBUGCTLMSR_FREEZE_LBRS_ON_PMI);
wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
}
static void intel_pmu_lbr_reset_32(void)
{
int i;
for (i = 0; i < x86_pmu.lbr_nr; i++)
wrmsrl(x86_pmu.lbr_from + i, 0);
}
static void intel_pmu_lbr_reset_64(void)
{
int i;
for (i = 0; i < x86_pmu.lbr_nr; i++) {
wrmsrl(x86_pmu.lbr_from + i, 0);
wrmsrl(x86_pmu.lbr_to + i, 0);
}
}
static void intel_pmu_lbr_reset(void)
{
if (!x86_pmu.lbr_nr)
return;
if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_32)
intel_pmu_lbr_reset_32();
else
intel_pmu_lbr_reset_64();
}
static void intel_pmu_lbr_enable(struct perf_event *event)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
if (!x86_pmu.lbr_nr)
return;
WARN_ON_ONCE(cpuc->enabled);
/*
* Reset the LBR stack if we changed task context to
* avoid data leaks.
*/
if (event->ctx->task && cpuc->lbr_context != event->ctx) {
intel_pmu_lbr_reset();
cpuc->lbr_context = event->ctx;
}
cpuc->lbr_users++;
}
static void intel_pmu_lbr_disable(struct perf_event *event)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
if (!x86_pmu.lbr_nr)
return;
cpuc->lbr_users--;
WARN_ON_ONCE(cpuc->lbr_users < 0);
if (cpuc->enabled && !cpuc->lbr_users)
__intel_pmu_lbr_disable();
}
static void intel_pmu_lbr_enable_all(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
if (cpuc->lbr_users)
__intel_pmu_lbr_enable();
}
static void intel_pmu_lbr_disable_all(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
if (cpuc->lbr_users)
__intel_pmu_lbr_disable();
}
static inline u64 intel_pmu_lbr_tos(void)
{
u64 tos;
rdmsrl(x86_pmu.lbr_tos, tos);
return tos;
}
static void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
{
unsigned long mask = x86_pmu.lbr_nr - 1;
u64 tos = intel_pmu_lbr_tos();
int i;
for (i = 0; i < x86_pmu.lbr_nr; i++) {
unsigned long lbr_idx = (tos - i) & mask;
union {
struct {
u32 from;
u32 to;
};
u64 lbr;
} msr_lastbranch;
rdmsrl(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
cpuc->lbr_entries[i].from = msr_lastbranch.from;
cpuc->lbr_entries[i].to = msr_lastbranch.to;
cpuc->lbr_entries[i].flags = 0;
}
cpuc->lbr_stack.nr = i;
}
#define LBR_FROM_FLAG_MISPRED (1ULL << 63)
/*
* Due to lack of segmentation in Linux the effective address (offset)
* is the same as the linear address, allowing us to merge the LIP and EIP
* LBR formats.
*/
static void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
{
unsigned long mask = x86_pmu.lbr_nr - 1;
int lbr_format = x86_pmu.intel_cap.lbr_format;
u64 tos = intel_pmu_lbr_tos();
int i;
for (i = 0; i < x86_pmu.lbr_nr; i++) {
unsigned long lbr_idx = (tos - i) & mask;
u64 from, to, flags = 0;
rdmsrl(x86_pmu.lbr_from + lbr_idx, from);
rdmsrl(x86_pmu.lbr_to + lbr_idx, to);
if (lbr_format == LBR_FORMAT_EIP_FLAGS) {
flags = !!(from & LBR_FROM_FLAG_MISPRED);
from = (u64)((((s64)from) << 1) >> 1);
}
cpuc->lbr_entries[i].from = from;
cpuc->lbr_entries[i].to = to;
cpuc->lbr_entries[i].flags = flags;
}
cpuc->lbr_stack.nr = i;
}
static void intel_pmu_lbr_read(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
if (!cpuc->lbr_users)
return;
if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_32)
intel_pmu_lbr_read_32(cpuc);
else
intel_pmu_lbr_read_64(cpuc);
}
static void intel_pmu_lbr_init_core(void)
{
x86_pmu.lbr_nr = 4;
x86_pmu.lbr_tos = 0x01c9;
x86_pmu.lbr_from = 0x40;
x86_pmu.lbr_to = 0x60;
}
static void intel_pmu_lbr_init_nhm(void)
{
x86_pmu.lbr_nr = 16;
x86_pmu.lbr_tos = 0x01c9;
x86_pmu.lbr_from = 0x680;
x86_pmu.lbr_to = 0x6c0;
}
static void intel_pmu_lbr_init_atom(void)
{
x86_pmu.lbr_nr = 8;
x86_pmu.lbr_tos = 0x01c9;
x86_pmu.lbr_from = 0x40;
x86_pmu.lbr_to = 0x60;
}
#endif /* CONFIG_CPU_SUP_INTEL */

857
arch/x86/kernel/cpu/perf_event_p4.c Normal file
View file

@ -0,0 +1,857 @@
/*
* Netburst Perfomance Events (P4, old Xeon)
*
* Copyright (C) 2010 Parallels, Inc., Cyrill Gorcunov <gorcunov@openvz.org>
* Copyright (C) 2010 Intel Corporation, Lin Ming <ming.m.lin@intel.com>
*
* For licencing details see kernel-base/COPYING
*/
#ifdef CONFIG_CPU_SUP_INTEL
#include <asm/perf_event_p4.h>
#define P4_CNTR_LIMIT 3
/*
* array indices: 0,1 - HT threads, used with HT enabled cpu
*/
struct p4_event_bind {
unsigned int opcode; /* Event code and ESCR selector */
unsigned int escr_msr[2]; /* ESCR MSR for this event */
char cntr[2][P4_CNTR_LIMIT]; /* counter index (offset), -1 on abscence */
};
struct p4_cache_event_bind {
unsigned int metric_pebs;
unsigned int metric_vert;
};
#define P4_GEN_CACHE_EVENT_BIND(name) \
[P4_CACHE__##name] = { \
.metric_pebs = P4_PEBS__##name, \
.metric_vert = P4_VERT__##name, \
}
static struct p4_cache_event_bind p4_cache_event_bind_map[] = {
P4_GEN_CACHE_EVENT_BIND(1stl_cache_load_miss_retired),
P4_GEN_CACHE_EVENT_BIND(2ndl_cache_load_miss_retired),
P4_GEN_CACHE_EVENT_BIND(dtlb_load_miss_retired),
P4_GEN_CACHE_EVENT_BIND(dtlb_store_miss_retired),
};
/*
* Note that we don't use CCCR1 here, there is an
* exception for P4_BSQ_ALLOCATION but we just have
* no workaround
*
* consider this binding as resources which particular
* event may borrow, it doesn't contain EventMask,
* Tags and friends -- they are left to a caller
*/
static struct p4_event_bind p4_event_bind_map[] = {
[P4_EVENT_TC_DELIVER_MODE] = {
.opcode = P4_OPCODE(P4_EVENT_TC_DELIVER_MODE),
.escr_msr = { MSR_P4_TC_ESCR0, MSR_P4_TC_ESCR1 },
.cntr = { {4, 5, -1}, {6, 7, -1} },
},
[P4_EVENT_BPU_FETCH_REQUEST] = {
.opcode = P4_OPCODE(P4_EVENT_BPU_FETCH_REQUEST),
.escr_msr = { MSR_P4_BPU_ESCR0, MSR_P4_BPU_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_ITLB_REFERENCE] = {
.opcode = P4_OPCODE(P4_EVENT_ITLB_REFERENCE),
.escr_msr = { MSR_P4_ITLB_ESCR0, MSR_P4_ITLB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_MEMORY_CANCEL] = {
.opcode = P4_OPCODE(P4_EVENT_MEMORY_CANCEL),
.escr_msr = { MSR_P4_DAC_ESCR0, MSR_P4_DAC_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_MEMORY_COMPLETE] = {
.opcode = P4_OPCODE(P4_EVENT_MEMORY_COMPLETE),
.escr_msr = { MSR_P4_SAAT_ESCR0 , MSR_P4_SAAT_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_LOAD_PORT_REPLAY] = {
.opcode = P4_OPCODE(P4_EVENT_LOAD_PORT_REPLAY),
.escr_msr = { MSR_P4_SAAT_ESCR0, MSR_P4_SAAT_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_STORE_PORT_REPLAY] = {
.opcode = P4_OPCODE(P4_EVENT_STORE_PORT_REPLAY),
.escr_msr = { MSR_P4_SAAT_ESCR0 , MSR_P4_SAAT_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_MOB_LOAD_REPLAY] = {
.opcode = P4_OPCODE(P4_EVENT_MOB_LOAD_REPLAY),
.escr_msr = { MSR_P4_MOB_ESCR0, MSR_P4_MOB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_PAGE_WALK_TYPE] = {
.opcode = P4_OPCODE(P4_EVENT_PAGE_WALK_TYPE),
.escr_msr = { MSR_P4_PMH_ESCR0, MSR_P4_PMH_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_BSQ_CACHE_REFERENCE] = {
.opcode = P4_OPCODE(P4_EVENT_BSQ_CACHE_REFERENCE),
.escr_msr = { MSR_P4_BSU_ESCR0, MSR_P4_BSU_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_IOQ_ALLOCATION] = {
.opcode = P4_OPCODE(P4_EVENT_IOQ_ALLOCATION),
.escr_msr = { MSR_P4_FSB_ESCR0, MSR_P4_FSB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_IOQ_ACTIVE_ENTRIES] = { /* shared ESCR */
.opcode = P4_OPCODE(P4_EVENT_IOQ_ACTIVE_ENTRIES),
.escr_msr = { MSR_P4_FSB_ESCR1, MSR_P4_FSB_ESCR1 },
.cntr = { {2, -1, -1}, {3, -1, -1} },
},
[P4_EVENT_FSB_DATA_ACTIVITY] = {
.opcode = P4_OPCODE(P4_EVENT_FSB_DATA_ACTIVITY),
.escr_msr = { MSR_P4_FSB_ESCR0, MSR_P4_FSB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_BSQ_ALLOCATION] = { /* shared ESCR, broken CCCR1 */
.opcode = P4_OPCODE(P4_EVENT_BSQ_ALLOCATION),
.escr_msr = { MSR_P4_BSU_ESCR0, MSR_P4_BSU_ESCR0 },
.cntr = { {0, -1, -1}, {1, -1, -1} },
},
[P4_EVENT_BSQ_ACTIVE_ENTRIES] = { /* shared ESCR */
.opcode = P4_OPCODE(P4_EVENT_BSQ_ACTIVE_ENTRIES),
.escr_msr = { MSR_P4_BSU_ESCR1 , MSR_P4_BSU_ESCR1 },
.cntr = { {2, -1, -1}, {3, -1, -1} },
},
[P4_EVENT_SSE_INPUT_ASSIST] = {
.opcode = P4_OPCODE(P4_EVENT_SSE_INPUT_ASSIST),
.escr_msr = { MSR_P4_FIRM_ESCR0, MSR_P4_FIRM_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_PACKED_SP_UOP] = {
.opcode = P4_OPCODE(P4_EVENT_PACKED_SP_UOP),
.escr_msr = { MSR_P4_FIRM_ESCR0, MSR_P4_FIRM_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_PACKED_DP_UOP] = {
.opcode = P4_OPCODE(P4_EVENT_PACKED_DP_UOP),
.escr_msr = { MSR_P4_FIRM_ESCR0, MSR_P4_FIRM_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_SCALAR_SP_UOP] = {
.opcode = P4_OPCODE(P4_EVENT_SCALAR_SP_UOP),
.escr_msr = { MSR_P4_FIRM_ESCR0, MSR_P4_FIRM_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_SCALAR_DP_UOP] = {
.opcode = P4_OPCODE(P4_EVENT_SCALAR_DP_UOP),
.escr_msr = { MSR_P4_FIRM_ESCR0, MSR_P4_FIRM_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_64BIT_MMX_UOP] = {
.opcode = P4_OPCODE(P4_EVENT_64BIT_MMX_UOP),
.escr_msr = { MSR_P4_FIRM_ESCR0, MSR_P4_FIRM_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_128BIT_MMX_UOP] = {
.opcode = P4_OPCODE(P4_EVENT_128BIT_MMX_UOP),
.escr_msr = { MSR_P4_FIRM_ESCR0, MSR_P4_FIRM_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_X87_FP_UOP] = {
.opcode = P4_OPCODE(P4_EVENT_X87_FP_UOP),
.escr_msr = { MSR_P4_FIRM_ESCR0, MSR_P4_FIRM_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_TC_MISC] = {
.opcode = P4_OPCODE(P4_EVENT_TC_MISC),
.escr_msr = { MSR_P4_TC_ESCR0, MSR_P4_TC_ESCR1 },
.cntr = { {4, 5, -1}, {6, 7, -1} },
},
[P4_EVENT_GLOBAL_POWER_EVENTS] = {
.opcode = P4_OPCODE(P4_EVENT_GLOBAL_POWER_EVENTS),
.escr_msr = { MSR_P4_FSB_ESCR0, MSR_P4_FSB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_TC_MS_XFER] = {
.opcode = P4_OPCODE(P4_EVENT_TC_MS_XFER),
.escr_msr = { MSR_P4_MS_ESCR0, MSR_P4_MS_ESCR1 },
.cntr = { {4, 5, -1}, {6, 7, -1} },
},
[P4_EVENT_UOP_QUEUE_WRITES] = {
.opcode = P4_OPCODE(P4_EVENT_UOP_QUEUE_WRITES),
.escr_msr = { MSR_P4_MS_ESCR0, MSR_P4_MS_ESCR1 },
.cntr = { {4, 5, -1}, {6, 7, -1} },
},
[P4_EVENT_RETIRED_MISPRED_BRANCH_TYPE] = {
.opcode = P4_OPCODE(P4_EVENT_RETIRED_MISPRED_BRANCH_TYPE),
.escr_msr = { MSR_P4_TBPU_ESCR0 , MSR_P4_TBPU_ESCR0 },
.cntr = { {4, 5, -1}, {6, 7, -1} },
},
[P4_EVENT_RETIRED_BRANCH_TYPE] = {
.opcode = P4_OPCODE(P4_EVENT_RETIRED_BRANCH_TYPE),
.escr_msr = { MSR_P4_TBPU_ESCR0 , MSR_P4_TBPU_ESCR1 },
.cntr = { {4, 5, -1}, {6, 7, -1} },
},
[P4_EVENT_RESOURCE_STALL] = {
.opcode = P4_OPCODE(P4_EVENT_RESOURCE_STALL),
.escr_msr = { MSR_P4_ALF_ESCR0, MSR_P4_ALF_ESCR1 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_WC_BUFFER] = {
.opcode = P4_OPCODE(P4_EVENT_WC_BUFFER),
.escr_msr = { MSR_P4_DAC_ESCR0, MSR_P4_DAC_ESCR1 },
.cntr = { {8, 9, -1}, {10, 11, -1} },
},
[P4_EVENT_B2B_CYCLES] = {
.opcode = P4_OPCODE(P4_EVENT_B2B_CYCLES),
.escr_msr = { MSR_P4_FSB_ESCR0, MSR_P4_FSB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_BNR] = {
.opcode = P4_OPCODE(P4_EVENT_BNR),
.escr_msr = { MSR_P4_FSB_ESCR0, MSR_P4_FSB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_SNOOP] = {
.opcode = P4_OPCODE(P4_EVENT_SNOOP),
.escr_msr = { MSR_P4_FSB_ESCR0, MSR_P4_FSB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_RESPONSE] = {
.opcode = P4_OPCODE(P4_EVENT_RESPONSE),
.escr_msr = { MSR_P4_FSB_ESCR0, MSR_P4_FSB_ESCR1 },
.cntr = { {0, -1, -1}, {2, -1, -1} },
},
[P4_EVENT_FRONT_END_EVENT] = {
.opcode = P4_OPCODE(P4_EVENT_FRONT_END_EVENT),
.escr_msr = { MSR_P4_CRU_ESCR2, MSR_P4_CRU_ESCR3 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_EXECUTION_EVENT] = {
.opcode = P4_OPCODE(P4_EVENT_EXECUTION_EVENT),
.escr_msr = { MSR_P4_CRU_ESCR2, MSR_P4_CRU_ESCR3 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_REPLAY_EVENT] = {
.opcode = P4_OPCODE(P4_EVENT_REPLAY_EVENT),
.escr_msr = { MSR_P4_CRU_ESCR2, MSR_P4_CRU_ESCR3 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_INSTR_RETIRED] = {
.opcode = P4_OPCODE(P4_EVENT_INSTR_RETIRED),
.escr_msr = { MSR_P4_CRU_ESCR0, MSR_P4_CRU_ESCR1 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_UOPS_RETIRED] = {
.opcode = P4_OPCODE(P4_EVENT_UOPS_RETIRED),
.escr_msr = { MSR_P4_CRU_ESCR0, MSR_P4_CRU_ESCR1 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_UOP_TYPE] = {
.opcode = P4_OPCODE(P4_EVENT_UOP_TYPE),
.escr_msr = { MSR_P4_RAT_ESCR0, MSR_P4_RAT_ESCR1 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_BRANCH_RETIRED] = {
.opcode = P4_OPCODE(P4_EVENT_BRANCH_RETIRED),
.escr_msr = { MSR_P4_CRU_ESCR2, MSR_P4_CRU_ESCR3 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_MISPRED_BRANCH_RETIRED] = {
.opcode = P4_OPCODE(P4_EVENT_MISPRED_BRANCH_RETIRED),
.escr_msr = { MSR_P4_CRU_ESCR0, MSR_P4_CRU_ESCR1 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_X87_ASSIST] = {
.opcode = P4_OPCODE(P4_EVENT_X87_ASSIST),
.escr_msr = { MSR_P4_CRU_ESCR2, MSR_P4_CRU_ESCR3 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_MACHINE_CLEAR] = {
.opcode = P4_OPCODE(P4_EVENT_MACHINE_CLEAR),
.escr_msr = { MSR_P4_CRU_ESCR2, MSR_P4_CRU_ESCR3 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
[P4_EVENT_INSTR_COMPLETED] = {
.opcode = P4_OPCODE(P4_EVENT_INSTR_COMPLETED),
.escr_msr = { MSR_P4_CRU_ESCR0, MSR_P4_CRU_ESCR1 },
.cntr = { {12, 13, 16}, {14, 15, 17} },
},
};
#define P4_GEN_CACHE_EVENT(event, bit, cache_event) \
p4_config_pack_escr(P4_ESCR_EVENT(event) | \
P4_ESCR_EMASK_BIT(event, bit)) | \
p4_config_pack_cccr(cache_event | \
P4_CCCR_ESEL(P4_OPCODE_ESEL(P4_OPCODE(event))))
static __initconst const u64 p4_hw_cache_event_ids
[PERF_COUNT_HW_CACHE_MAX]
[PERF_COUNT_HW_CACHE_OP_MAX]
[PERF_COUNT_HW_CACHE_RESULT_MAX] =
{
[ C(L1D ) ] = {
[ C(OP_READ) ] = {
[ C(RESULT_ACCESS) ] = 0x0,
[ C(RESULT_MISS) ] = P4_GEN_CACHE_EVENT(P4_EVENT_REPLAY_EVENT, NBOGUS,
P4_CACHE__1stl_cache_load_miss_retired),
},
},
[ C(LL ) ] = {
[ C(OP_READ) ] = {
[ C(RESULT_ACCESS) ] = 0x0,
[ C(RESULT_MISS) ] = P4_GEN_CACHE_EVENT(P4_EVENT_REPLAY_EVENT, NBOGUS,
P4_CACHE__2ndl_cache_load_miss_retired),
},
},
[ C(DTLB) ] = {
[ C(OP_READ) ] = {
[ C(RESULT_ACCESS) ] = 0x0,
[ C(RESULT_MISS) ] = P4_GEN_CACHE_EVENT(P4_EVENT_REPLAY_EVENT, NBOGUS,
P4_CACHE__dtlb_load_miss_retired),
},
[ C(OP_WRITE) ] = {
[ C(RESULT_ACCESS) ] = 0x0,
[ C(RESULT_MISS) ] = P4_GEN_CACHE_EVENT(P4_EVENT_REPLAY_EVENT, NBOGUS,
P4_CACHE__dtlb_store_miss_retired),
},
},
[ C(ITLB) ] = {
[ C(OP_READ) ] = {
[ C(RESULT_ACCESS) ] = P4_GEN_CACHE_EVENT(P4_EVENT_ITLB_REFERENCE, HIT,
P4_CACHE__itlb_reference_hit),
[ C(RESULT_MISS) ] = P4_GEN_CACHE_EVENT(P4_EVENT_ITLB_REFERENCE, MISS,
P4_CACHE__itlb_reference_miss),
},
[ C(OP_WRITE) ] = {
[ C(RESULT_ACCESS) ] = -1,
[ C(RESULT_MISS) ] = -1,
},
[ C(OP_PREFETCH) ] = {
[ C(RESULT_ACCESS) ] = -1,
[ C(RESULT_MISS) ] = -1,
},
},
};
static u64 p4_general_events[PERF_COUNT_HW_MAX] = {
/* non-halted CPU clocks */
[PERF_COUNT_HW_CPU_CYCLES] =
p4_config_pack_escr(P4_ESCR_EVENT(P4_EVENT_GLOBAL_POWER_EVENTS) |
P4_ESCR_EMASK_BIT(P4_EVENT_GLOBAL_POWER_EVENTS, RUNNING)),
/*
* retired instructions
* in a sake of simplicity we don't use the FSB tagging
*/
[PERF_COUNT_HW_INSTRUCTIONS] =
p4_config_pack_escr(P4_ESCR_EVENT(P4_EVENT_INSTR_RETIRED) |
P4_ESCR_EMASK_BIT(P4_EVENT_INSTR_RETIRED, NBOGUSNTAG) |
P4_ESCR_EMASK_BIT(P4_EVENT_INSTR_RETIRED, BOGUSNTAG)),
/* cache hits */
[PERF_COUNT_HW_CACHE_REFERENCES] =
p4_config_pack_escr(P4_ESCR_EVENT(P4_EVENT_BSQ_CACHE_REFERENCE) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, RD_2ndL_HITS) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, RD_2ndL_HITE) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, RD_2ndL_HITM) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, RD_3rdL_HITS) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, RD_3rdL_HITE) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, RD_3rdL_HITM)),
/* cache misses */
[PERF_COUNT_HW_CACHE_MISSES] =
p4_config_pack_escr(P4_ESCR_EVENT(P4_EVENT_BSQ_CACHE_REFERENCE) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, RD_2ndL_MISS) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, RD_3rdL_MISS) |
P4_ESCR_EMASK_BIT(P4_EVENT_BSQ_CACHE_REFERENCE, WR_2ndL_MISS)),
/* branch instructions retired */
[PERF_COUNT_HW_BRANCH_INSTRUCTIONS] =
p4_config_pack_escr(P4_ESCR_EVENT(P4_EVENT_RETIRED_BRANCH_TYPE) |
P4_ESCR_EMASK_BIT(P4_EVENT_RETIRED_BRANCH_TYPE, CONDITIONAL) |
P4_ESCR_EMASK_BIT(P4_EVENT_RETIRED_BRANCH_TYPE, CALL) |
P4_ESCR_EMASK_BIT(P4_EVENT_RETIRED_BRANCH_TYPE, RETURN) |
P4_ESCR_EMASK_BIT(P4_EVENT_RETIRED_BRANCH_TYPE, INDIRECT)),
/* mispredicted branches retired */
[PERF_COUNT_HW_BRANCH_MISSES] =
p4_config_pack_escr(P4_ESCR_EVENT(P4_EVENT_MISPRED_BRANCH_RETIRED) |
P4_ESCR_EMASK_BIT(P4_EVENT_MISPRED_BRANCH_RETIRED, NBOGUS)),
/* bus ready clocks (cpu is driving #DRDY_DRV\#DRDY_OWN): */
[PERF_COUNT_HW_BUS_CYCLES] =
p4_config_pack_escr(P4_ESCR_EVENT(P4_EVENT_FSB_DATA_ACTIVITY) |
P4_ESCR_EMASK_BIT(P4_EVENT_FSB_DATA_ACTIVITY, DRDY_DRV) |
P4_ESCR_EMASK_BIT(P4_EVENT_FSB_DATA_ACTIVITY, DRDY_OWN)) |
p4_config_pack_cccr(P4_CCCR_EDGE | P4_CCCR_COMPARE),
};
static struct p4_event_bind *p4_config_get_bind(u64 config)
{
unsigned int evnt = p4_config_unpack_event(config);
struct p4_event_bind *bind = NULL;
if (evnt < ARRAY_SIZE(p4_event_bind_map))
bind = &p4_event_bind_map[evnt];
return bind;
}
static u64 p4_pmu_event_map(int hw_event)
{
struct p4_event_bind *bind;
unsigned int esel;
u64 config;
config = p4_general_events[hw_event];
bind = p4_config_get_bind(config);
esel = P4_OPCODE_ESEL(bind->opcode);
config |= p4_config_pack_cccr(P4_CCCR_ESEL(esel));
return config;
}
static int p4_hw_config(struct perf_event *event)
{
int cpu = get_cpu();
int rc = 0;
unsigned int evnt;
u32 escr, cccr;
/*
* the reason we use cpu that early is that: if we get scheduled
* first time on the same cpu -- we will not need swap thread
* specific flags in config (and will save some cpu cycles)
*/
cccr = p4_default_cccr_conf(cpu);
escr = p4_default_escr_conf(cpu, event->attr.exclude_kernel,
event->attr.exclude_user);
event->hw.config = p4_config_pack_escr(escr) |
p4_config_pack_cccr(cccr);
if (p4_ht_active() && p4_ht_thread(cpu))
event->hw.config = p4_set_ht_bit(event->hw.config);
if (event->attr.type == PERF_TYPE_RAW) {
/* user data may have out-of-bound event index */
evnt = p4_config_unpack_event(event->attr.config);
if (evnt >= ARRAY_SIZE(p4_event_bind_map)) {
rc = -EINVAL;
goto out;
}
/*
* We don't control raw events so it's up to the caller
* to pass sane values (and we don't count the thread number
* on HT machine but allow HT-compatible specifics to be
* passed on)
*
* XXX: HT wide things should check perf_paranoid_cpu() &&
* CAP_SYS_ADMIN
*/
event->hw.config |= event->attr.config &
(p4_config_pack_escr(P4_ESCR_MASK_HT) |
p4_config_pack_cccr(P4_CCCR_MASK_HT));
}
rc = x86_setup_perfctr(event);
out:
put_cpu();
return rc;
}
static inline void p4_pmu_clear_cccr_ovf(struct hw_perf_event *hwc)
{
unsigned long dummy;
rdmsrl(hwc->config_base + hwc->idx, dummy);
if (dummy & P4_CCCR_OVF) {
(void)checking_wrmsrl(hwc->config_base + hwc->idx,
((u64)dummy) & ~P4_CCCR_OVF);
}
}
static inline void p4_pmu_disable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
/*
* If event gets disabled while counter is in overflowed
* state we need to clear P4_CCCR_OVF, otherwise interrupt get
* asserted again and again
*/
(void)checking_wrmsrl(hwc->config_base + hwc->idx,
(u64)(p4_config_unpack_cccr(hwc->config)) &
~P4_CCCR_ENABLE & ~P4_CCCR_OVF & ~P4_CCCR_RESERVED);
}
static void p4_pmu_disable_all(void)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
struct perf_event *event = cpuc->events[idx];
if (!test_bit(idx, cpuc->active_mask))
continue;
p4_pmu_disable_event(event);
}
}
static void p4_pmu_enable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
int thread = p4_ht_config_thread(hwc->config);
u64 escr_conf = p4_config_unpack_escr(p4_clear_ht_bit(hwc->config));
unsigned int idx = p4_config_unpack_event(hwc->config);
unsigned int idx_cache = p4_config_unpack_cache_event(hwc->config);
struct p4_event_bind *bind;
struct p4_cache_event_bind *bind_cache;
u64 escr_addr, cccr;
bind = &p4_event_bind_map[idx];
escr_addr = (u64)bind->escr_msr[thread];
/*
* - we dont support cascaded counters yet
* - and counter 1 is broken (erratum)
*/
WARN_ON_ONCE(p4_is_event_cascaded(hwc->config));
WARN_ON_ONCE(hwc->idx == 1);
/* we need a real Event value */
escr_conf &= ~P4_ESCR_EVENT_MASK;
escr_conf |= P4_ESCR_EVENT(P4_OPCODE_EVNT(bind->opcode));
cccr = p4_config_unpack_cccr(hwc->config);
/*
* it could be Cache event so that we need to
* set metrics into additional MSRs
*/
BUILD_BUG_ON(P4_CACHE__MAX > P4_CCCR_CACHE_OPS_MASK);
if (idx_cache > P4_CACHE__NONE &&
idx_cache < ARRAY_SIZE(p4_cache_event_bind_map)) {
bind_cache = &p4_cache_event_bind_map[idx_cache];
(void)checking_wrmsrl(MSR_IA32_PEBS_ENABLE, (u64)bind_cache->metric_pebs);
(void)checking_wrmsrl(MSR_P4_PEBS_MATRIX_VERT, (u64)bind_cache->metric_vert);
}
(void)checking_wrmsrl(escr_addr, escr_conf);
(void)checking_wrmsrl(hwc->config_base + hwc->idx,
(cccr & ~P4_CCCR_RESERVED) | P4_CCCR_ENABLE);
}
static void p4_pmu_enable_all(int added)
{
struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
int idx;
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
struct perf_event *event = cpuc->events[idx];
if (!test_bit(idx, cpuc->active_mask))
continue;
p4_pmu_enable_event(event);
}
}
static int p4_pmu_handle_irq(struct pt_regs *regs)
{
struct perf_sample_data data;
struct cpu_hw_events *cpuc;
struct perf_event *event;
struct hw_perf_event *hwc;
int idx, handled = 0;
u64 val;
data.addr = 0;
data.raw = NULL;
cpuc = &__get_cpu_var(cpu_hw_events);
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
if (!test_bit(idx, cpuc->active_mask))
continue;
event = cpuc->events[idx];
hwc = &event->hw;
WARN_ON_ONCE(hwc->idx != idx);
/*
* FIXME: Redundant call, actually not needed
* but just to check if we're screwed
*/
p4_pmu_clear_cccr_ovf(hwc);
val = x86_perf_event_update(event);
if (val & (1ULL << (x86_pmu.cntval_bits - 1)))
continue;
/*
* event overflow
*/
handled = 1;
data.period = event->hw.last_period;
if (!x86_perf_event_set_period(event))
continue;
if (perf_event_overflow(event, 1, &data, regs))
p4_pmu_disable_event(event);
}
if (handled) {
/* p4 quirk: unmask it again */
apic_write(APIC_LVTPC, apic_read(APIC_LVTPC) & ~APIC_LVT_MASKED);
inc_irq_stat(apic_perf_irqs);
}
return handled;
}
/*
* swap thread specific fields according to a thread
* we are going to run on
*/
static void p4_pmu_swap_config_ts(struct hw_perf_event *hwc, int cpu)
{
u32 escr, cccr;
/*
* we either lucky and continue on same cpu or no HT support
*/
if (!p4_should_swap_ts(hwc->config, cpu))
return;
/*
* the event is migrated from an another logical
* cpu, so we need to swap thread specific flags
*/
escr = p4_config_unpack_escr(hwc->config);
cccr = p4_config_unpack_cccr(hwc->config);
if (p4_ht_thread(cpu)) {
cccr &= ~P4_CCCR_OVF_PMI_T0;
cccr |= P4_CCCR_OVF_PMI_T1;
if (escr & P4_ESCR_T0_OS) {
escr &= ~P4_ESCR_T0_OS;
escr |= P4_ESCR_T1_OS;
}
if (escr & P4_ESCR_T0_USR) {
escr &= ~P4_ESCR_T0_USR;
escr |= P4_ESCR_T1_USR;
}
hwc->config = p4_config_pack_escr(escr);
hwc->config |= p4_config_pack_cccr(cccr);
hwc->config |= P4_CONFIG_HT;
} else {
cccr &= ~P4_CCCR_OVF_PMI_T1;
cccr |= P4_CCCR_OVF_PMI_T0;
if (escr & P4_ESCR_T1_OS) {
escr &= ~P4_ESCR_T1_OS;
escr |= P4_ESCR_T0_OS;
}
if (escr & P4_ESCR_T1_USR) {
escr &= ~P4_ESCR_T1_USR;
escr |= P4_ESCR_T0_USR;
}
hwc->config = p4_config_pack_escr(escr);
hwc->config |= p4_config_pack_cccr(cccr);
hwc->config &= ~P4_CONFIG_HT;
}
}
/*
* ESCR address hashing is tricky, ESCRs are not sequential
* in memory but all starts from MSR_P4_BSU_ESCR0 (0x03e0) and
* the metric between any ESCRs is laid in range [0xa0,0xe1]
*
* so we make ~70% filled hashtable
*/
#define P4_ESCR_MSR_BASE 0x000003a0
#define P4_ESCR_MSR_MAX 0x000003e1
#define P4_ESCR_MSR_TABLE_SIZE (P4_ESCR_MSR_MAX - P4_ESCR_MSR_BASE + 1)
#define P4_ESCR_MSR_IDX(msr) (msr - P4_ESCR_MSR_BASE)
#define P4_ESCR_MSR_TABLE_ENTRY(msr) [P4_ESCR_MSR_IDX(msr)] = msr
static const unsigned int p4_escr_table[P4_ESCR_MSR_TABLE_SIZE] = {
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_ALF_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_ALF_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_BPU_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_BPU_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_BSU_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_BSU_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_CRU_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_CRU_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_CRU_ESCR2),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_CRU_ESCR3),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_CRU_ESCR4),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_CRU_ESCR5),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_DAC_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_DAC_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_FIRM_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_FIRM_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_FLAME_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_FLAME_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_FSB_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_FSB_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_IQ_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_IQ_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_IS_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_IS_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_ITLB_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_ITLB_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_IX_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_IX_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_MOB_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_MOB_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_MS_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_MS_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_PMH_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_PMH_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_RAT_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_RAT_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_SAAT_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_SAAT_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_SSU_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_SSU_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_TBPU_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_TBPU_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_TC_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_TC_ESCR1),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_U2L_ESCR0),
P4_ESCR_MSR_TABLE_ENTRY(MSR_P4_U2L_ESCR1),
};
static int p4_get_escr_idx(unsigned int addr)
{
unsigned int idx = P4_ESCR_MSR_IDX(addr);
if (unlikely(idx >= P4_ESCR_MSR_TABLE_SIZE ||
!p4_escr_table[idx])) {
WARN_ONCE(1, "P4 PMU: Wrong address passed: %x\n", addr);
return -1;
}
return idx;
}
static int p4_next_cntr(int thread, unsigned long *used_mask,
struct p4_event_bind *bind)
{
int i, j;
for (i = 0; i < P4_CNTR_LIMIT; i++) {
j = bind->cntr[thread][i];
if (j != -1 && !test_bit(j, used_mask))
return j;
}
return -1;
}
static int p4_pmu_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign)
{
unsigned long used_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
unsigned long escr_mask[BITS_TO_LONGS(P4_ESCR_MSR_TABLE_SIZE)];
int cpu = raw_smp_processor_id();
struct hw_perf_event *hwc;
struct p4_event_bind *bind;
unsigned int i, thread, num;
int cntr_idx, escr_idx;
bitmap_zero(used_mask, X86_PMC_IDX_MAX);
bitmap_zero(escr_mask, P4_ESCR_MSR_TABLE_SIZE);
for (i = 0, num = n; i < n; i++, num--) {
hwc = &cpuc->event_list[i]->hw;
thread = p4_ht_thread(cpu);
bind = p4_config_get_bind(hwc->config);
escr_idx = p4_get_escr_idx(bind->escr_msr[thread]);
if (unlikely(escr_idx == -1))
goto done;
if (hwc->idx != -1 && !p4_should_swap_ts(hwc->config, cpu)) {
cntr_idx = hwc->idx;
if (assign)
assign[i] = hwc->idx;
goto reserve;
}
cntr_idx = p4_next_cntr(thread, used_mask, bind);
if (cntr_idx == -1 || test_bit(escr_idx, escr_mask))
goto done;
p4_pmu_swap_config_ts(hwc, cpu);
if (assign)
assign[i] = cntr_idx;
reserve:
set_bit(cntr_idx, used_mask);
set_bit(escr_idx, escr_mask);
}
done:
return num ? -ENOSPC : 0;
}
static __initconst const struct x86_pmu p4_pmu = {
.name = "Netburst P4/Xeon",
.handle_irq = p4_pmu_handle_irq,
.disable_all = p4_pmu_disable_all,
.enable_all = p4_pmu_enable_all,
.enable = p4_pmu_enable_event,
.disable = p4_pmu_disable_event,
.eventsel = MSR_P4_BPU_CCCR0,
.perfctr = MSR_P4_BPU_PERFCTR0,
.event_map = p4_pmu_event_map,
.max_events = ARRAY_SIZE(p4_general_events),
.get_event_constraints = x86_get_event_constraints,
/*
* IF HT disabled we may need to use all
* ARCH_P4_MAX_CCCR counters simulaneously
* though leave it restricted at moment assuming
* HT is on
*/
.num_counters = ARCH_P4_MAX_CCCR,
.apic = 1,
.cntval_bits = 40,
.cntval_mask = (1ULL << 40) - 1,
.max_period = (1ULL << 39) - 1,
.hw_config = p4_hw_config,
.schedule_events = p4_pmu_schedule_events,
};
static __init int p4_pmu_init(void)
{
unsigned int low, high;
/* If we get stripped -- indexig fails */
BUILD_BUG_ON(ARCH_P4_MAX_CCCR > X86_PMC_MAX_GENERIC);
rdmsr(MSR_IA32_MISC_ENABLE, low, high);
if (!(low & (1 << 7))) {
pr_cont("unsupported Netburst CPU model %d ",
boot_cpu_data.x86_model);
return -ENODEV;
}
memcpy(hw_cache_event_ids, p4_hw_cache_event_ids,
sizeof(hw_cache_event_ids));
pr_cont("Netburst events, ");
x86_pmu = p4_pmu;
return 0;
}
#endif /* CONFIG_CPU_SUP_INTEL */

31
arch/x86/kernel/cpu/perf_event_p6.c
View file

@ -27,24 +27,6 @@ static u64 p6_pmu_event_map(int hw_event)
*/ */
#define P6_NOP_EVENT 0x0000002EULL #define P6_NOP_EVENT 0x0000002EULL
static u64 p6_pmu_raw_event(u64 hw_event)
{
#define P6_EVNTSEL_EVENT_MASK 0x000000FFULL
#define P6_EVNTSEL_UNIT_MASK 0x0000FF00ULL
#define P6_EVNTSEL_EDGE_MASK 0x00040000ULL
#define P6_EVNTSEL_INV_MASK 0x00800000ULL
#define P6_EVNTSEL_REG_MASK 0xFF000000ULL
#define P6_EVNTSEL_MASK \
(P6_EVNTSEL_EVENT_MASK | \
P6_EVNTSEL_UNIT_MASK | \
P6_EVNTSEL_EDGE_MASK | \
P6_EVNTSEL_INV_MASK | \
P6_EVNTSEL_REG_MASK)
return hw_event & P6_EVNTSEL_MASK;
}
static struct event_constraint p6_event_constraints[] = static struct event_constraint p6_event_constraints[] =
{ {
INTEL_EVENT_CONSTRAINT(0xc1, 0x1), /* FLOPS */ INTEL_EVENT_CONSTRAINT(0xc1, 0x1), /* FLOPS */
@ -66,7 +48,7 @@ static void p6_pmu_disable_all(void)
wrmsrl(MSR_P6_EVNTSEL0, val); wrmsrl(MSR_P6_EVNTSEL0, val);
} }
static void p6_pmu_enable_all(void) static void p6_pmu_enable_all(int added)
{ {
unsigned long val; unsigned long val;
@ -102,22 +84,23 @@ static void p6_pmu_enable_event(struct perf_event *event)
(void)checking_wrmsrl(hwc->config_base + hwc->idx, val); (void)checking_wrmsrl(hwc->config_base + hwc->idx, val);
} }
static __initconst struct x86_pmu p6_pmu = { static __initconst const struct x86_pmu p6_pmu = {
.name = "p6", .name = "p6",
.handle_irq = x86_pmu_handle_irq, .handle_irq = x86_pmu_handle_irq,
.disable_all = p6_pmu_disable_all, .disable_all = p6_pmu_disable_all,
.enable_all = p6_pmu_enable_all, .enable_all = p6_pmu_enable_all,
.enable = p6_pmu_enable_event, .enable = p6_pmu_enable_event,
.disable = p6_pmu_disable_event, .disable = p6_pmu_disable_event,
.hw_config = x86_pmu_hw_config,
.schedule_events = x86_schedule_events,
.eventsel = MSR_P6_EVNTSEL0, .eventsel = MSR_P6_EVNTSEL0,
.perfctr = MSR_P6_PERFCTR0, .perfctr = MSR_P6_PERFCTR0,
.event_map = p6_pmu_event_map, .event_map = p6_pmu_event_map,
.raw_event = p6_pmu_raw_event,
.max_events = ARRAY_SIZE(p6_perfmon_event_map), .max_events = ARRAY_SIZE(p6_perfmon_event_map),
.apic = 1, .apic = 1,
.max_period = (1ULL << 31) - 1, .max_period = (1ULL << 31) - 1,
.version = 0, .version = 0,
.num_events = 2, .num_counters = 2,
/* /*
* Events have 40 bits implemented. However they are designed such * Events have 40 bits implemented. However they are designed such
* that bits [32-39] are sign extensions of bit 31. As such the * that bits [32-39] are sign extensions of bit 31. As such the
@ -125,8 +108,8 @@ static __initconst struct x86_pmu p6_pmu = {
* *
* See IA-32 Intel Architecture Software developer manual Vol 3B * See IA-32 Intel Architecture Software developer manual Vol 3B
*/ */
.event_bits = 32, .cntval_bits = 32,
.event_mask = (1ULL << 32) - 1, .cntval_mask = (1ULL << 32) - 1,
.get_event_constraints = x86_get_event_constraints, .get_event_constraints = x86_get_event_constraints,
.event_constraints = p6_event_constraints, .event_constraints = p6_event_constraints,
}; };

1437
arch/x86/kernel/ds.c
View file

File diff suppressed because it is too large Load diff

408
arch/x86/kernel/ds_selftest.c
View file

@ -1,408 +0,0 @@
/*
* Debug Store support - selftest
*
*
* Copyright (C) 2009 Intel Corporation.
* Markus Metzger <markus.t.metzger@intel.com>, 2009
*/
#include "ds_selftest.h"
#include <linux/kernel.h>
#include <linux/string.h>
#include <linux/smp.h>
#include <linux/cpu.h>
#include <asm/ds.h>
#define BUFFER_SIZE 521 /* Intentionally chose an odd size. */
#define SMALL_BUFFER_SIZE 24 /* A single bts entry. */
struct ds_selftest_bts_conf {
struct bts_tracer *tracer;
int error;
int (*suspend)(struct bts_tracer *);
int (*resume)(struct bts_tracer *);
};
static int ds_selftest_bts_consistency(const struct bts_trace *trace)
{
int error = 0;
if (!trace) {
printk(KERN_CONT "failed to access trace...");
/* Bail out. Other tests are pointless. */
return -1;
}
if (!trace->read) {
printk(KERN_CONT "bts read not available...");
error = -1;
}
/* Do some sanity checks on the trace configuration. */
if (!trace->ds.n) {
printk(KERN_CONT "empty bts buffer...");
error = -1;
}
if (!trace->ds.size) {
printk(KERN_CONT "bad bts trace setup...");
error = -1;
}
if (trace->ds.end !=
(char *)trace->ds.begin + (trace->ds.n * trace->ds.size)) {
printk(KERN_CONT "bad bts buffer setup...");
error = -1;
}
/*
* We allow top in [begin; end], since its not clear when the
* overflow adjustment happens: after the increment or before the
* write.
*/
if ((trace->ds.top < trace->ds.begin) ||
(trace->ds.end < trace->ds.top)) {
printk(KERN_CONT "bts top out of bounds...");
error = -1;
}
return error;
}
static int ds_selftest_bts_read(struct bts_tracer *tracer,
const struct bts_trace *trace,
const void *from, const void *to)
{
const unsigned char *at;
/*
* Check a few things which do not belong to this test.
* They should be covered by other tests.
*/
if (!trace)
return -1;
if (!trace->read)
return -1;
if (to < from)
return -1;
if (from < trace->ds.begin)
return -1;
if (trace->ds.end < to)
return -1;
if (!trace->ds.size)
return -1;
/* Now to the test itself. */
for (at = from; (void *)at < to; at += trace->ds.size) {
struct bts_struct bts;
unsigned long index;
int error;
if (((void *)at - trace->ds.begin) % trace->ds.size) {
printk(KERN_CONT
"read from non-integer index...");
return -1;
}
index = ((void *)at - trace->ds.begin) / trace->ds.size;
memset(&bts, 0, sizeof(bts));
error = trace->read(tracer, at, &bts);
if (error < 0) {
printk(KERN_CONT
"error reading bts trace at [%lu] (0x%p)...",
index, at);
return error;
}
switch (bts.qualifier) {
case BTS_BRANCH:
break;
default:
printk(KERN_CONT
"unexpected bts entry %llu at [%lu] (0x%p)...",
bts.qualifier, index, at);
return -1;
}
}
return 0;
}
static void ds_selftest_bts_cpu(void *arg)
{
struct ds_selftest_bts_conf *conf = arg;
const struct bts_trace *trace;
void *top;
if (IS_ERR(conf->tracer)) {
conf->error = PTR_ERR(conf->tracer);
conf->tracer = NULL;
printk(KERN_CONT
"initialization failed (err: %d)...", conf->error);
return;
}
/* We should meanwhile have enough trace. */
conf->error = conf->suspend(conf->tracer);
if (conf->error < 0)
return;
/* Let's see if we can access the trace. */
trace = ds_read_bts(conf->tracer);
conf->error = ds_selftest_bts_consistency(trace);
if (conf->error < 0)
return;
/* If everything went well, we should have a few trace entries. */
if (trace->ds.top == trace->ds.begin) {
/*
* It is possible but highly unlikely that we got a
* buffer overflow and end up at exactly the same
* position we started from.
* Let's issue a warning, but continue.
*/
printk(KERN_CONT "no trace/overflow...");
}
/* Let's try to read the trace we collected. */
conf->error =
ds_selftest_bts_read(conf->tracer, trace,
trace->ds.begin, trace->ds.top);
if (conf->error < 0)
return;
/*
* Let's read the trace again.
* Since we suspended tracing, we should get the same result.
*/
top = trace->ds.top;
trace = ds_read_bts(conf->tracer);
conf->error = ds_selftest_bts_consistency(trace);
if (conf->error < 0)
return;
if (top != trace->ds.top) {
printk(KERN_CONT "suspend not working...");
conf->error = -1;
return;
}
/* Let's collect some more trace - see if resume is working. */
conf->error = conf->resume(conf->tracer);
if (conf->error < 0)
return;
conf->error = conf->suspend(conf->tracer);
if (conf->error < 0)
return;
trace = ds_read_bts(conf->tracer);
conf->error = ds_selftest_bts_consistency(trace);
if (conf->error < 0)
return;
if (trace->ds.top == top) {
/*
* It is possible but highly unlikely that we got a
* buffer overflow and end up at exactly the same
* position we started from.
* Let's issue a warning and check the full trace.
*/
printk(KERN_CONT
"no resume progress/overflow...");
conf->error =
ds_selftest_bts_read(conf->tracer, trace,
trace->ds.begin, trace->ds.end);
} else if (trace->ds.top < top) {
/*
* We had a buffer overflow - the entire buffer should
* contain trace records.
*/
conf->error =
ds_selftest_bts_read(conf->tracer, trace,
trace->ds.begin, trace->ds.end);
} else {
/*
* It is quite likely that the buffer did not overflow.
* Let's just check the delta trace.
*/
conf->error =
ds_selftest_bts_read(conf->tracer, trace, top,
trace->ds.top);
}
if (conf->error < 0)
return;
conf->error = 0;
}
static int ds_suspend_bts_wrap(struct bts_tracer *tracer)
{
ds_suspend_bts(tracer);
return 0;
}
static int ds_resume_bts_wrap(struct bts_tracer *tracer)
{
ds_resume_bts(tracer);
return 0;
}
static void ds_release_bts_noirq_wrap(void *tracer)
{
(void)ds_release_bts_noirq(tracer);
}
static int ds_selftest_bts_bad_release_noirq(int cpu,
struct bts_tracer *tracer)
{
int error = -EPERM;
/* Try to release the tracer on the wrong cpu. */
get_cpu();
if (cpu != smp_processor_id()) {
error = ds_release_bts_noirq(tracer);
if (error != -EPERM)
printk(KERN_CONT "release on wrong cpu...");
}
put_cpu();
return error ? 0 : -1;
}
static int ds_selftest_bts_bad_request_cpu(int cpu, void *buffer)
{
struct bts_tracer *tracer;
int error;
/* Try to request cpu tracing while task tracing is active. */
tracer = ds_request_bts_cpu(cpu, buffer, BUFFER_SIZE, NULL,
(size_t)-1, BTS_KERNEL);
error = PTR_ERR(tracer);
if (!IS_ERR(tracer)) {
ds_release_bts(tracer);
error = 0;
}
if (error != -EPERM)
printk(KERN_CONT "cpu/task tracing overlap...");
return error ? 0 : -1;
}
static int ds_selftest_bts_bad_request_task(void *buffer)
{
struct bts_tracer *tracer;
int error;
/* Try to request cpu tracing while task tracing is active. */
tracer = ds_request_bts_task(current, buffer, BUFFER_SIZE, NULL,
(size_t)-1, BTS_KERNEL);
error = PTR_ERR(tracer);
if (!IS_ERR(tracer)) {
error = 0;
ds_release_bts(tracer);
}
if (error != -EPERM)
printk(KERN_CONT "task/cpu tracing overlap...");
return error ? 0 : -1;
}
int ds_selftest_bts(void)
{
struct ds_selftest_bts_conf conf;
unsigned char buffer[BUFFER_SIZE], *small_buffer;
unsigned long irq;
int cpu;
printk(KERN_INFO "[ds] bts selftest...");
conf.error = 0;
small_buffer = (unsigned char *)ALIGN((unsigned long)buffer, 8) + 8;
get_online_cpus();
for_each_online_cpu(cpu) {
conf.suspend = ds_suspend_bts_wrap;
conf.resume = ds_resume_bts_wrap;
conf.tracer =
ds_request_bts_cpu(cpu, buffer, BUFFER_SIZE,
NULL, (size_t)-1, BTS_KERNEL);
ds_selftest_bts_cpu(&conf);
if (conf.error >= 0)
conf.error = ds_selftest_bts_bad_request_task(buffer);
ds_release_bts(conf.tracer);
if (conf.error < 0)
goto out;
conf.suspend = ds_suspend_bts_noirq;
conf.resume = ds_resume_bts_noirq;
conf.tracer =
ds_request_bts_cpu(cpu, buffer, BUFFER_SIZE,
NULL, (size_t)-1, BTS_KERNEL);
smp_call_function_single(cpu, ds_selftest_bts_cpu, &conf, 1);
if (conf.error >= 0) {
conf.error =
ds_selftest_bts_bad_release_noirq(cpu,
conf.tracer);
/* We must not release the tracer twice. */
if (conf.error < 0)
conf.tracer = NULL;
}
if (conf.error >= 0)
conf.error = ds_selftest_bts_bad_request_task(buffer);
smp_call_function_single(cpu, ds_release_bts_noirq_wrap,
conf.tracer, 1);
if (conf.error < 0)
goto out;
}
conf.suspend = ds_suspend_bts_wrap;
conf.resume = ds_resume_bts_wrap;
conf.tracer =
ds_request_bts_task(current, buffer, BUFFER_SIZE,
NULL, (size_t)-1, BTS_KERNEL);
ds_selftest_bts_cpu(&conf);
if (conf.error >= 0)
conf.error = ds_selftest_bts_bad_request_cpu(0, buffer);
ds_release_bts(conf.tracer);
if (conf.error < 0)
goto out;
conf.suspend = ds_suspend_bts_noirq;
conf.resume = ds_resume_bts_noirq;
conf.tracer =
ds_request_bts_task(current, small_buffer, SMALL_BUFFER_SIZE,
NULL, (size_t)-1, BTS_KERNEL);
local_irq_save(irq);
ds_selftest_bts_cpu(&conf);
if (conf.error >= 0)
conf.error = ds_selftest_bts_bad_request_cpu(0, buffer);
ds_release_bts_noirq(conf.tracer);
local_irq_restore(irq);
if (conf.error < 0)
goto out;
conf.error = 0;
out:
put_online_cpus();
printk(KERN_CONT "%s.\n", (conf.error ? "failed" : "passed"));
return conf.error;
}
int ds_selftest_pebs(void)
{
return 0;
}

15
arch/x86/kernel/ds_selftest.h
View file

@ -1,15 +0,0 @@
/*
* Debug Store support - selftest
*
*
* Copyright (C) 2009 Intel Corporation.
* Markus Metzger <markus.t.metzger@intel.com>, 2009
*/
#ifdef CONFIG_X86_DS_SELFTEST
extern int ds_selftest_bts(void);
extern int ds_selftest_pebs(void);
#else
static inline int ds_selftest_bts(void) { return 0; }
static inline int ds_selftest_pebs(void) { return 0; }
#endif

5
arch/x86/kernel/dumpstack.c
View file

@ -224,11 +224,6 @@ unsigned __kprobes long oops_begin(void)
int cpu; int cpu;
unsigned long flags; unsigned long flags;
/* notify the hw-branch tracer so it may disable tracing and
add the last trace to the trace buffer -
the earlier this happens, the more useful the trace. */
trace_hw_branch_oops();
oops_enter(); oops_enter();
/* racy, but better than risking deadlock. */ /* racy, but better than risking deadlock. */

41
arch/x86/kernel/hw_breakpoint.c
View file

@ -188,26 +188,17 @@ static int get_hbp_len(u8 hbp_len)
return len_in_bytes; return len_in_bytes;
} }
/*
* Check for virtual address in user space.
*/
int arch_check_va_in_userspace(unsigned long va, u8 hbp_len)
{
unsigned int len;
len = get_hbp_len(hbp_len);
return (va <= TASK_SIZE - len);
}
/* /*
* Check for virtual address in kernel space. * Check for virtual address in kernel space.
*/ */
static int arch_check_va_in_kernelspace(unsigned long va, u8 hbp_len) int arch_check_bp_in_kernelspace(struct perf_event *bp)
{ {
unsigned int len; unsigned int len;
unsigned long va;
struct arch_hw_breakpoint *info = counter_arch_bp(bp);
len = get_hbp_len(hbp_len); va = info->address;
len = get_hbp_len(info->len);
return (va >= TASK_SIZE) && ((va + len - 1) >= TASK_SIZE); return (va >= TASK_SIZE) && ((va + len - 1) >= TASK_SIZE);
} }
@ -300,8 +291,7 @@ static int arch_build_bp_info(struct perf_event *bp)
/* /*
* Validate the arch-specific HW Breakpoint register settings * Validate the arch-specific HW Breakpoint register settings
*/ */
int arch_validate_hwbkpt_settings(struct perf_event *bp, int arch_validate_hwbkpt_settings(struct perf_event *bp)
struct task_struct *tsk)
{ {
struct arch_hw_breakpoint *info = counter_arch_bp(bp); struct arch_hw_breakpoint *info = counter_arch_bp(bp);
unsigned int align; unsigned int align;
@ -314,16 +304,6 @@ int arch_validate_hwbkpt_settings(struct perf_event *bp,
ret = -EINVAL; ret = -EINVAL;
if (info->type == X86_BREAKPOINT_EXECUTE)
/*
* Ptrace-refactoring code
* For now, we'll allow instruction breakpoint only for user-space
* addresses
*/
if ((!arch_check_va_in_userspace(info->address, info->len)) &&
info->len != X86_BREAKPOINT_EXECUTE)
return ret;
switch (info->len) { switch (info->len) {
case X86_BREAKPOINT_LEN_1: case X86_BREAKPOINT_LEN_1:
align = 0; align = 0;
@ -350,15 +330,6 @@ int arch_validate_hwbkpt_settings(struct perf_event *bp,
if (info->address & align) if (info->address & align)
return -EINVAL; return -EINVAL;
/* Check that the virtual address is in the proper range */
if (tsk) {
if (!arch_check_va_in_userspace(info->address, info->len))
return -EFAULT;
} else {
if (!arch_check_va_in_kernelspace(info->address, info->len))
return -EFAULT;
}
return 0; return 0;
} }

16
arch/x86/kernel/kprobes.c
View file

@ -422,14 +422,22 @@ static void __kprobes set_current_kprobe(struct kprobe *p, struct pt_regs *regs,
static void __kprobes clear_btf(void) static void __kprobes clear_btf(void)
{ {
if (test_thread_flag(TIF_DEBUGCTLMSR)) if (test_thread_flag(TIF_BLOCKSTEP)) {
update_debugctlmsr(0); unsigned long debugctl = get_debugctlmsr();
debugctl &= ~DEBUGCTLMSR_BTF;
update_debugctlmsr(debugctl);
}
} }
static void __kprobes restore_btf(void) static void __kprobes restore_btf(void)
{ {
if (test_thread_flag(TIF_DEBUGCTLMSR)) if (test_thread_flag(TIF_BLOCKSTEP)) {
update_debugctlmsr(current->thread.debugctlmsr); unsigned long debugctl = get_debugctlmsr();
debugctl |= DEBUGCTLMSR_BTF;
update_debugctlmsr(debugctl);
}
} }
void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,

18
arch/x86/kernel/process.c
View file

@ -20,7 +20,6 @@
#include <asm/idle.h> #include <asm/idle.h>
#include <asm/uaccess.h> #include <asm/uaccess.h>
#include <asm/i387.h> #include <asm/i387.h>
#include <asm/ds.h>
#include <asm/debugreg.h> #include <asm/debugreg.h>
unsigned long idle_halt; unsigned long idle_halt;
@ -50,8 +49,6 @@ void free_thread_xstate(struct task_struct *tsk)
kmem_cache_free(task_xstate_cachep, tsk->thread.xstate); kmem_cache_free(task_xstate_cachep, tsk->thread.xstate);
tsk->thread.xstate = NULL; tsk->thread.xstate = NULL;
} }
WARN(tsk->thread.ds_ctx, "leaking DS context\n");
} }
void free_thread_info(struct thread_info *ti) void free_thread_info(struct thread_info *ti)
@ -198,11 +195,16 @@ void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p,
prev = &prev_p->thread; prev = &prev_p->thread;
next = &next_p->thread; next = &next_p->thread;
if (test_tsk_thread_flag(next_p, TIF_DS_AREA_MSR) || if (test_tsk_thread_flag(prev_p, TIF_BLOCKSTEP) ^
test_tsk_thread_flag(prev_p, TIF_DS_AREA_MSR)) test_tsk_thread_flag(next_p, TIF_BLOCKSTEP)) {
ds_switch_to(prev_p, next_p); unsigned long debugctl = get_debugctlmsr();
else if (next->debugctlmsr != prev->debugctlmsr)
update_debugctlmsr(next->debugctlmsr); debugctl &= ~DEBUGCTLMSR_BTF;
if (test_tsk_thread_flag(next_p, TIF_BLOCKSTEP))
debugctl |= DEBUGCTLMSR_BTF;
update_debugctlmsr(debugctl);
}
if (test_tsk_thread_flag(prev_p, TIF_NOTSC) ^ if (test_tsk_thread_flag(prev_p, TIF_NOTSC) ^
test_tsk_thread_flag(next_p, TIF_NOTSC)) { test_tsk_thread_flag(next_p, TIF_NOTSC)) {

8
arch/x86/kernel/process_32.c
View file

@ -55,7 +55,6 @@
#include <asm/cpu.h> #include <asm/cpu.h>
#include <asm/idle.h> #include <asm/idle.h>
#include <asm/syscalls.h> #include <asm/syscalls.h>
#include <asm/ds.h>
#include <asm/debugreg.h> #include <asm/debugreg.h>
asmlinkage void ret_from_fork(void) __asm__("ret_from_fork"); asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");
@ -238,13 +237,6 @@ int copy_thread(unsigned long clone_flags, unsigned long sp,
kfree(p->thread.io_bitmap_ptr); kfree(p->thread.io_bitmap_ptr);
p->thread.io_bitmap_max = 0; p->thread.io_bitmap_max = 0;
} }
clear_tsk_thread_flag(p, TIF_DS_AREA_MSR);
p->thread.ds_ctx = NULL;
clear_tsk_thread_flag(p, TIF_DEBUGCTLMSR);
p->thread.debugctlmsr = 0;
return err; return err;
} }

8
arch/x86/kernel/process_64.c
View file

@ -49,7 +49,6 @@
#include <asm/ia32.h> #include <asm/ia32.h>
#include <asm/idle.h> #include <asm/idle.h>
#include <asm/syscalls.h> #include <asm/syscalls.h>
#include <asm/ds.h>
#include <asm/debugreg.h> #include <asm/debugreg.h>
asmlinkage extern void ret_from_fork(void); asmlinkage extern void ret_from_fork(void);
@ -313,13 +312,6 @@ int copy_thread(unsigned long clone_flags, unsigned long sp,
if (err) if (err)
goto out; goto out;
} }
clear_tsk_thread_flag(p, TIF_DS_AREA_MSR);
p->thread.ds_ctx = NULL;
clear_tsk_thread_flag(p, TIF_DEBUGCTLMSR);
p->thread.debugctlmsr = 0;
err = 0; err = 0;
out: out:
if (err && p->thread.io_bitmap_ptr) { if (err && p->thread.io_bitmap_ptr) {

384
arch/x86/kernel/ptrace.c
View file

@ -2,9 +2,6 @@
/* /*
* Pentium III FXSR, SSE support * Pentium III FXSR, SSE support
* Gareth Hughes <gareth@valinux.com>, May 2000 * Gareth Hughes <gareth@valinux.com>, May 2000
*
* BTS tracing
* Markus Metzger <markus.t.metzger@intel.com>, Dec 2007
*/ */
#include <linux/kernel.h> #include <linux/kernel.h>
@ -22,7 +19,6 @@
#include <linux/audit.h> #include <linux/audit.h>
#include <linux/seccomp.h> #include <linux/seccomp.h>
#include <linux/signal.h> #include <linux/signal.h>
#include <linux/workqueue.h>
#include <linux/perf_event.h> #include <linux/perf_event.h>
#include <linux/hw_breakpoint.h> #include <linux/hw_breakpoint.h>
@ -36,7 +32,6 @@
#include <asm/desc.h> #include <asm/desc.h>
#include <asm/prctl.h> #include <asm/prctl.h>
#include <asm/proto.h> #include <asm/proto.h>
#include <asm/ds.h>
#include <asm/hw_breakpoint.h> #include <asm/hw_breakpoint.h>
#include "tls.h" #include "tls.h"
@ -693,7 +688,7 @@ static int ptrace_set_breakpoint_addr(struct task_struct *tsk, int nr,
struct perf_event_attr attr; struct perf_event_attr attr;
if (!t->ptrace_bps[nr]) { if (!t->ptrace_bps[nr]) {
hw_breakpoint_init(&attr); ptrace_breakpoint_init(&attr);
/* /*
* Put stub len and type to register (reserve) an inactive but * Put stub len and type to register (reserve) an inactive but
* correct bp * correct bp
@ -789,342 +784,6 @@ static int ioperm_get(struct task_struct *target,
0, IO_BITMAP_BYTES); 0, IO_BITMAP_BYTES);
} }
#ifdef CONFIG_X86_PTRACE_BTS
/*
* A branch trace store context.
*
* Contexts may only be installed by ptrace_bts_config() and only for
* ptraced tasks.
*
* Contexts are destroyed when the tracee is detached from the tracer.
* The actual destruction work requires interrupts enabled, so the
* work is deferred and will be scheduled during __ptrace_unlink().
*
* Contexts hold an additional task_struct reference on the traced
* task, as well as a reference on the tracer's mm.
*
* Ptrace already holds a task_struct for the duration of ptrace operations,
* but since destruction is deferred, it may be executed after both
* tracer and tracee exited.
*/
struct bts_context {
/* The branch trace handle. */
struct bts_tracer *tracer;
/* The buffer used to store the branch trace and its size. */
void *buffer;
unsigned int size;
/* The mm that paid for the above buffer. */
struct mm_struct *mm;
/* The task this context belongs to. */
struct task_struct *task;
/* The signal to send on a bts buffer overflow. */
unsigned int bts_ovfl_signal;
/* The work struct to destroy a context. */
struct work_struct work;
};
static int alloc_bts_buffer(struct bts_context *context, unsigned int size)
{
void *buffer = NULL;
int err = -ENOMEM;
err = account_locked_memory(current->mm, current->signal->rlim, size);
if (err < 0)
return err;
buffer = kzalloc(size, GFP_KERNEL);
if (!buffer)
goto out_refund;
context->buffer = buffer;
context->size = size;
context->mm = get_task_mm(current);
return 0;
out_refund:
refund_locked_memory(current->mm, size);
return err;
}
static inline void free_bts_buffer(struct bts_context *context)
{
if (!context->buffer)
return;
kfree(context->buffer);
context->buffer = NULL;
refund_locked_memory(context->mm, context->size);
context->size = 0;
mmput(context->mm);
context->mm = NULL;
}
static void free_bts_context_work(struct work_struct *w)
{
struct bts_context *context;
context = container_of(w, struct bts_context, work);
ds_release_bts(context->tracer);
put_task_struct(context->task);
free_bts_buffer(context);
kfree(context);
}
static inline void free_bts_context(struct bts_context *context)
{
INIT_WORK(&context->work, free_bts_context_work);
schedule_work(&context->work);
}
static inline struct bts_context *alloc_bts_context(struct task_struct *task)
{
struct bts_context *context = kzalloc(sizeof(*context), GFP_KERNEL);
if (context) {
context->task = task;
task->bts = context;
get_task_struct(task);
}
return context;
}
static int ptrace_bts_read_record(struct task_struct *child, size_t index,
struct bts_struct __user *out)
{
struct bts_context *context;
const struct bts_trace *trace;
struct bts_struct bts;
const unsigned char *at;
int error;
context = child->bts;
if (!context)
return -ESRCH;
trace = ds_read_bts(context->tracer);
if (!trace)
return -ESRCH;
at = trace->ds.top - ((index + 1) * trace->ds.size);
if ((void *)at < trace->ds.begin)
at += (trace->ds.n * trace->ds.size);
if (!trace->read)
return -EOPNOTSUPP;
error = trace->read(context->tracer, at, &bts);
if (error < 0)
return error;
if (copy_to_user(out, &bts, sizeof(bts)))
return -EFAULT;
return sizeof(bts);
}
static int ptrace_bts_drain(struct task_struct *child,
long size,
struct bts_struct __user *out)
{
struct bts_context *context;
const struct bts_trace *trace;
const unsigned char *at;
int error, drained = 0;
context = child->bts;
if (!context)
return -ESRCH;
trace = ds_read_bts(context->tracer);
if (!trace)
return -ESRCH;
if (!trace->read)
return -EOPNOTSUPP;
if (size < (trace->ds.top - trace->ds.begin))
return -EIO;
for (at = trace->ds.begin; (void *)at < trace->ds.top;
out++, drained++, at += trace->ds.size) {
struct bts_struct bts;
error = trace->read(context->tracer, at, &bts);
if (error < 0)
return error;
if (copy_to_user(out, &bts, sizeof(bts)))
return -EFAULT;
}
memset(trace->ds.begin, 0, trace->ds.n * trace->ds.size);
error = ds_reset_bts(context->tracer);
if (error < 0)
return error;
return drained;
}
static int ptrace_bts_config(struct task_struct *child,
long cfg_size,
const struct ptrace_bts_config __user *ucfg)
{
struct bts_context *context;
struct ptrace_bts_config cfg;
unsigned int flags = 0;
if (cfg_size < sizeof(cfg))
return -EIO;
if (copy_from_user(&cfg, ucfg, sizeof(cfg)))
return -EFAULT;
context = child->bts;
if (!context)
context = alloc_bts_context(child);
if (!context)
return -ENOMEM;
if (cfg.flags & PTRACE_BTS_O_SIGNAL) {
if (!cfg.signal)
return -EINVAL;
return -EOPNOTSUPP;
context->bts_ovfl_signal = cfg.signal;
}
ds_release_bts(context->tracer);
context->tracer = NULL;
if ((cfg.flags & PTRACE_BTS_O_ALLOC) && (cfg.size != context->size)) {
int err;
free_bts_buffer(context);
if (!cfg.size)
return 0;
err = alloc_bts_buffer(context, cfg.size);
if (err < 0)
return err;
}
if (cfg.flags & PTRACE_BTS_O_TRACE)
flags |= BTS_USER;
if (cfg.flags & PTRACE_BTS_O_SCHED)
flags |= BTS_TIMESTAMPS;
context->tracer =
ds_request_bts_task(child, context->buffer, context->size,
NULL, (size_t)-1, flags);
if (unlikely(IS_ERR(context->tracer))) {
int error = PTR_ERR(context->tracer);
free_bts_buffer(context);
context->tracer = NULL;
return error;
}
return sizeof(cfg);
}
static int ptrace_bts_status(struct task_struct *child,
long cfg_size,
struct ptrace_bts_config __user *ucfg)
{
struct bts_context *context;
const struct bts_trace *trace;
struct ptrace_bts_config cfg;
context = child->bts;
if (!context)
return -ESRCH;
if (cfg_size < sizeof(cfg))
return -EIO;
trace = ds_read_bts(context->tracer);
if (!trace)
return -ESRCH;
memset(&cfg, 0, sizeof(cfg));
cfg.size = trace->ds.end - trace->ds.begin;
cfg.signal = context->bts_ovfl_signal;
cfg.bts_size = sizeof(struct bts_struct);
if (cfg.signal)
cfg.flags |= PTRACE_BTS_O_SIGNAL;
if (trace->ds.flags & BTS_USER)
cfg.flags |= PTRACE_BTS_O_TRACE;
if (trace->ds.flags & BTS_TIMESTAMPS)
cfg.flags |= PTRACE_BTS_O_SCHED;
if (copy_to_user(ucfg, &cfg, sizeof(cfg)))
return -EFAULT;
return sizeof(cfg);
}
static int ptrace_bts_clear(struct task_struct *child)
{
struct bts_context *context;
const struct bts_trace *trace;
context = child->bts;
if (!context)
return -ESRCH;
trace = ds_read_bts(context->tracer);
if (!trace)
return -ESRCH;
memset(trace->ds.begin, 0, trace->ds.n * trace->ds.size);
return ds_reset_bts(context->tracer);
}
static int ptrace_bts_size(struct task_struct *child)
{
struct bts_context *context;
const struct bts_trace *trace;
context = child->bts;
if (!context)
return -ESRCH;
trace = ds_read_bts(context->tracer);
if (!trace)
return -ESRCH;
return (trace->ds.top - trace->ds.begin) / trace->ds.size;
}
/*
* Called from __ptrace_unlink() after the child has been moved back
* to its original parent.
*/
void ptrace_bts_untrace(struct task_struct *child)
{
if (unlikely(child->bts)) {
free_bts_context(child->bts);
child->bts = NULL;
}
}
#endif /* CONFIG_X86_PTRACE_BTS */
/* /*
* Called by kernel/ptrace.c when detaching.. * Called by kernel/ptrace.c when detaching..
* *
@ -1252,39 +911,6 @@ long arch_ptrace(struct task_struct *child, long request, long addr, long data)
break; break;
#endif #endif
/*
* These bits need more cooking - not enabled yet:
*/
#ifdef CONFIG_X86_PTRACE_BTS
case PTRACE_BTS_CONFIG:
ret = ptrace_bts_config
(child, data, (struct ptrace_bts_config __user *)addr);
break;
case PTRACE_BTS_STATUS:
ret = ptrace_bts_status
(child, data, (struct ptrace_bts_config __user *)addr);
break;
case PTRACE_BTS_SIZE:
ret = ptrace_bts_size(child);
break;
case PTRACE_BTS_GET:
ret = ptrace_bts_read_record
(child, data, (struct bts_struct __user *) addr);
break;
case PTRACE_BTS_CLEAR:
ret = ptrace_bts_clear(child);
break;
case PTRACE_BTS_DRAIN:
ret = ptrace_bts_drain
(child, data, (struct bts_struct __user *) addr);
break;
#endif /* CONFIG_X86_PTRACE_BTS */
default: default:
ret = ptrace_request(child, request, addr, data); ret = ptrace_request(child, request, addr, data);
break; break;
@ -1544,14 +1170,6 @@ long compat_arch_ptrace(struct task_struct *child, compat_long_t request,
case PTRACE_GET_THREAD_AREA: case PTRACE_GET_THREAD_AREA:
case PTRACE_SET_THREAD_AREA: case PTRACE_SET_THREAD_AREA:
#ifdef CONFIG_X86_PTRACE_BTS
case PTRACE_BTS_CONFIG:
case PTRACE_BTS_STATUS:
case PTRACE_BTS_SIZE:
case PTRACE_BTS_GET:
case PTRACE_BTS_CLEAR:
case PTRACE_BTS_DRAIN:
#endif /* CONFIG_X86_PTRACE_BTS */
return arch_ptrace(child, request, addr, data); return arch_ptrace(child, request, addr, data);
default: default:

44
arch/x86/kernel/step.c
View file

@ -157,22 +157,6 @@ static int enable_single_step(struct task_struct *child)
return 1; return 1;
} }
/*
* Install this value in MSR_IA32_DEBUGCTLMSR whenever child is running.
*/
static void write_debugctlmsr(struct task_struct *child, unsigned long val)
{
if (child->thread.debugctlmsr == val)
return;
child->thread.debugctlmsr = val;
if (child != current)
return;
update_debugctlmsr(val);
}
/* /*
* Enable single or block step. * Enable single or block step.
*/ */
@ -186,15 +170,17 @@ static void enable_step(struct task_struct *child, bool block)
* that uses user-mode single stepping itself. * that uses user-mode single stepping itself.
*/ */
if (enable_single_step(child) && block) { if (enable_single_step(child) && block) {
set_tsk_thread_flag(child, TIF_DEBUGCTLMSR); unsigned long debugctl = get_debugctlmsr();
write_debugctlmsr(child,
child->thread.debugctlmsr | DEBUGCTLMSR_BTF);
} else {
write_debugctlmsr(child,
child->thread.debugctlmsr & ~DEBUGCTLMSR_BTF);
if (!child->thread.debugctlmsr) debugctl |= DEBUGCTLMSR_BTF;
clear_tsk_thread_flag(child, TIF_DEBUGCTLMSR); update_debugctlmsr(debugctl);
set_tsk_thread_flag(child, TIF_BLOCKSTEP);
} else if (test_tsk_thread_flag(child, TIF_BLOCKSTEP)) {
unsigned long debugctl = get_debugctlmsr();
debugctl &= ~DEBUGCTLMSR_BTF;
update_debugctlmsr(debugctl);
clear_tsk_thread_flag(child, TIF_BLOCKSTEP);
} }
} }
@ -213,11 +199,13 @@ void user_disable_single_step(struct task_struct *child)
/* /*
* Make sure block stepping (BTF) is disabled. * Make sure block stepping (BTF) is disabled.
*/ */
write_debugctlmsr(child, if (test_tsk_thread_flag(child, TIF_BLOCKSTEP)) {
child->thread.debugctlmsr & ~DEBUGCTLMSR_BTF); unsigned long debugctl = get_debugctlmsr();
if (!child->thread.debugctlmsr) debugctl &= ~DEBUGCTLMSR_BTF;
clear_tsk_thread_flag(child, TIF_DEBUGCTLMSR); update_debugctlmsr(debugctl);
clear_tsk_thread_flag(child, TIF_BLOCKSTEP);
}
/* Always clear TIF_SINGLESTEP... */ /* Always clear TIF_SINGLESTEP... */
clear_tsk_thread_flag(child, TIF_SINGLESTEP); clear_tsk_thread_flag(child, TIF_SINGLESTEP);

4
arch/x86/kernel/traps.c
View file

@ -543,11 +543,11 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)
/* DR6 may or may not be cleared by the CPU */ /* DR6 may or may not be cleared by the CPU */
set_debugreg(0, 6); set_debugreg(0, 6);
/* /*
* The processor cleared BTF, so don't mark that we need it set. * The processor cleared BTF, so don't mark that we need it set.
*/ */
clear_tsk_thread_flag(tsk, TIF_DEBUGCTLMSR); clear_tsk_thread_flag(tsk, TIF_BLOCKSTEP);
tsk->thread.debugctlmsr = 0;
/* Store the virtualized DR6 value */ /* Store the virtualized DR6 value */
tsk->thread.debugreg6 = dr6; tsk->thread.debugreg6 = dr6;

5
arch/x86/kvm/vmx.c
View file

@ -3659,8 +3659,11 @@ static void vmx_complete_interrupts(struct vcpu_vmx *vmx)
/* We need to handle NMIs before interrupts are enabled */ /* We need to handle NMIs before interrupts are enabled */
if ((exit_intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR && if ((exit_intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR &&
(exit_intr_info & INTR_INFO_VALID_MASK)) (exit_intr_info & INTR_INFO_VALID_MASK)) {
kvm_before_handle_nmi(&vmx->vcpu);
asm("int $2"); asm("int $2");
kvm_after_handle_nmi(&vmx->vcpu);
}
idtv_info_valid = idt_vectoring_info & VECTORING_INFO_VALID_MASK; idtv_info_valid = idt_vectoring_info & VECTORING_INFO_VALID_MASK;

50
arch/x86/kvm/x86.c
View file

@ -40,6 +40,7 @@
#include <linux/user-return-notifier.h> #include <linux/user-return-notifier.h>
#include <linux/srcu.h> #include <linux/srcu.h>
#include <linux/slab.h> #include <linux/slab.h>
#include <linux/perf_event.h>
#include <trace/events/kvm.h> #include <trace/events/kvm.h>
#undef TRACE_INCLUDE_FILE #undef TRACE_INCLUDE_FILE
#define CREATE_TRACE_POINTS #define CREATE_TRACE_POINTS
@ -3747,6 +3748,51 @@ static void kvm_timer_init(void)
} }
} }
static DEFINE_PER_CPU(struct kvm_vcpu *, current_vcpu);
static int kvm_is_in_guest(void)
{
return percpu_read(current_vcpu) != NULL;
}
static int kvm_is_user_mode(void)
{
int user_mode = 3;
if (percpu_read(current_vcpu))
user_mode = kvm_x86_ops->get_cpl(percpu_read(current_vcpu));
return user_mode != 0;
}
static unsigned long kvm_get_guest_ip(void)
{
unsigned long ip = 0;
if (percpu_read(current_vcpu))
ip = kvm_rip_read(percpu_read(current_vcpu));
return ip;
}
static struct perf_guest_info_callbacks kvm_guest_cbs = {
.is_in_guest = kvm_is_in_guest,
.is_user_mode = kvm_is_user_mode,
.get_guest_ip = kvm_get_guest_ip,
};
void kvm_before_handle_nmi(struct kvm_vcpu *vcpu)
{
percpu_write(current_vcpu, vcpu);
}
EXPORT_SYMBOL_GPL(kvm_before_handle_nmi);
void kvm_after_handle_nmi(struct kvm_vcpu *vcpu)
{
percpu_write(current_vcpu, NULL);
}
EXPORT_SYMBOL_GPL(kvm_after_handle_nmi);
int kvm_arch_init(void *opaque) int kvm_arch_init(void *opaque)
{ {
int r; int r;
@ -3783,6 +3829,8 @@ int kvm_arch_init(void *opaque)
kvm_timer_init(); kvm_timer_init();
perf_register_guest_info_callbacks(&kvm_guest_cbs);
return 0; return 0;
out: out:
@ -3791,6 +3839,8 @@ out:
void kvm_arch_exit(void) void kvm_arch_exit(void)
{ {
perf_unregister_guest_info_callbacks(&kvm_guest_cbs);
if (!boot_cpu_has(X86_FEATURE_CONSTANT_TSC)) if (!boot_cpu_has(X86_FEATURE_CONSTANT_TSC))
cpufreq_unregister_notifier(&kvmclock_cpufreq_notifier_block, cpufreq_unregister_notifier(&kvmclock_cpufreq_notifier_block,
CPUFREQ_TRANSITION_NOTIFIER); CPUFREQ_TRANSITION_NOTIFIER);

3
arch/x86/kvm/x86.h
View file

@ -65,4 +65,7 @@ static inline int is_paging(struct kvm_vcpu *vcpu)
return kvm_read_cr0_bits(vcpu, X86_CR0_PG); return kvm_read_cr0_bits(vcpu, X86_CR0_PG);
} }
void kvm_before_handle_nmi(struct kvm_vcpu *vcpu);
void kvm_after_handle_nmi(struct kvm_vcpu *vcpu);
#endif #endif

2
arch/x86/lib/Makefile
View file

@ -20,7 +20,7 @@ lib-y := delay.o
lib-y += thunk_$(BITS).o lib-y += thunk_$(BITS).o
lib-y += usercopy_$(BITS).o getuser.o putuser.o lib-y += usercopy_$(BITS).o getuser.o putuser.o
lib-y += memcpy_$(BITS).o lib-y += memcpy_$(BITS).o
lib-$(CONFIG_KPROBES) += insn.o inat.o lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o
obj-y += msr.o msr-reg.o msr-reg-export.o obj-y += msr.o msr-reg.o msr-reg-export.o

4
arch/x86/oprofile/op_model_ppro.c
View file

@ -238,11 +238,11 @@ static void arch_perfmon_setup_counters(void)
if (eax.split.version_id == 0 && current_cpu_data.x86 == 6 && if (eax.split.version_id == 0 && current_cpu_data.x86 == 6 &&
current_cpu_data.x86_model == 15) { current_cpu_data.x86_model == 15) {
eax.split.version_id = 2; eax.split.version_id = 2;
eax.split.num_events = 2; eax.split.num_counters = 2;
eax.split.bit_width = 40; eax.split.bit_width = 40;
} }
num_counters = eax.split.num_events; num_counters = eax.split.num_counters;
op_arch_perfmon_spec.num_counters = num_counters; op_arch_perfmon_spec.num_counters = num_counters;
op_arch_perfmon_spec.num_controls = num_counters; op_arch_perfmon_spec.num_controls = num_counters;

12
include/linux/ftrace.h
View file

@ -504,18 +504,6 @@ extern int ftrace_dump_on_oops;
#define INIT_TRACE_RECURSION #define INIT_TRACE_RECURSION
#endif #endif
#ifdef CONFIG_HW_BRANCH_TRACER
void trace_hw_branch(u64 from, u64 to);
void trace_hw_branch_oops(void);
#else /* CONFIG_HW_BRANCH_TRACER */
static inline void trace_hw_branch(u64 from, u64 to) {}
static inline void trace_hw_branch_oops(void) {}
#endif /* CONFIG_HW_BRANCH_TRACER */
#ifdef CONFIG_FTRACE_SYSCALLS #ifdef CONFIG_FTRACE_SYSCALLS
unsigned long arch_syscall_addr(int nr); unsigned long arch_syscall_addr(int nr);

19
include/linux/hw_breakpoint.h
View file

@ -9,9 +9,22 @@ enum {
}; };
enum { enum {
HW_BREAKPOINT_EMPTY = 0,
HW_BREAKPOINT_R = 1, HW_BREAKPOINT_R = 1,
HW_BREAKPOINT_W = 2, HW_BREAKPOINT_W = 2,
HW_BREAKPOINT_RW = HW_BREAKPOINT_R | HW_BREAKPOINT_W,
HW_BREAKPOINT_X = 4, HW_BREAKPOINT_X = 4,
HW_BREAKPOINT_INVALID = HW_BREAKPOINT_RW | HW_BREAKPOINT_X,
};
enum bp_type_idx {
TYPE_INST = 0,
#ifdef CONFIG_HAVE_MIXED_BREAKPOINTS_REGS
TYPE_DATA = 0,
#else
TYPE_DATA = 1,
#endif
TYPE_MAX
}; };
#ifdef __KERNEL__ #ifdef __KERNEL__
@ -34,6 +47,12 @@ static inline void hw_breakpoint_init(struct perf_event_attr *attr)
attr->sample_period = 1; attr->sample_period = 1;
} }
static inline void ptrace_breakpoint_init(struct perf_event_attr *attr)
{
hw_breakpoint_init(attr);
attr->exclude_kernel = 1;
}
static inline unsigned long hw_breakpoint_addr(struct perf_event *bp) static inline unsigned long hw_breakpoint_addr(struct perf_event *bp)
{ {
return bp->attr.bp_addr; return bp->attr.bp_addr;

4
include/linux/mm.h
View file

@ -19,7 +19,6 @@ struct anon_vma;
struct file_ra_state; struct file_ra_state;
struct user_struct; struct user_struct;
struct writeback_control; struct writeback_control;
struct rlimit;
#ifndef CONFIG_DISCONTIGMEM /* Don't use mapnrs, do it properly */ #ifndef CONFIG_DISCONTIGMEM /* Don't use mapnrs, do it properly */
extern unsigned long max_mapnr; extern unsigned long max_mapnr;
@ -1449,9 +1448,6 @@ int vmemmap_populate_basepages(struct page *start_page,
int vmemmap_populate(struct page *start_page, unsigned long pages, int node); int vmemmap_populate(struct page *start_page, unsigned long pages, int node);
void vmemmap_populate_print_last(void); void vmemmap_populate_print_last(void);
extern int account_locked_memory(struct mm_struct *mm, struct rlimit *rlim,
size_t size);
extern void refund_locked_memory(struct mm_struct *mm, size_t size);
enum mf_flags { enum mf_flags {
MF_COUNT_INCREASED = 1 << 0, MF_COUNT_INCREASED = 1 << 0,

81
include/linux/perf_event.h
View file

@ -203,8 +203,19 @@ struct perf_event_attr {
enable_on_exec : 1, /* next exec enables */ enable_on_exec : 1, /* next exec enables */
task : 1, /* trace fork/exit */ task : 1, /* trace fork/exit */
watermark : 1, /* wakeup_watermark */ watermark : 1, /* wakeup_watermark */
/*
* precise_ip:
*
* 0 - SAMPLE_IP can have arbitrary skid
* 1 - SAMPLE_IP must have constant skid
* 2 - SAMPLE_IP requested to have 0 skid
* 3 - SAMPLE_IP must have 0 skid
*
* See also PERF_RECORD_MISC_EXACT_IP
*/
precise_ip : 2, /* skid constraint */
__reserved_1 : 49; __reserved_1 : 47;
union { union {
__u32 wakeup_events; /* wakeup every n events */ __u32 wakeup_events; /* wakeup every n events */
@ -287,11 +298,24 @@ struct perf_event_mmap_page {
__u64 data_tail; /* user-space written tail */ __u64 data_tail; /* user-space written tail */
}; };
#define PERF_RECORD_MISC_CPUMODE_MASK (3 << 0) #define PERF_RECORD_MISC_CPUMODE_MASK (7 << 0)
#define PERF_RECORD_MISC_CPUMODE_UNKNOWN (0 << 0) #define PERF_RECORD_MISC_CPUMODE_UNKNOWN (0 << 0)
#define PERF_RECORD_MISC_KERNEL (1 << 0) #define PERF_RECORD_MISC_KERNEL (1 << 0)
#define PERF_RECORD_MISC_USER (2 << 0) #define PERF_RECORD_MISC_USER (2 << 0)
#define PERF_RECORD_MISC_HYPERVISOR (3 << 0) #define PERF_RECORD_MISC_HYPERVISOR (3 << 0)
#define PERF_RECORD_MISC_GUEST_KERNEL (4 << 0)
#define PERF_RECORD_MISC_GUEST_USER (5 << 0)
/*
* Indicates that the content of PERF_SAMPLE_IP points to
* the actual instruction that triggered the event. See also
* perf_event_attr::precise_ip.
*/
#define PERF_RECORD_MISC_EXACT_IP (1 << 14)
/*
* Reserve the last bit to indicate some extended misc field
*/
#define PERF_RECORD_MISC_EXT_RESERVED (1 << 15)
struct perf_event_header { struct perf_event_header {
__u32 type; __u32 type;
@ -439,6 +463,12 @@ enum perf_callchain_context {
# include <asm/perf_event.h> # include <asm/perf_event.h>
#endif #endif
struct perf_guest_info_callbacks {
int (*is_in_guest) (void);
int (*is_user_mode) (void);
unsigned long (*get_guest_ip) (void);
};
#ifdef CONFIG_HAVE_HW_BREAKPOINT #ifdef CONFIG_HAVE_HW_BREAKPOINT
#include <asm/hw_breakpoint.h> #include <asm/hw_breakpoint.h>
#endif #endif
@ -468,6 +498,17 @@ struct perf_raw_record {
void *data; void *data;
}; };
struct perf_branch_entry {
__u64 from;
__u64 to;
__u64 flags;
};
struct perf_branch_stack {
__u64 nr;
struct perf_branch_entry entries[0];
};
struct task_struct; struct task_struct;
/** /**
@ -506,6 +547,8 @@ struct hw_perf_event {
struct perf_event; struct perf_event;
#define PERF_EVENT_TXN_STARTED 1
/** /**
* struct pmu - generic performance monitoring unit * struct pmu - generic performance monitoring unit
*/ */
@ -516,6 +559,16 @@ struct pmu {
void (*stop) (struct perf_event *event); void (*stop) (struct perf_event *event);
void (*read) (struct perf_event *event); void (*read) (struct perf_event *event);
void (*unthrottle) (struct perf_event *event); void (*unthrottle) (struct perf_event *event);
/*
* group events scheduling is treated as a transaction,
* add group events as a whole and perform one schedulability test.
* If test fails, roll back the whole group
*/
void (*start_txn) (const struct pmu *pmu);
void (*cancel_txn) (const struct pmu *pmu);
int (*commit_txn) (const struct pmu *pmu);
}; };
/** /**
@ -571,6 +624,14 @@ enum perf_group_flag {
PERF_GROUP_SOFTWARE = 0x1, PERF_GROUP_SOFTWARE = 0x1,
}; };
#define SWEVENT_HLIST_BITS 8
#define SWEVENT_HLIST_SIZE (1 << SWEVENT_HLIST_BITS)
struct swevent_hlist {
struct hlist_head heads[SWEVENT_HLIST_SIZE];
struct rcu_head rcu_head;
};
/** /**
* struct perf_event - performance event kernel representation: * struct perf_event - performance event kernel representation:
*/ */
@ -579,6 +640,7 @@ struct perf_event {
struct list_head group_entry; struct list_head group_entry;
struct list_head event_entry; struct list_head event_entry;
struct list_head sibling_list; struct list_head sibling_list;
struct hlist_node hlist_entry;
int nr_siblings; int nr_siblings;
int group_flags; int group_flags;
struct perf_event *group_leader; struct perf_event *group_leader;
@ -726,6 +788,9 @@ struct perf_cpu_context {
int active_oncpu; int active_oncpu;
int max_pertask; int max_pertask;
int exclusive; int exclusive;
struct swevent_hlist *swevent_hlist;
struct mutex hlist_mutex;
int hlist_refcount;
/* /*
* Recursion avoidance: * Recursion avoidance:
@ -769,9 +834,6 @@ extern void perf_disable(void);
extern void perf_enable(void); extern void perf_enable(void);
extern int perf_event_task_disable(void); extern int perf_event_task_disable(void);
extern int perf_event_task_enable(void); extern int perf_event_task_enable(void);
extern int hw_perf_group_sched_in(struct perf_event *group_leader,
struct perf_cpu_context *cpuctx,
struct perf_event_context *ctx);
extern void perf_event_update_userpage(struct perf_event *event); extern void perf_event_update_userpage(struct perf_event *event);
extern int perf_event_release_kernel(struct perf_event *event); extern int perf_event_release_kernel(struct perf_event *event);
extern struct perf_event * extern struct perf_event *
@ -902,6 +964,10 @@ static inline void perf_event_mmap(struct vm_area_struct *vma)
__perf_event_mmap(vma); __perf_event_mmap(vma);
} }
extern struct perf_guest_info_callbacks *perf_guest_cbs;
extern int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks);
extern int perf_unregister_guest_info_callbacks(struct perf_guest_info_callbacks *callbacks);
extern void perf_event_comm(struct task_struct *tsk); extern void perf_event_comm(struct task_struct *tsk);
extern void perf_event_fork(struct task_struct *tsk); extern void perf_event_fork(struct task_struct *tsk);
@ -971,6 +1037,11 @@ perf_sw_event(u32 event_id, u64 nr, int nmi,
static inline void static inline void
perf_bp_event(struct perf_event *event, void *data) { } perf_bp_event(struct perf_event *event, void *data) { }
static inline int perf_register_guest_info_callbacks
(struct perf_guest_info_callbacks *callbacks) { return 0; }
static inline int perf_unregister_guest_info_callbacks
(struct perf_guest_info_callbacks *callbacks) { return 0; }
static inline void perf_event_mmap(struct vm_area_struct *vma) { } static inline void perf_event_mmap(struct vm_area_struct *vma) { }
static inline void perf_event_comm(struct task_struct *tsk) { } static inline void perf_event_comm(struct task_struct *tsk) { }
static inline void perf_event_fork(struct task_struct *tsk) { } static inline void perf_event_fork(struct task_struct *tsk) { }

12
include/linux/ptrace.h
View file

@ -345,18 +345,6 @@ static inline void user_single_step_siginfo(struct task_struct *tsk,
#define arch_ptrace_stop(code, info) do { } while (0) #define arch_ptrace_stop(code, info) do { } while (0)
#endif #endif
#ifndef arch_ptrace_untrace
/*
* Do machine-specific work before untracing child.
*
* This is called for a normal detach as well as from ptrace_exit()
* when the tracing task dies.
*
* Called with write_lock(&tasklist_lock) held.
*/
#define arch_ptrace_untrace(task) do { } while (0)
#endif
extern int task_current_syscall(struct task_struct *target, long *callno, extern int task_current_syscall(struct task_struct *target, long *callno,
unsigned long args[6], unsigned int maxargs, unsigned long args[6], unsigned int maxargs,
unsigned long *sp, unsigned long *pc); unsigned long *sp, unsigned long *pc);

9
include/linux/sched.h
View file

@ -99,7 +99,6 @@ struct futex_pi_state;
struct robust_list_head; struct robust_list_head;
struct bio_list; struct bio_list;
struct fs_struct; struct fs_struct;
struct bts_context;
struct perf_event_context; struct perf_event_context;
/* /*
@ -1272,12 +1271,6 @@ struct task_struct {
struct list_head ptraced; struct list_head ptraced;
struct list_head ptrace_entry; struct list_head ptrace_entry;
/*
* This is the tracer handle for the ptrace BTS extension.
* This field actually belongs to the ptracer task.
*/
struct bts_context *bts;
/* PID/PID hash table linkage. */ /* PID/PID hash table linkage. */
struct pid_link pids[PIDTYPE_MAX]; struct pid_link pids[PIDTYPE_MAX];
struct list_head thread_group; struct list_head thread_group;
@ -2122,10 +2115,8 @@ extern void set_task_comm(struct task_struct *tsk, char *from);
extern char *get_task_comm(char *to, struct task_struct *tsk); extern char *get_task_comm(char *to, struct task_struct *tsk);
#ifdef CONFIG_SMP #ifdef CONFIG_SMP
extern void wait_task_context_switch(struct task_struct *p);
extern unsigned long wait_task_inactive(struct task_struct *, long match_state); extern unsigned long wait_task_inactive(struct task_struct *, long match_state);
#else #else
static inline void wait_task_context_switch(struct task_struct *p) {}
static inline unsigned long wait_task_inactive(struct task_struct *p, static inline unsigned long wait_task_inactive(struct task_struct *p,
long match_state) long match_state)
{ {

61
include/trace/events/lock.h
View file

@ -35,29 +35,7 @@ TRACE_EVENT(lock_acquire,
__get_str(name)) __get_str(name))
); );
TRACE_EVENT(lock_release, DECLARE_EVENT_CLASS(lock,
TP_PROTO(struct lockdep_map *lock, int nested, unsigned long ip),
TP_ARGS(lock, nested, ip),
TP_STRUCT__entry(
__string(name, lock->name)
__field(void *, lockdep_addr)
),
TP_fast_assign(
__assign_str(name, lock->name);
__entry->lockdep_addr = lock;
),
TP_printk("%p %s",
__entry->lockdep_addr, __get_str(name))
);
#ifdef CONFIG_LOCK_STAT
TRACE_EVENT(lock_contended,
TP_PROTO(struct lockdep_map *lock, unsigned long ip), TP_PROTO(struct lockdep_map *lock, unsigned long ip),
@ -73,29 +51,30 @@ TRACE_EVENT(lock_contended,
__entry->lockdep_addr = lock; __entry->lockdep_addr = lock;
), ),
TP_printk("%p %s", TP_printk("%p %s", __entry->lockdep_addr, __get_str(name))
__entry->lockdep_addr, __get_str(name))
); );
TRACE_EVENT(lock_acquired, DEFINE_EVENT(lock, lock_release,
TP_PROTO(struct lockdep_map *lock, unsigned long ip, s64 waittime),
TP_ARGS(lock, ip, waittime), TP_PROTO(struct lockdep_map *lock, unsigned long ip),
TP_STRUCT__entry( TP_ARGS(lock, ip)
__string(name, lock->name) );
__field(s64, wait_nsec)
__field(void *, lockdep_addr)
),
TP_fast_assign( #ifdef CONFIG_LOCK_STAT
__assign_str(name, lock->name);
__entry->wait_nsec = waittime; DEFINE_EVENT(lock, lock_contended,
__entry->lockdep_addr = lock;
), TP_PROTO(struct lockdep_map *lock, unsigned long ip),
TP_printk("%p %s (%llu ns)", __entry->lockdep_addr,
__get_str(name), TP_ARGS(lock, ip)
__entry->wait_nsec) );
DEFINE_EVENT(lock, lock_acquired,
TP_PROTO(struct lockdep_map *lock, unsigned long ip),
TP_ARGS(lock, ip)
); );
#endif #endif

13
include/trace/ftrace.h
View file

@ -763,13 +763,12 @@ __attribute__((section("_ftrace_events"))) event_##call = { \
#define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ #define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \
static notrace void \ static notrace void \
perf_trace_templ_##call(struct ftrace_event_call *event_call, \ perf_trace_templ_##call(struct ftrace_event_call *event_call, \
proto) \ struct pt_regs *__regs, proto) \
{ \ { \
struct ftrace_data_offsets_##call __maybe_unused __data_offsets;\ struct ftrace_data_offsets_##call __maybe_unused __data_offsets;\
struct ftrace_raw_##call *entry; \ struct ftrace_raw_##call *entry; \
u64 __addr = 0, __count = 1; \ u64 __addr = 0, __count = 1; \
unsigned long irq_flags; \ unsigned long irq_flags; \
struct pt_regs *__regs; \
int __entry_size; \ int __entry_size; \
int __data_size; \ int __data_size; \
int rctx; \ int rctx; \
@ -790,9 +789,6 @@ perf_trace_templ_##call(struct ftrace_event_call *event_call, \
\ \
{ assign; } \ { assign; } \
\ \
__regs = &__get_cpu_var(perf_trace_regs); \
perf_fetch_caller_regs(__regs, 2); \
\
perf_trace_buf_submit(entry, __entry_size, rctx, __addr, \ perf_trace_buf_submit(entry, __entry_size, rctx, __addr, \
__count, irq_flags, __regs); \ __count, irq_flags, __regs); \
} }
@ -802,8 +798,13 @@ perf_trace_templ_##call(struct ftrace_event_call *event_call, \
static notrace void perf_trace_##call(proto) \ static notrace void perf_trace_##call(proto) \
{ \ { \
struct ftrace_event_call *event_call = &event_##call; \ struct ftrace_event_call *event_call = &event_##call; \
struct pt_regs *__regs = &get_cpu_var(perf_trace_regs); \
\ \
perf_trace_templ_##template(event_call, args); \ perf_fetch_caller_regs(__regs, 1); \
\
perf_trace_templ_##template(event_call, __regs, args); \
\
put_cpu_var(perf_trace_regs); \
} }
#undef DEFINE_EVENT_PRINT #undef DEFINE_EVENT_PRINT

2
kernel/fork.c
View file

@ -1112,8 +1112,6 @@ static struct task_struct *copy_process(unsigned long clone_flags,
p->memcg_batch.memcg = NULL; p->memcg_batch.memcg = NULL;
#endif #endif
p->bts = NULL;
/* Perform scheduler related setup. Assign this task to a CPU. */ /* Perform scheduler related setup. Assign this task to a CPU. */
sched_fork(p, clone_flags); sched_fork(p, clone_flags);

196
kernel/hw_breakpoint.c
View file

@ -40,23 +40,29 @@
#include <linux/percpu.h> #include <linux/percpu.h>
#include <linux/sched.h> #include <linux/sched.h>
#include <linux/init.h> #include <linux/init.h>
#include <linux/slab.h>
#include <linux/cpu.h> #include <linux/cpu.h>
#include <linux/smp.h> #include <linux/smp.h>
#include <linux/hw_breakpoint.h> #include <linux/hw_breakpoint.h>
/* /*
* Constraints data * Constraints data
*/ */
/* Number of pinned cpu breakpoints in a cpu */ /* Number of pinned cpu breakpoints in a cpu */
static DEFINE_PER_CPU(unsigned int, nr_cpu_bp_pinned); static DEFINE_PER_CPU(unsigned int, nr_cpu_bp_pinned[TYPE_MAX]);
/* Number of pinned task breakpoints in a cpu */ /* Number of pinned task breakpoints in a cpu */
static DEFINE_PER_CPU(unsigned int, nr_task_bp_pinned[HBP_NUM]); static DEFINE_PER_CPU(unsigned int *, nr_task_bp_pinned[TYPE_MAX]);
/* Number of non-pinned cpu/task breakpoints in a cpu */ /* Number of non-pinned cpu/task breakpoints in a cpu */
static DEFINE_PER_CPU(unsigned int, nr_bp_flexible); static DEFINE_PER_CPU(unsigned int, nr_bp_flexible[TYPE_MAX]);
static int nr_slots[TYPE_MAX];
static int constraints_initialized;
/* Gather the number of total pinned and un-pinned bp in a cpuset */ /* Gather the number of total pinned and un-pinned bp in a cpuset */
struct bp_busy_slots { struct bp_busy_slots {
@ -67,16 +73,29 @@ struct bp_busy_slots {
/* Serialize accesses to the above constraints */ /* Serialize accesses to the above constraints */
static DEFINE_MUTEX(nr_bp_mutex); static DEFINE_MUTEX(nr_bp_mutex);
__weak int hw_breakpoint_weight(struct perf_event *bp)
{
return 1;
}
static inline enum bp_type_idx find_slot_idx(struct perf_event *bp)
{
if (bp->attr.bp_type & HW_BREAKPOINT_RW)
return TYPE_DATA;
return TYPE_INST;
}
/* /*
* Report the maximum number of pinned breakpoints a task * Report the maximum number of pinned breakpoints a task
* have in this cpu * have in this cpu
*/ */
static unsigned int max_task_bp_pinned(int cpu) static unsigned int max_task_bp_pinned(int cpu, enum bp_type_idx type)
{ {
int i; int i;
unsigned int *tsk_pinned = per_cpu(nr_task_bp_pinned, cpu); unsigned int *tsk_pinned = per_cpu(nr_task_bp_pinned[type], cpu);
for (i = HBP_NUM -1; i >= 0; i--) { for (i = nr_slots[type] - 1; i >= 0; i--) {
if (tsk_pinned[i] > 0) if (tsk_pinned[i] > 0)
return i + 1; return i + 1;
} }
@ -84,7 +103,7 @@ static unsigned int max_task_bp_pinned(int cpu)
return 0; return 0;
} }
static int task_bp_pinned(struct task_struct *tsk) static int task_bp_pinned(struct task_struct *tsk, enum bp_type_idx type)
{ {
struct perf_event_context *ctx = tsk->perf_event_ctxp; struct perf_event_context *ctx = tsk->perf_event_ctxp;
struct list_head *list; struct list_head *list;
@ -105,7 +124,8 @@ static int task_bp_pinned(struct task_struct *tsk)
*/ */
list_for_each_entry(bp, list, event_entry) { list_for_each_entry(bp, list, event_entry) {
if (bp->attr.type == PERF_TYPE_BREAKPOINT) if (bp->attr.type == PERF_TYPE_BREAKPOINT)
count++; if (find_slot_idx(bp) == type)
count += hw_breakpoint_weight(bp);
} }
raw_spin_unlock_irqrestore(&ctx->lock, flags); raw_spin_unlock_irqrestore(&ctx->lock, flags);
@ -118,18 +138,19 @@ static int task_bp_pinned(struct task_struct *tsk)
* a given cpu (cpu > -1) or in all of them (cpu = -1). * a given cpu (cpu > -1) or in all of them (cpu = -1).
*/ */
static void static void
fetch_bp_busy_slots(struct bp_busy_slots *slots, struct perf_event *bp) fetch_bp_busy_slots(struct bp_busy_slots *slots, struct perf_event *bp,
enum bp_type_idx type)
{ {
int cpu = bp->cpu; int cpu = bp->cpu;
struct task_struct *tsk = bp->ctx->task; struct task_struct *tsk = bp->ctx->task;
if (cpu >= 0) { if (cpu >= 0) {
slots->pinned = per_cpu(nr_cpu_bp_pinned, cpu); slots->pinned = per_cpu(nr_cpu_bp_pinned[type], cpu);
if (!tsk) if (!tsk)
slots->pinned += max_task_bp_pinned(cpu); slots->pinned += max_task_bp_pinned(cpu, type);
else else
slots->pinned += task_bp_pinned(tsk); slots->pinned += task_bp_pinned(tsk, type);
slots->flexible = per_cpu(nr_bp_flexible, cpu); slots->flexible = per_cpu(nr_bp_flexible[type], cpu);
return; return;
} }
@ -137,48 +158,66 @@ fetch_bp_busy_slots(struct bp_busy_slots *slots, struct perf_event *bp)
for_each_online_cpu(cpu) { for_each_online_cpu(cpu) {
unsigned int nr; unsigned int nr;
nr = per_cpu(nr_cpu_bp_pinned, cpu); nr = per_cpu(nr_cpu_bp_pinned[type], cpu);
if (!tsk) if (!tsk)
nr += max_task_bp_pinned(cpu); nr += max_task_bp_pinned(cpu, type);
else else
nr += task_bp_pinned(tsk); nr += task_bp_pinned(tsk, type);
if (nr > slots->pinned) if (nr > slots->pinned)
slots->pinned = nr; slots->pinned = nr;
nr = per_cpu(nr_bp_flexible, cpu); nr = per_cpu(nr_bp_flexible[type], cpu);
if (nr > slots->flexible) if (nr > slots->flexible)
slots->flexible = nr; slots->flexible = nr;
} }
} }
/*
* For now, continue to consider flexible as pinned, until we can
* ensure no flexible event can ever be scheduled before a pinned event
* in a same cpu.
*/
static void
fetch_this_slot(struct bp_busy_slots *slots, int weight)
{
slots->pinned += weight;
}
/* /*
* Add a pinned breakpoint for the given task in our constraint table * Add a pinned breakpoint for the given task in our constraint table
*/ */
static void toggle_bp_task_slot(struct task_struct *tsk, int cpu, bool enable) static void toggle_bp_task_slot(struct task_struct *tsk, int cpu, bool enable,
enum bp_type_idx type, int weight)
{ {
unsigned int *tsk_pinned; unsigned int *tsk_pinned;
int count = 0; int old_count = 0;
int old_idx = 0;
int idx = 0;
count = task_bp_pinned(tsk); old_count = task_bp_pinned(tsk, type);
old_idx = old_count - 1;
idx = old_idx + weight;
tsk_pinned = per_cpu(nr_task_bp_pinned, cpu); tsk_pinned = per_cpu(nr_task_bp_pinned[type], cpu);
if (enable) { if (enable) {
tsk_pinned[count]++; tsk_pinned[idx]++;
if (count > 0) if (old_count > 0)
tsk_pinned[count-1]--; tsk_pinned[old_idx]--;
} else { } else {
tsk_pinned[count]--; tsk_pinned[idx]--;
if (count > 0) if (old_count > 0)
tsk_pinned[count-1]++; tsk_pinned[old_idx]++;
} }
} }
/* /*
* Add/remove the given breakpoint in our constraint table * Add/remove the given breakpoint in our constraint table
*/ */
static void toggle_bp_slot(struct perf_event *bp, bool enable) static void
toggle_bp_slot(struct perf_event *bp, bool enable, enum bp_type_idx type,
int weight)
{ {
int cpu = bp->cpu; int cpu = bp->cpu;
struct task_struct *tsk = bp->ctx->task; struct task_struct *tsk = bp->ctx->task;
@ -186,20 +225,20 @@ static void toggle_bp_slot(struct perf_event *bp, bool enable)
/* Pinned counter task profiling */ /* Pinned counter task profiling */
if (tsk) { if (tsk) {
if (cpu >= 0) { if (cpu >= 0) {
toggle_bp_task_slot(tsk, cpu, enable); toggle_bp_task_slot(tsk, cpu, enable, type, weight);
return; return;
} }
for_each_online_cpu(cpu) for_each_online_cpu(cpu)
toggle_bp_task_slot(tsk, cpu, enable); toggle_bp_task_slot(tsk, cpu, enable, type, weight);
return; return;
} }
/* Pinned counter cpu profiling */ /* Pinned counter cpu profiling */
if (enable) if (enable)
per_cpu(nr_cpu_bp_pinned, bp->cpu)++; per_cpu(nr_cpu_bp_pinned[type], bp->cpu) += weight;
else else
per_cpu(nr_cpu_bp_pinned, bp->cpu)--; per_cpu(nr_cpu_bp_pinned[type], bp->cpu) -= weight;
} }
/* /*
@ -246,14 +285,29 @@ static void toggle_bp_slot(struct perf_event *bp, bool enable)
static int __reserve_bp_slot(struct perf_event *bp) static int __reserve_bp_slot(struct perf_event *bp)
{ {
struct bp_busy_slots slots = {0}; struct bp_busy_slots slots = {0};
enum bp_type_idx type;
int weight;
fetch_bp_busy_slots(&slots, bp); /* We couldn't initialize breakpoint constraints on boot */
if (!constraints_initialized)
return -ENOMEM;
/* Basic checks */
if (bp->attr.bp_type == HW_BREAKPOINT_EMPTY ||
bp->attr.bp_type == HW_BREAKPOINT_INVALID)
return -EINVAL;
type = find_slot_idx(bp);
weight = hw_breakpoint_weight(bp);
fetch_bp_busy_slots(&slots, bp, type);
fetch_this_slot(&slots, weight);
/* Flexible counters need to keep at least one slot */ /* Flexible counters need to keep at least one slot */
if (slots.pinned + (!!slots.flexible) == HBP_NUM) if (slots.pinned + (!!slots.flexible) > nr_slots[type])
return -ENOSPC; return -ENOSPC;
toggle_bp_slot(bp, true); toggle_bp_slot(bp, true, type, weight);
return 0; return 0;
} }
@ -273,7 +327,12 @@ int reserve_bp_slot(struct perf_event *bp)
static void __release_bp_slot(struct perf_event *bp) static void __release_bp_slot(struct perf_event *bp)
{ {
toggle_bp_slot(bp, false); enum bp_type_idx type;
int weight;
type = find_slot_idx(bp);
weight = hw_breakpoint_weight(bp);
toggle_bp_slot(bp, false, type, weight);
} }
void release_bp_slot(struct perf_event *bp) void release_bp_slot(struct perf_event *bp)
@ -308,6 +367,28 @@ int dbg_release_bp_slot(struct perf_event *bp)
return 0; return 0;
} }
static int validate_hw_breakpoint(struct perf_event *bp)
{
int ret;
ret = arch_validate_hwbkpt_settings(bp);
if (ret)
return ret;
if (arch_check_bp_in_kernelspace(bp)) {
if (bp->attr.exclude_kernel)
return -EINVAL;
/*
* Don't let unprivileged users set a breakpoint in the trap
* path to avoid trap recursion attacks.
*/
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
}
return 0;
}
int register_perf_hw_breakpoint(struct perf_event *bp) int register_perf_hw_breakpoint(struct perf_event *bp)
{ {
int ret; int ret;
@ -316,17 +397,7 @@ int register_perf_hw_breakpoint(struct perf_event *bp)
if (ret) if (ret)
return ret; return ret;
/* ret = validate_hw_breakpoint(bp);
* Ptrace breakpoints can be temporary perf events only
* meant to reserve a slot. In this case, it is created disabled and
* we don't want to check the params right now (as we put a null addr)
* But perf tools create events as disabled and we want to check
* the params for them.
* This is a quick hack that will be removed soon, once we remove
* the tmp breakpoints from ptrace
*/
if (!bp->attr.disabled || !bp->overflow_handler)
ret = arch_validate_hwbkpt_settings(bp, bp->ctx->task);
/* if arch_validate_hwbkpt_settings() fails then release bp slot */ /* if arch_validate_hwbkpt_settings() fails then release bp slot */
if (ret) if (ret)
@ -373,7 +444,7 @@ int modify_user_hw_breakpoint(struct perf_event *bp, struct perf_event_attr *att
if (attr->disabled) if (attr->disabled)
goto end; goto end;
err = arch_validate_hwbkpt_settings(bp, bp->ctx->task); err = validate_hw_breakpoint(bp);
if (!err) if (!err)
perf_event_enable(bp); perf_event_enable(bp);
@ -480,7 +551,36 @@ static struct notifier_block hw_breakpoint_exceptions_nb = {
static int __init init_hw_breakpoint(void) static int __init init_hw_breakpoint(void)
{ {
unsigned int **task_bp_pinned;
int cpu, err_cpu;
int i;
for (i = 0; i < TYPE_MAX; i++)
nr_slots[i] = hw_breakpoint_slots(i);
for_each_possible_cpu(cpu) {
for (i = 0; i < TYPE_MAX; i++) {
task_bp_pinned = &per_cpu(nr_task_bp_pinned[i], cpu);
*task_bp_pinned = kzalloc(sizeof(int) * nr_slots[i],
GFP_KERNEL);
if (!*task_bp_pinned)
goto err_alloc;
}
}
constraints_initialized = 1;
return register_die_notifier(&hw_breakpoint_exceptions_nb); return register_die_notifier(&hw_breakpoint_exceptions_nb);
err_alloc:
for_each_possible_cpu(err_cpu) {
if (err_cpu == cpu)
break;
for (i = 0; i < TYPE_MAX; i++)
kfree(per_cpu(nr_task_bp_pinned[i], cpu));
}
return -ENOMEM;
} }
core_initcall(init_hw_breakpoint); core_initcall(init_hw_breakpoint);

132
kernel/kprobes.c
View file

@ -1588,6 +1588,72 @@ static void __kprobes kill_kprobe(struct kprobe *p)
arch_remove_kprobe(p); arch_remove_kprobe(p);
} }
/* Disable one kprobe */
int __kprobes disable_kprobe(struct kprobe *kp)
{
int ret = 0;
struct kprobe *p;
mutex_lock(&kprobe_mutex);
/* Check whether specified probe is valid. */
p = __get_valid_kprobe(kp);
if (unlikely(p == NULL)) {
ret = -EINVAL;
goto out;
}
/* If the probe is already disabled (or gone), just return */
if (kprobe_disabled(kp))
goto out;
kp->flags |= KPROBE_FLAG_DISABLED;
if (p != kp)
/* When kp != p, p is always enabled. */
try_to_disable_aggr_kprobe(p);
if (!kprobes_all_disarmed && kprobe_disabled(p))
disarm_kprobe(p);
out:
mutex_unlock(&kprobe_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(disable_kprobe);
/* Enable one kprobe */
int __kprobes enable_kprobe(struct kprobe *kp)
{
int ret = 0;
struct kprobe *p;
mutex_lock(&kprobe_mutex);
/* Check whether specified probe is valid. */
p = __get_valid_kprobe(kp);
if (unlikely(p == NULL)) {
ret = -EINVAL;
goto out;
}
if (kprobe_gone(kp)) {
/* This kprobe has gone, we couldn't enable it. */
ret = -EINVAL;
goto out;
}
if (p != kp)
kp->flags &= ~KPROBE_FLAG_DISABLED;
if (!kprobes_all_disarmed && kprobe_disabled(p)) {
p->flags &= ~KPROBE_FLAG_DISABLED;
arm_kprobe(p);
}
out:
mutex_unlock(&kprobe_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(enable_kprobe);
void __kprobes dump_kprobe(struct kprobe *kp) void __kprobes dump_kprobe(struct kprobe *kp)
{ {
printk(KERN_WARNING "Dumping kprobe:\n"); printk(KERN_WARNING "Dumping kprobe:\n");
@ -1805,72 +1871,6 @@ static const struct file_operations debugfs_kprobes_operations = {
.release = seq_release, .release = seq_release,
}; };
/* Disable one kprobe */
int __kprobes disable_kprobe(struct kprobe *kp)
{
int ret = 0;
struct kprobe *p;
mutex_lock(&kprobe_mutex);
/* Check whether specified probe is valid. */
p = __get_valid_kprobe(kp);
if (unlikely(p == NULL)) {
ret = -EINVAL;
goto out;
}
/* If the probe is already disabled (or gone), just return */
if (kprobe_disabled(kp))
goto out;
kp->flags |= KPROBE_FLAG_DISABLED;
if (p != kp)
/* When kp != p, p is always enabled. */
try_to_disable_aggr_kprobe(p);
if (!kprobes_all_disarmed && kprobe_disabled(p))
disarm_kprobe(p);
out:
mutex_unlock(&kprobe_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(disable_kprobe);
/* Enable one kprobe */
int __kprobes enable_kprobe(struct kprobe *kp)
{
int ret = 0;
struct kprobe *p;
mutex_lock(&kprobe_mutex);
/* Check whether specified probe is valid. */
p = __get_valid_kprobe(kp);
if (unlikely(p == NULL)) {
ret = -EINVAL;
goto out;
}
if (kprobe_gone(kp)) {
/* This kprobe has gone, we couldn't enable it. */
ret = -EINVAL;
goto out;
}
if (p != kp)
kp->flags &= ~KPROBE_FLAG_DISABLED;
if (!kprobes_all_disarmed && kprobe_disabled(p)) {
p->flags &= ~KPROBE_FLAG_DISABLED;
arm_kprobe(p);
}
out:
mutex_unlock(&kprobe_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(enable_kprobe);
static void __kprobes arm_all_kprobes(void) static void __kprobes arm_all_kprobes(void)
{ {
struct hlist_head *head; struct hlist_head *head;

4
kernel/lockdep.c
View file

@ -3232,7 +3232,7 @@ void lock_release(struct lockdep_map *lock, int nested,
raw_local_irq_save(flags); raw_local_irq_save(flags);
check_flags(flags); check_flags(flags);
current->lockdep_recursion = 1; current->lockdep_recursion = 1;
trace_lock_release(lock, nested, ip); trace_lock_release(lock, ip);
__lock_release(lock, nested, ip); __lock_release(lock, nested, ip);
current->lockdep_recursion = 0; current->lockdep_recursion = 0;
raw_local_irq_restore(flags); raw_local_irq_restore(flags);
@ -3385,7 +3385,7 @@ found_it:
hlock->holdtime_stamp = now; hlock->holdtime_stamp = now;
} }
trace_lock_acquired(lock, ip, waittime); trace_lock_acquired(lock, ip);
stats = get_lock_stats(hlock_class(hlock)); stats = get_lock_stats(hlock_class(hlock));
if (waittime) { if (waittime) {

371
kernel/perf_event.c
View file

@ -16,6 +16,7 @@
#include <linux/file.h> #include <linux/file.h>
#include <linux/poll.h> #include <linux/poll.h>
#include <linux/slab.h> #include <linux/slab.h>
#include <linux/hash.h>
#include <linux/sysfs.h> #include <linux/sysfs.h>
#include <linux/dcache.h> #include <linux/dcache.h>
#include <linux/percpu.h> #include <linux/percpu.h>
@ -82,14 +83,6 @@ extern __weak const struct pmu *hw_perf_event_init(struct perf_event *event)
void __weak hw_perf_disable(void) { barrier(); } void __weak hw_perf_disable(void) { barrier(); }
void __weak hw_perf_enable(void) { barrier(); } void __weak hw_perf_enable(void) { barrier(); }
int __weak
hw_perf_group_sched_in(struct perf_event *group_leader,
struct perf_cpu_context *cpuctx,
struct perf_event_context *ctx)
{
return 0;
}
void __weak perf_event_print_debug(void) { } void __weak perf_event_print_debug(void) { }
static DEFINE_PER_CPU(int, perf_disable_count); static DEFINE_PER_CPU(int, perf_disable_count);
@ -262,6 +255,18 @@ static void update_event_times(struct perf_event *event)
event->total_time_running = run_end - event->tstamp_running; event->total_time_running = run_end - event->tstamp_running;
} }
/*
* Update total_time_enabled and total_time_running for all events in a group.
*/
static void update_group_times(struct perf_event *leader)
{
struct perf_event *event;
update_event_times(leader);
list_for_each_entry(event, &leader->sibling_list, group_entry)
update_event_times(event);
}
static struct list_head * static struct list_head *
ctx_group_list(struct perf_event *event, struct perf_event_context *ctx) ctx_group_list(struct perf_event *event, struct perf_event_context *ctx)
{ {
@ -315,8 +320,6 @@ list_add_event(struct perf_event *event, struct perf_event_context *ctx)
static void static void
list_del_event(struct perf_event *event, struct perf_event_context *ctx) list_del_event(struct perf_event *event, struct perf_event_context *ctx)
{ {
struct perf_event *sibling, *tmp;
if (list_empty(&event->group_entry)) if (list_empty(&event->group_entry))
return; return;
ctx->nr_events--; ctx->nr_events--;
@ -329,7 +332,7 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)
if (event->group_leader != event) if (event->group_leader != event)
event->group_leader->nr_siblings--; event->group_leader->nr_siblings--;
update_event_times(event); update_group_times(event);
/* /*
* If event was in error state, then keep it * If event was in error state, then keep it
@ -340,6 +343,12 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)
*/ */
if (event->state > PERF_EVENT_STATE_OFF) if (event->state > PERF_EVENT_STATE_OFF)
event->state = PERF_EVENT_STATE_OFF; event->state = PERF_EVENT_STATE_OFF;
}
static void
perf_destroy_group(struct perf_event *event, struct perf_event_context *ctx)
{
struct perf_event *sibling, *tmp;
/* /*
* If this was a group event with sibling events then * If this was a group event with sibling events then
@ -504,18 +513,6 @@ retry:
raw_spin_unlock_irq(&ctx->lock); raw_spin_unlock_irq(&ctx->lock);
} }
/*
* Update total_time_enabled and total_time_running for all events in a group.
*/
static void update_group_times(struct perf_event *leader)
{
struct perf_event *event;
update_event_times(leader);
list_for_each_entry(event, &leader->sibling_list, group_entry)
update_event_times(event);
}
/* /*
* Cross CPU call to disable a performance event * Cross CPU call to disable a performance event
*/ */
@ -640,15 +637,20 @@ group_sched_in(struct perf_event *group_event,
struct perf_cpu_context *cpuctx, struct perf_cpu_context *cpuctx,
struct perf_event_context *ctx) struct perf_event_context *ctx)
{ {
struct perf_event *event, *partial_group; struct perf_event *event, *partial_group = NULL;
const struct pmu *pmu = group_event->pmu;
bool txn = false;
int ret; int ret;
if (group_event->state == PERF_EVENT_STATE_OFF) if (group_event->state == PERF_EVENT_STATE_OFF)
return 0; return 0;
ret = hw_perf_group_sched_in(group_event, cpuctx, ctx); /* Check if group transaction availabe */
if (ret) if (pmu->start_txn)
return ret < 0 ? ret : 0; txn = true;
if (txn)
pmu->start_txn(pmu);
if (event_sched_in(group_event, cpuctx, ctx)) if (event_sched_in(group_event, cpuctx, ctx))
return -EAGAIN; return -EAGAIN;
@ -663,9 +665,19 @@ group_sched_in(struct perf_event *group_event,
} }
} }
if (!txn)
return 0; return 0;
ret = pmu->commit_txn(pmu);
if (!ret) {
pmu->cancel_txn(pmu);
return 0;
}
group_error: group_error:
if (txn)
pmu->cancel_txn(pmu);
/* /*
* Groups can be scheduled in as one unit only, so undo any * Groups can be scheduled in as one unit only, so undo any
* partial group before returning: * partial group before returning:
@ -1367,6 +1379,8 @@ void perf_event_task_sched_in(struct task_struct *task)
if (cpuctx->task_ctx == ctx) if (cpuctx->task_ctx == ctx)
return; return;
perf_disable();
/* /*
* We want to keep the following priority order: * We want to keep the following priority order:
* cpu pinned (that don't need to move), task pinned, * cpu pinned (that don't need to move), task pinned,
@ -1379,6 +1393,8 @@ void perf_event_task_sched_in(struct task_struct *task)
ctx_sched_in(ctx, cpuctx, EVENT_FLEXIBLE); ctx_sched_in(ctx, cpuctx, EVENT_FLEXIBLE);
cpuctx->task_ctx = ctx; cpuctx->task_ctx = ctx;
perf_enable();
} }
#define MAX_INTERRUPTS (~0ULL) #define MAX_INTERRUPTS (~0ULL)
@ -1856,9 +1872,30 @@ int perf_event_release_kernel(struct perf_event *event)
{ {
struct perf_event_context *ctx = event->ctx; struct perf_event_context *ctx = event->ctx;
/*
* Remove from the PMU, can't get re-enabled since we got
* here because the last ref went.
*/
perf_event_disable(event);
WARN_ON_ONCE(ctx->parent_ctx); WARN_ON_ONCE(ctx->parent_ctx);
mutex_lock(&ctx->mutex); /*
perf_event_remove_from_context(event); * There are two ways this annotation is useful:
*
* 1) there is a lock recursion from perf_event_exit_task
* see the comment there.
*
* 2) there is a lock-inversion with mmap_sem through
* perf_event_read_group(), which takes faults while
* holding ctx->mutex, however this is called after
* the last filedesc died, so there is no possibility
* to trigger the AB-BA case.
*/
mutex_lock_nested(&ctx->mutex, SINGLE_DEPTH_NESTING);
raw_spin_lock_irq(&ctx->lock);
list_del_event(event, ctx);
perf_destroy_group(event, ctx);
raw_spin_unlock_irq(&ctx->lock);
mutex_unlock(&ctx->mutex); mutex_unlock(&ctx->mutex);
mutex_lock(&event->owner->perf_event_mutex); mutex_lock(&event->owner->perf_event_mutex);
@ -2642,6 +2679,7 @@ static int perf_fasync(int fd, struct file *filp, int on)
} }
static const struct file_operations perf_fops = { static const struct file_operations perf_fops = {
.llseek = no_llseek,
.release = perf_release, .release = perf_release,
.read = perf_read, .read = perf_read,
.poll = perf_poll, .poll = perf_poll,
@ -2791,6 +2829,27 @@ void perf_arch_fetch_caller_regs(struct pt_regs *regs, unsigned long ip, int ski
} }
/*
* We assume there is only KVM supporting the callbacks.
* Later on, we might change it to a list if there is
* another virtualization implementation supporting the callbacks.
*/
struct perf_guest_info_callbacks *perf_guest_cbs;
int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *cbs)
{
perf_guest_cbs = cbs;
return 0;
}
EXPORT_SYMBOL_GPL(perf_register_guest_info_callbacks);
int perf_unregister_guest_info_callbacks(struct perf_guest_info_callbacks *cbs)
{
perf_guest_cbs = NULL;
return 0;
}
EXPORT_SYMBOL_GPL(perf_unregister_guest_info_callbacks);
/* /*
* Output * Output
*/ */
@ -3743,7 +3802,7 @@ void __perf_event_mmap(struct vm_area_struct *vma)
.event_id = { .event_id = {
.header = { .header = {
.type = PERF_RECORD_MMAP, .type = PERF_RECORD_MMAP,
.misc = 0, .misc = PERF_RECORD_MISC_USER,
/* .size */ /* .size */
}, },
/* .pid */ /* .pid */
@ -3961,36 +4020,6 @@ static void perf_swevent_add(struct perf_event *event, u64 nr,
perf_swevent_overflow(event, 0, nmi, data, regs); perf_swevent_overflow(event, 0, nmi, data, regs);
} }
static int perf_swevent_is_counting(struct perf_event *event)
{
/*
* The event is active, we're good!
*/
if (event->state == PERF_EVENT_STATE_ACTIVE)
return 1;
/*
* The event is off/error, not counting.
*/
if (event->state != PERF_EVENT_STATE_INACTIVE)
return 0;
/*
* The event is inactive, if the context is active
* we're part of a group that didn't make it on the 'pmu',
* not counting.
*/
if (event->ctx->is_active)
return 0;
/*
* We're inactive and the context is too, this means the
* task is scheduled out, we're counting events that happen
* to us, like migration events.
*/
return 1;
}
static int perf_tp_event_match(struct perf_event *event, static int perf_tp_event_match(struct perf_event *event,
struct perf_sample_data *data); struct perf_sample_data *data);
@ -4014,12 +4043,6 @@ static int perf_swevent_match(struct perf_event *event,
struct perf_sample_data *data, struct perf_sample_data *data,
struct pt_regs *regs) struct pt_regs *regs)
{ {
if (event->cpu != -1 && event->cpu != smp_processor_id())
return 0;
if (!perf_swevent_is_counting(event))
return 0;
if (event->attr.type != type) if (event->attr.type != type)
return 0; return 0;
@ -4036,18 +4059,53 @@ static int perf_swevent_match(struct perf_event *event,
return 1; return 1;
} }
static void perf_swevent_ctx_event(struct perf_event_context *ctx, static inline u64 swevent_hash(u64 type, u32 event_id)
enum perf_type_id type, {
u32 event_id, u64 nr, int nmi, u64 val = event_id | (type << 32);
return hash_64(val, SWEVENT_HLIST_BITS);
}
static struct hlist_head *
find_swevent_head(struct perf_cpu_context *ctx, u64 type, u32 event_id)
{
u64 hash;
struct swevent_hlist *hlist;
hash = swevent_hash(type, event_id);
hlist = rcu_dereference(ctx->swevent_hlist);
if (!hlist)
return NULL;
return &hlist->heads[hash];
}
static void do_perf_sw_event(enum perf_type_id type, u32 event_id,
u64 nr, int nmi,
struct perf_sample_data *data, struct perf_sample_data *data,
struct pt_regs *regs) struct pt_regs *regs)
{ {
struct perf_cpu_context *cpuctx;
struct perf_event *event; struct perf_event *event;
struct hlist_node *node;
struct hlist_head *head;
list_for_each_entry_rcu(event, &ctx->event_list, event_entry) { cpuctx = &__get_cpu_var(perf_cpu_context);
rcu_read_lock();
head = find_swevent_head(cpuctx, type, event_id);
if (!head)
goto end;
hlist_for_each_entry_rcu(event, node, head, hlist_entry) {
if (perf_swevent_match(event, type, event_id, data, regs)) if (perf_swevent_match(event, type, event_id, data, regs))
perf_swevent_add(event, nr, nmi, data, regs); perf_swevent_add(event, nr, nmi, data, regs);
} }
end:
rcu_read_unlock();
} }
int perf_swevent_get_recursion_context(void) int perf_swevent_get_recursion_context(void)
@ -4085,27 +4143,6 @@ void perf_swevent_put_recursion_context(int rctx)
} }
EXPORT_SYMBOL_GPL(perf_swevent_put_recursion_context); EXPORT_SYMBOL_GPL(perf_swevent_put_recursion_context);
static void do_perf_sw_event(enum perf_type_id type, u32 event_id,
u64 nr, int nmi,
struct perf_sample_data *data,
struct pt_regs *regs)
{
struct perf_cpu_context *cpuctx;
struct perf_event_context *ctx;
cpuctx = &__get_cpu_var(perf_cpu_context);
rcu_read_lock();
perf_swevent_ctx_event(&cpuctx->ctx, type, event_id,
nr, nmi, data, regs);
/*
* doesn't really matter which of the child contexts the
* events ends up in.
*/
ctx = rcu_dereference(current->perf_event_ctxp);
if (ctx)
perf_swevent_ctx_event(ctx, type, event_id, nr, nmi, data, regs);
rcu_read_unlock();
}
void __perf_sw_event(u32 event_id, u64 nr, int nmi, void __perf_sw_event(u32 event_id, u64 nr, int nmi,
struct pt_regs *regs, u64 addr) struct pt_regs *regs, u64 addr)
@ -4131,16 +4168,28 @@ static void perf_swevent_read(struct perf_event *event)
static int perf_swevent_enable(struct perf_event *event) static int perf_swevent_enable(struct perf_event *event)
{ {
struct hw_perf_event *hwc = &event->hw; struct hw_perf_event *hwc = &event->hw;
struct perf_cpu_context *cpuctx;
struct hlist_head *head;
cpuctx = &__get_cpu_var(perf_cpu_context);
if (hwc->sample_period) { if (hwc->sample_period) {
hwc->last_period = hwc->sample_period; hwc->last_period = hwc->sample_period;
perf_swevent_set_period(event); perf_swevent_set_period(event);
} }
head = find_swevent_head(cpuctx, event->attr.type, event->attr.config);
if (WARN_ON_ONCE(!head))
return -EINVAL;
hlist_add_head_rcu(&event->hlist_entry, head);
return 0; return 0;
} }
static void perf_swevent_disable(struct perf_event *event) static void perf_swevent_disable(struct perf_event *event)
{ {
hlist_del_rcu(&event->hlist_entry);
} }
static const struct pmu perf_ops_generic = { static const struct pmu perf_ops_generic = {
@ -4168,15 +4217,8 @@ static enum hrtimer_restart perf_swevent_hrtimer(struct hrtimer *hrtimer)
perf_sample_data_init(&data, 0); perf_sample_data_init(&data, 0);
data.period = event->hw.last_period; data.period = event->hw.last_period;
regs = get_irq_regs(); regs = get_irq_regs();
/*
* In case we exclude kernel IPs or are somehow not in interrupt
* context, provide the next best thing, the user IP.
*/
if ((event->attr.exclude_kernel || !regs) &&
!event->attr.exclude_user)
regs = task_pt_regs(current);
if (regs) { if (regs && !perf_exclude_event(event, regs)) {
if (!(event->attr.exclude_idle && current->pid == 0)) if (!(event->attr.exclude_idle && current->pid == 0))
if (perf_event_overflow(event, 0, &data, regs)) if (perf_event_overflow(event, 0, &data, regs))
ret = HRTIMER_NORESTART; ret = HRTIMER_NORESTART;
@ -4324,6 +4366,105 @@ static const struct pmu perf_ops_task_clock = {
.read = task_clock_perf_event_read, .read = task_clock_perf_event_read,
}; };
static void swevent_hlist_release_rcu(struct rcu_head *rcu_head)
{
struct swevent_hlist *hlist;
hlist = container_of(rcu_head, struct swevent_hlist, rcu_head);
kfree(hlist);
}
static void swevent_hlist_release(struct perf_cpu_context *cpuctx)
{
struct swevent_hlist *hlist;
if (!cpuctx->swevent_hlist)
return;
hlist = cpuctx->swevent_hlist;
rcu_assign_pointer(cpuctx->swevent_hlist, NULL);
call_rcu(&hlist->rcu_head, swevent_hlist_release_rcu);
}
static void swevent_hlist_put_cpu(struct perf_event *event, int cpu)
{
struct perf_cpu_context *cpuctx = &per_cpu(perf_cpu_context, cpu);
mutex_lock(&cpuctx->hlist_mutex);
if (!--cpuctx->hlist_refcount)
swevent_hlist_release(cpuctx);
mutex_unlock(&cpuctx->hlist_mutex);
}
static void swevent_hlist_put(struct perf_event *event)
{
int cpu;
if (event->cpu != -1) {
swevent_hlist_put_cpu(event, event->cpu);
return;
}
for_each_possible_cpu(cpu)
swevent_hlist_put_cpu(event, cpu);
}
static int swevent_hlist_get_cpu(struct perf_event *event, int cpu)
{
struct perf_cpu_context *cpuctx = &per_cpu(perf_cpu_context, cpu);
int err = 0;
mutex_lock(&cpuctx->hlist_mutex);
if (!cpuctx->swevent_hlist && cpu_online(cpu)) {
struct swevent_hlist *hlist;
hlist = kzalloc(sizeof(*hlist), GFP_KERNEL);
if (!hlist) {
err = -ENOMEM;
goto exit;
}
rcu_assign_pointer(cpuctx->swevent_hlist, hlist);
}
cpuctx->hlist_refcount++;
exit:
mutex_unlock(&cpuctx->hlist_mutex);
return err;
}
static int swevent_hlist_get(struct perf_event *event)
{
int err;
int cpu, failed_cpu;
if (event->cpu != -1)
return swevent_hlist_get_cpu(event, event->cpu);
get_online_cpus();
for_each_possible_cpu(cpu) {
err = swevent_hlist_get_cpu(event, cpu);
if (err) {
failed_cpu = cpu;
goto fail;
}
}
put_online_cpus();
return 0;
fail:
for_each_possible_cpu(cpu) {
if (cpu == failed_cpu)
break;
swevent_hlist_put_cpu(event, cpu);
}
put_online_cpus();
return err;
}
#ifdef CONFIG_EVENT_TRACING #ifdef CONFIG_EVENT_TRACING
void perf_tp_event(int event_id, u64 addr, u64 count, void *record, void perf_tp_event(int event_id, u64 addr, u64 count, void *record,
@ -4357,10 +4498,13 @@ static int perf_tp_event_match(struct perf_event *event,
static void tp_perf_event_destroy(struct perf_event *event) static void tp_perf_event_destroy(struct perf_event *event)
{ {
perf_trace_disable(event->attr.config); perf_trace_disable(event->attr.config);
swevent_hlist_put(event);
} }
static const struct pmu *tp_perf_event_init(struct perf_event *event) static const struct pmu *tp_perf_event_init(struct perf_event *event)
{ {
int err;
/* /*
* Raw tracepoint data is a severe data leak, only allow root to * Raw tracepoint data is a severe data leak, only allow root to
* have these. * have these.
@ -4374,6 +4518,11 @@ static const struct pmu *tp_perf_event_init(struct perf_event *event)
return NULL; return NULL;
event->destroy = tp_perf_event_destroy; event->destroy = tp_perf_event_destroy;
err = swevent_hlist_get(event);
if (err) {
perf_trace_disable(event->attr.config);
return ERR_PTR(err);
}
return &perf_ops_generic; return &perf_ops_generic;
} }
@ -4474,6 +4623,7 @@ static void sw_perf_event_destroy(struct perf_event *event)
WARN_ON(event->parent); WARN_ON(event->parent);
atomic_dec(&perf_swevent_enabled[event_id]); atomic_dec(&perf_swevent_enabled[event_id]);
swevent_hlist_put(event);
} }
static const struct pmu *sw_perf_event_init(struct perf_event *event) static const struct pmu *sw_perf_event_init(struct perf_event *event)
@ -4512,6 +4662,12 @@ static const struct pmu *sw_perf_event_init(struct perf_event *event)
case PERF_COUNT_SW_ALIGNMENT_FAULTS: case PERF_COUNT_SW_ALIGNMENT_FAULTS:
case PERF_COUNT_SW_EMULATION_FAULTS: case PERF_COUNT_SW_EMULATION_FAULTS:
if (!event->parent) { if (!event->parent) {
int err;
err = swevent_hlist_get(event);
if (err)
return ERR_PTR(err);
atomic_inc(&perf_swevent_enabled[event_id]); atomic_inc(&perf_swevent_enabled[event_id]);
event->destroy = sw_perf_event_destroy; event->destroy = sw_perf_event_destroy;
} }
@ -5176,7 +5332,7 @@ void perf_event_exit_task(struct task_struct *child)
* *
* But since its the parent context it won't be the same instance. * But since its the parent context it won't be the same instance.
*/ */
mutex_lock_nested(&child_ctx->mutex, SINGLE_DEPTH_NESTING); mutex_lock(&child_ctx->mutex);
again: again:
list_for_each_entry_safe(child_event, tmp, &child_ctx->pinned_groups, list_for_each_entry_safe(child_event, tmp, &child_ctx->pinned_groups,
@ -5384,6 +5540,7 @@ static void __init perf_event_init_all_cpus(void)
for_each_possible_cpu(cpu) { for_each_possible_cpu(cpu) {
cpuctx = &per_cpu(perf_cpu_context, cpu); cpuctx = &per_cpu(perf_cpu_context, cpu);
mutex_init(&cpuctx->hlist_mutex);
__perf_event_init_context(&cpuctx->ctx, NULL); __perf_event_init_context(&cpuctx->ctx, NULL);
} }
} }
@ -5397,6 +5554,16 @@ static void __cpuinit perf_event_init_cpu(int cpu)
spin_lock(&perf_resource_lock); spin_lock(&perf_resource_lock);
cpuctx->max_pertask = perf_max_events - perf_reserved_percpu; cpuctx->max_pertask = perf_max_events - perf_reserved_percpu;
spin_unlock(&perf_resource_lock); spin_unlock(&perf_resource_lock);
mutex_lock(&cpuctx->hlist_mutex);
if (cpuctx->hlist_refcount > 0) {
struct swevent_hlist *hlist;
hlist = kzalloc(sizeof(*hlist), GFP_KERNEL);
WARN_ON_ONCE(!hlist);
rcu_assign_pointer(cpuctx->swevent_hlist, hlist);
}
mutex_unlock(&cpuctx->hlist_mutex);
} }
#ifdef CONFIG_HOTPLUG_CPU #ifdef CONFIG_HOTPLUG_CPU
@ -5416,6 +5583,10 @@ static void perf_event_exit_cpu(int cpu)
struct perf_cpu_context *cpuctx = &per_cpu(perf_cpu_context, cpu); struct perf_cpu_context *cpuctx = &per_cpu(perf_cpu_context, cpu);
struct perf_event_context *ctx = &cpuctx->ctx; struct perf_event_context *ctx = &cpuctx->ctx;
mutex_lock(&cpuctx->hlist_mutex);
swevent_hlist_release(cpuctx);
mutex_unlock(&cpuctx->hlist_mutex);
mutex_lock(&ctx->mutex); mutex_lock(&ctx->mutex);
smp_call_function_single(cpu, __perf_event_exit_cpu, NULL, 1); smp_call_function_single(cpu, __perf_event_exit_cpu, NULL, 1);
mutex_unlock(&ctx->mutex); mutex_unlock(&ctx->mutex);

1
kernel/ptrace.c
View file

@ -75,7 +75,6 @@ void __ptrace_unlink(struct task_struct *child)
child->parent = child->real_parent; child->parent = child->real_parent;
list_del_init(&child->ptrace_entry); list_del_init(&child->ptrace_entry);
arch_ptrace_untrace(child);
if (task_is_traced(child)) if (task_is_traced(child))
ptrace_untrace(child); ptrace_untrace(child);
} }

43
kernel/sched.c
View file

@ -2087,49 +2087,6 @@ migrate_task(struct task_struct *p, int dest_cpu, struct migration_req *req)
return 1; return 1;
} }
/*
* wait_task_context_switch - wait for a thread to complete at least one
* context switch.
*
* @p must not be current.
*/
void wait_task_context_switch(struct task_struct *p)
{
unsigned long nvcsw, nivcsw, flags;
int running;
struct rq *rq;
nvcsw = p->nvcsw;
nivcsw = p->nivcsw;
for (;;) {
/*
* The runqueue is assigned before the actual context
* switch. We need to take the runqueue lock.
*
* We could check initially without the lock but it is
* very likely that we need to take the lock in every
* iteration.
*/
rq = task_rq_lock(p, &flags);
running = task_running(rq, p);
task_rq_unlock(rq, &flags);
if (likely(!running))
break;
/*
* The switch count is incremented before the actual
* context switch. We thus wait for two switches to be
* sure at least one completed.
*/
if ((p->nvcsw - nvcsw) > 1)
break;
if ((p->nivcsw - nivcsw) > 1)
break;
cpu_relax();
}
}
/* /*
* wait_task_inactive - wait for a thread to unschedule. * wait_task_inactive - wait for a thread to unschedule.
* *

11
kernel/trace/Kconfig
View file

@ -44,9 +44,6 @@ config HAVE_FTRACE_MCOUNT_RECORD
help help
See Documentation/trace/ftrace-design.txt See Documentation/trace/ftrace-design.txt
config HAVE_HW_BRANCH_TRACER
bool
config HAVE_SYSCALL_TRACEPOINTS config HAVE_SYSCALL_TRACEPOINTS
bool bool
help help
@ -374,14 +371,6 @@ config STACK_TRACER
Say N if unsure. Say N if unsure.
config HW_BRANCH_TRACER
depends on HAVE_HW_BRANCH_TRACER
bool "Trace hw branches"
select GENERIC_TRACER
help
This tracer records all branches on the system in a circular
buffer, giving access to the last N branches for each cpu.
config KMEMTRACE config KMEMTRACE
bool "Trace SLAB allocations" bool "Trace SLAB allocations"
select GENERIC_TRACER select GENERIC_TRACER

1
kernel/trace/Makefile
View file

@ -41,7 +41,6 @@ obj-$(CONFIG_MMIOTRACE) += trace_mmiotrace.o
obj-$(CONFIG_BOOT_TRACER) += trace_boot.o obj-$(CONFIG_BOOT_TRACER) += trace_boot.o
obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += trace_functions_graph.o obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += trace_functions_graph.o
obj-$(CONFIG_TRACE_BRANCH_PROFILING) += trace_branch.o obj-$(CONFIG_TRACE_BRANCH_PROFILING) += trace_branch.o
obj-$(CONFIG_HW_BRANCH_TRACER) += trace_hw_branches.o
obj-$(CONFIG_KMEMTRACE) += kmemtrace.o obj-$(CONFIG_KMEMTRACE) += kmemtrace.o
obj-$(CONFIG_WORKQUEUE_TRACER) += trace_workqueue.o obj-$(CONFIG_WORKQUEUE_TRACER) += trace_workqueue.o
obj-$(CONFIG_BLK_DEV_IO_TRACE) += blktrace.o obj-$(CONFIG_BLK_DEV_IO_TRACE) += blktrace.o

20
kernel/trace/trace.h
View file

@ -34,7 +34,6 @@ enum trace_type {
TRACE_GRAPH_RET, TRACE_GRAPH_RET,
TRACE_GRAPH_ENT, TRACE_GRAPH_ENT,
TRACE_USER_STACK, TRACE_USER_STACK,
TRACE_HW_BRANCHES,
TRACE_KMEM_ALLOC, TRACE_KMEM_ALLOC,
TRACE_KMEM_FREE, TRACE_KMEM_FREE,
TRACE_BLK, TRACE_BLK,
@ -103,29 +102,17 @@ struct syscall_trace_exit {
long ret; long ret;
}; };
struct kprobe_trace_entry { struct kprobe_trace_entry_head {
struct trace_entry ent; struct trace_entry ent;
unsigned long ip; unsigned long ip;
int nargs;
unsigned long args[];
}; };
#define SIZEOF_KPROBE_TRACE_ENTRY(n) \ struct kretprobe_trace_entry_head {
(offsetof(struct kprobe_trace_entry, args) + \
(sizeof(unsigned long) * (n)))
struct kretprobe_trace_entry {
struct trace_entry ent; struct trace_entry ent;
unsigned long func; unsigned long func;
unsigned long ret_ip; unsigned long ret_ip;
int nargs;
unsigned long args[];
}; };
#define SIZEOF_KRETPROBE_TRACE_ENTRY(n) \
(offsetof(struct kretprobe_trace_entry, args) + \
(sizeof(unsigned long) * (n)))
/* /*
* trace_flag_type is an enumeration that holds different * trace_flag_type is an enumeration that holds different
* states when a trace occurs. These are: * states when a trace occurs. These are:
@ -229,7 +216,6 @@ extern void __ftrace_bad_type(void);
TRACE_GRAPH_ENT); \ TRACE_GRAPH_ENT); \
IF_ASSIGN(var, ent, struct ftrace_graph_ret_entry, \ IF_ASSIGN(var, ent, struct ftrace_graph_ret_entry, \
TRACE_GRAPH_RET); \ TRACE_GRAPH_RET); \
IF_ASSIGN(var, ent, struct hw_branch_entry, TRACE_HW_BRANCHES);\
IF_ASSIGN(var, ent, struct kmemtrace_alloc_entry, \ IF_ASSIGN(var, ent, struct kmemtrace_alloc_entry, \
TRACE_KMEM_ALLOC); \ TRACE_KMEM_ALLOC); \
IF_ASSIGN(var, ent, struct kmemtrace_free_entry, \ IF_ASSIGN(var, ent, struct kmemtrace_free_entry, \
@ -467,8 +453,6 @@ extern int trace_selftest_startup_sysprof(struct tracer *trace,
struct trace_array *tr); struct trace_array *tr);
extern int trace_selftest_startup_branch(struct tracer *trace, extern int trace_selftest_startup_branch(struct tracer *trace,
struct trace_array *tr); struct trace_array *tr);
extern int trace_selftest_startup_hw_branches(struct tracer *trace,
struct trace_array *tr);
extern int trace_selftest_startup_ksym(struct tracer *trace, extern int trace_selftest_startup_ksym(struct tracer *trace,
struct trace_array *tr); struct trace_array *tr);
#endif /* CONFIG_FTRACE_STARTUP_TEST */ #endif /* CONFIG_FTRACE_STARTUP_TEST */

12
kernel/trace/trace_entries.h
View file

@ -318,18 +318,6 @@ FTRACE_ENTRY(branch, trace_branch,
__entry->func, __entry->file, __entry->correct) __entry->func, __entry->file, __entry->correct)
); );
FTRACE_ENTRY(hw_branch, hw_branch_entry,
TRACE_HW_BRANCHES,
F_STRUCT(
__field( u64, from )
__field( u64, to )
),
F_printk("from: %llx to: %llx", __entry->from, __entry->to)
);
FTRACE_ENTRY(kmem_alloc, kmemtrace_alloc_entry, FTRACE_ENTRY(kmem_alloc, kmemtrace_alloc_entry,
TRACE_KMEM_ALLOC, TRACE_KMEM_ALLOC,

2
kernel/trace/trace_events_filter.c
View file

@ -1398,7 +1398,7 @@ int ftrace_profile_set_filter(struct perf_event *event, int event_id,
} }
err = -EINVAL; err = -EINVAL;
if (!call) if (&call->list == &ftrace_events)
goto out_unlock; goto out_unlock;
err = -EEXIST; err = -EEXIST;

312
kernel/trace/trace_hw_branches.c
View file

@ -1,312 +0,0 @@
/*
* h/w branch tracer for x86 based on BTS
*
* Copyright (C) 2008-2009 Intel Corporation.
* Markus Metzger <markus.t.metzger@gmail.com>, 2008-2009
*/
#include <linux/kallsyms.h>
#include <linux/debugfs.h>
#include <linux/ftrace.h>
#include <linux/module.h>
#include <linux/cpu.h>
#include <linux/smp.h>
#include <linux/fs.h>
#include <asm/ds.h>
#include "trace_output.h"
#include "trace.h"
#define BTS_BUFFER_SIZE (1 << 13)
static DEFINE_PER_CPU(struct bts_tracer *, hwb_tracer);
static DEFINE_PER_CPU(unsigned char[BTS_BUFFER_SIZE], hwb_buffer);
#define this_tracer per_cpu(hwb_tracer, smp_processor_id())
static int trace_hw_branches_enabled __read_mostly;
static int trace_hw_branches_suspended __read_mostly;
static struct trace_array *hw_branch_trace __read_mostly;
static void bts_trace_init_cpu(int cpu)
{
per_cpu(hwb_tracer, cpu) =
ds_request_bts_cpu(cpu, per_cpu(hwb_buffer, cpu),
BTS_BUFFER_SIZE, NULL, (size_t)-1,
BTS_KERNEL);
if (IS_ERR(per_cpu(hwb_tracer, cpu)))
per_cpu(hwb_tracer, cpu) = NULL;
}
static int bts_trace_init(struct trace_array *tr)
{
int cpu;
hw_branch_trace = tr;
trace_hw_branches_enabled = 0;
get_online_cpus();
for_each_online_cpu(cpu) {
bts_trace_init_cpu(cpu);
if (likely(per_cpu(hwb_tracer, cpu)))
trace_hw_branches_enabled = 1;
}
trace_hw_branches_suspended = 0;
put_online_cpus();
/* If we could not enable tracing on a single cpu, we fail. */
return trace_hw_branches_enabled ? 0 : -EOPNOTSUPP;
}
static void bts_trace_reset(struct trace_array *tr)
{
int cpu;
get_online_cpus();
for_each_online_cpu(cpu) {
if (likely(per_cpu(hwb_tracer, cpu))) {
ds_release_bts(per_cpu(hwb_tracer, cpu));
per_cpu(hwb_tracer, cpu) = NULL;
}
}
trace_hw_branches_enabled = 0;
trace_hw_branches_suspended = 0;
put_online_cpus();
}
static void bts_trace_start(struct trace_array *tr)
{
int cpu;
get_online_cpus();
for_each_online_cpu(cpu)
if (likely(per_cpu(hwb_tracer, cpu)))
ds_resume_bts(per_cpu(hwb_tracer, cpu));
trace_hw_branches_suspended = 0;
put_online_cpus();
}
static void bts_trace_stop(struct trace_array *tr)
{
int cpu;
get_online_cpus();
for_each_online_cpu(cpu)
if (likely(per_cpu(hwb_tracer, cpu)))
ds_suspend_bts(per_cpu(hwb_tracer, cpu));
trace_hw_branches_suspended = 1;
put_online_cpus();
}
static int __cpuinit bts_hotcpu_handler(struct notifier_block *nfb,
unsigned long action, void *hcpu)
{
int cpu = (long)hcpu;
switch (action) {
case CPU_ONLINE:
case CPU_DOWN_FAILED:
/* The notification is sent with interrupts enabled. */
if (trace_hw_branches_enabled) {
bts_trace_init_cpu(cpu);
if (trace_hw_branches_suspended &&
likely(per_cpu(hwb_tracer, cpu)))
ds_suspend_bts(per_cpu(hwb_tracer, cpu));
}
break;
case CPU_DOWN_PREPARE:
/* The notification is sent with interrupts enabled. */
if (likely(per_cpu(hwb_tracer, cpu))) {
ds_release_bts(per_cpu(hwb_tracer, cpu));
per_cpu(hwb_tracer, cpu) = NULL;
}
}
return NOTIFY_DONE;
}
static struct notifier_block bts_hotcpu_notifier __cpuinitdata = {
.notifier_call = bts_hotcpu_handler
};
static void bts_trace_print_header(struct seq_file *m)
{
seq_puts(m, "# CPU# TO <- FROM\n");
}
static enum print_line_t bts_trace_print_line(struct trace_iterator *iter)
{
unsigned long symflags = TRACE_ITER_SYM_OFFSET;
struct trace_entry *entry = iter->ent;
struct trace_seq *seq = &iter->seq;
struct hw_branch_entry *it;
trace_assign_type(it, entry);
if (entry->type == TRACE_HW_BRANCHES) {
if (trace_seq_printf(seq, "%4d ", iter->cpu) &&
seq_print_ip_sym(seq, it->to, symflags) &&
trace_seq_printf(seq, "\t <- ") &&
seq_print_ip_sym(seq, it->from, symflags) &&
trace_seq_printf(seq, "\n"))
return TRACE_TYPE_HANDLED;
return TRACE_TYPE_PARTIAL_LINE;
}
return TRACE_TYPE_UNHANDLED;
}
void trace_hw_branch(u64 from, u64 to)
{
struct ftrace_event_call *call = &event_hw_branch;
struct trace_array *tr = hw_branch_trace;
struct ring_buffer_event *event;
struct ring_buffer *buf;
struct hw_branch_entry *entry;
unsigned long irq1;
int cpu;
if (unlikely(!tr))
return;
if (unlikely(!trace_hw_branches_enabled))
return;
local_irq_save(irq1);
cpu = raw_smp_processor_id();
if (atomic_inc_return(&tr->data[cpu]->disabled) != 1)
goto out;
buf = tr->buffer;
event = trace_buffer_lock_reserve(buf, TRACE_HW_BRANCHES,
sizeof(*entry), 0, 0);
if (!event)
goto out;
entry = ring_buffer_event_data(event);
tracing_generic_entry_update(&entry->ent, 0, from);
entry->ent.type = TRACE_HW_BRANCHES;
entry->from = from;
entry->to = to;
if (!filter_check_discard(call, entry, buf, event))
trace_buffer_unlock_commit(buf, event, 0, 0);
out:
atomic_dec(&tr->data[cpu]->disabled);
local_irq_restore(irq1);
}
static void trace_bts_at(const struct bts_trace *trace, void *at)
{
struct bts_struct bts;
int err = 0;
WARN_ON_ONCE(!trace->read);
if (!trace->read)
return;
err = trace->read(this_tracer, at, &bts);
if (err < 0)
return;
switch (bts.qualifier) {
case BTS_BRANCH:
trace_hw_branch(bts.variant.lbr.from, bts.variant.lbr.to);
break;
}
}
/*
* Collect the trace on the current cpu and write it into the ftrace buffer.
*
* pre: tracing must be suspended on the current cpu
*/
static void trace_bts_cpu(void *arg)
{
struct trace_array *tr = (struct trace_array *)arg;
const struct bts_trace *trace;
unsigned char *at;
if (unlikely(!tr))
return;
if (unlikely(atomic_read(&tr->data[raw_smp_processor_id()]->disabled)))
return;
if (unlikely(!this_tracer))
return;
trace = ds_read_bts(this_tracer);
if (!trace)
return;
for (at = trace->ds.top; (void *)at < trace->ds.end;
at += trace->ds.size)
trace_bts_at(trace, at);
for (at = trace->ds.begin; (void *)at < trace->ds.top;
at += trace->ds.size)
trace_bts_at(trace, at);
}
static void trace_bts_prepare(struct trace_iterator *iter)
{
int cpu;
get_online_cpus();
for_each_online_cpu(cpu)
if (likely(per_cpu(hwb_tracer, cpu)))
ds_suspend_bts(per_cpu(hwb_tracer, cpu));
/*
* We need to collect the trace on the respective cpu since ftrace
* implicitly adds the record for the current cpu.
* Once that is more flexible, we could collect the data from any cpu.
*/
on_each_cpu(trace_bts_cpu, iter->tr, 1);
for_each_online_cpu(cpu)
if (likely(per_cpu(hwb_tracer, cpu)))
ds_resume_bts(per_cpu(hwb_tracer, cpu));
put_online_cpus();
}
static void trace_bts_close(struct trace_iterator *iter)
{
tracing_reset_online_cpus(iter->tr);
}
void trace_hw_branch_oops(void)
{
if (this_tracer) {
ds_suspend_bts_noirq(this_tracer);
trace_bts_cpu(hw_branch_trace);
ds_resume_bts_noirq(this_tracer);
}
}
struct tracer bts_tracer __read_mostly =
{
.name = "hw-branch-tracer",
.init = bts_trace_init,
.reset = bts_trace_reset,
.print_header = bts_trace_print_header,
.print_line = bts_trace_print_line,
.start = bts_trace_start,
.stop = bts_trace_stop,
.open = trace_bts_prepare,
.close = trace_bts_close,
#ifdef CONFIG_FTRACE_SELFTEST
.selftest = trace_selftest_startup_hw_branches,
#endif /* CONFIG_FTRACE_SELFTEST */
};
__init static int init_bts_trace(void)
{
register_hotcpu_notifier(&bts_hotcpu_notifier);
return register_tracer(&bts_tracer);
}
device_initcall(init_bts_trace);

529
kernel/trace/trace_kprobe.c
View file

@ -29,6 +29,8 @@
#include <linux/ctype.h> #include <linux/ctype.h>
#include <linux/ptrace.h> #include <linux/ptrace.h>
#include <linux/perf_event.h> #include <linux/perf_event.h>
#include <linux/stringify.h>
#include <asm/bitsperlong.h>
#include "trace.h" #include "trace.h"
#include "trace_output.h" #include "trace_output.h"
@ -40,7 +42,6 @@
/* Reserved field names */ /* Reserved field names */
#define FIELD_STRING_IP "__probe_ip" #define FIELD_STRING_IP "__probe_ip"
#define FIELD_STRING_NARGS "__probe_nargs"
#define FIELD_STRING_RETIP "__probe_ret_ip" #define FIELD_STRING_RETIP "__probe_ret_ip"
#define FIELD_STRING_FUNC "__probe_func" #define FIELD_STRING_FUNC "__probe_func"
@ -52,56 +53,102 @@ const char *reserved_field_names[] = {
"common_tgid", "common_tgid",
"common_lock_depth", "common_lock_depth",
FIELD_STRING_IP, FIELD_STRING_IP,
FIELD_STRING_NARGS,
FIELD_STRING_RETIP, FIELD_STRING_RETIP,
FIELD_STRING_FUNC, FIELD_STRING_FUNC,
}; };
struct fetch_func { /* Printing function type */
unsigned long (*func)(struct pt_regs *, void *); typedef int (*print_type_func_t)(struct trace_seq *, const char *, void *);
#define PRINT_TYPE_FUNC_NAME(type) print_type_##type
#define PRINT_TYPE_FMT_NAME(type) print_type_format_##type
/* Printing in basic type function template */
#define DEFINE_BASIC_PRINT_TYPE_FUNC(type, fmt, cast) \
static __kprobes int PRINT_TYPE_FUNC_NAME(type)(struct trace_seq *s, \
const char *name, void *data)\
{ \
return trace_seq_printf(s, " %s=" fmt, name, (cast)*(type *)data);\
} \
static const char PRINT_TYPE_FMT_NAME(type)[] = fmt;
DEFINE_BASIC_PRINT_TYPE_FUNC(u8, "%x", unsigned int)
DEFINE_BASIC_PRINT_TYPE_FUNC(u16, "%x", unsigned int)
DEFINE_BASIC_PRINT_TYPE_FUNC(u32, "%lx", unsigned long)
DEFINE_BASIC_PRINT_TYPE_FUNC(u64, "%llx", unsigned long long)
DEFINE_BASIC_PRINT_TYPE_FUNC(s8, "%d", int)
DEFINE_BASIC_PRINT_TYPE_FUNC(s16, "%d", int)
DEFINE_BASIC_PRINT_TYPE_FUNC(s32, "%ld", long)
DEFINE_BASIC_PRINT_TYPE_FUNC(s64, "%lld", long long)
/* Data fetch function type */
typedef void (*fetch_func_t)(struct pt_regs *, void *, void *);
struct fetch_param {
fetch_func_t fn;
void *data; void *data;
}; };
static __kprobes unsigned long call_fetch(struct fetch_func *f, static __kprobes void call_fetch(struct fetch_param *fprm,
struct pt_regs *regs) struct pt_regs *regs, void *dest)
{ {
return f->func(regs, f->data); return fprm->fn(regs, fprm->data, dest);
} }
/* fetch handlers */ #define FETCH_FUNC_NAME(kind, type) fetch_##kind##_##type
static __kprobes unsigned long fetch_register(struct pt_regs *regs, /*
void *offset) * Define macro for basic types - we don't need to define s* types, because
{ * we have to care only about bitwidth at recording time.
return regs_get_register(regs, (unsigned int)((unsigned long)offset)); */
} #define DEFINE_BASIC_FETCH_FUNCS(kind) \
DEFINE_FETCH_##kind(u8) \
DEFINE_FETCH_##kind(u16) \
DEFINE_FETCH_##kind(u32) \
DEFINE_FETCH_##kind(u64)
static __kprobes unsigned long fetch_stack(struct pt_regs *regs, #define CHECK_BASIC_FETCH_FUNCS(kind, fn) \
void *num) ((FETCH_FUNC_NAME(kind, u8) == fn) || \
{ (FETCH_FUNC_NAME(kind, u16) == fn) || \
return regs_get_kernel_stack_nth(regs, (FETCH_FUNC_NAME(kind, u32) == fn) || \
(unsigned int)((unsigned long)num)); (FETCH_FUNC_NAME(kind, u64) == fn))
}
static __kprobes unsigned long fetch_memory(struct pt_regs *regs, void *addr) /* Data fetch function templates */
{ #define DEFINE_FETCH_reg(type) \
unsigned long retval; static __kprobes void FETCH_FUNC_NAME(reg, type)(struct pt_regs *regs, \
void *offset, void *dest) \
if (probe_kernel_address(addr, retval)) { \
return 0; *(type *)dest = (type)regs_get_register(regs, \
return retval; (unsigned int)((unsigned long)offset)); \
} }
DEFINE_BASIC_FETCH_FUNCS(reg)
static __kprobes unsigned long fetch_retvalue(struct pt_regs *regs, #define DEFINE_FETCH_stack(type) \
void *dummy) static __kprobes void FETCH_FUNC_NAME(stack, type)(struct pt_regs *regs,\
{ void *offset, void *dest) \
return regs_return_value(regs); { \
*(type *)dest = (type)regs_get_kernel_stack_nth(regs, \
(unsigned int)((unsigned long)offset)); \
} }
DEFINE_BASIC_FETCH_FUNCS(stack)
static __kprobes unsigned long fetch_stack_address(struct pt_regs *regs, #define DEFINE_FETCH_retval(type) \
void *dummy) static __kprobes void FETCH_FUNC_NAME(retval, type)(struct pt_regs *regs,\
{ void *dummy, void *dest) \
return kernel_stack_pointer(regs); { \
*(type *)dest = (type)regs_return_value(regs); \
} }
DEFINE_BASIC_FETCH_FUNCS(retval)
#define DEFINE_FETCH_memory(type) \
static __kprobes void FETCH_FUNC_NAME(memory, type)(struct pt_regs *regs,\
void *addr, void *dest) \
{ \
type retval; \
if (probe_kernel_address(addr, retval)) \
*(type *)dest = 0; \
else \
*(type *)dest = retval; \
}
DEFINE_BASIC_FETCH_FUNCS(memory)
/* Memory fetching by symbol */ /* Memory fetching by symbol */
struct symbol_cache { struct symbol_cache {
@ -145,51 +192,126 @@ static struct symbol_cache *alloc_symbol_cache(const char *sym, long offset)
return sc; return sc;
} }
static __kprobes unsigned long fetch_symbol(struct pt_regs *regs, void *data) #define DEFINE_FETCH_symbol(type) \
{ static __kprobes void FETCH_FUNC_NAME(symbol, type)(struct pt_regs *regs,\
struct symbol_cache *sc = data; void *data, void *dest) \
{ \
if (sc->addr) struct symbol_cache *sc = data; \
return fetch_memory(regs, (void *)sc->addr); if (sc->addr) \
else fetch_memory_##type(regs, (void *)sc->addr, dest); \
return 0; else \
*(type *)dest = 0; \
} }
DEFINE_BASIC_FETCH_FUNCS(symbol)
/* Special indirect memory access interface */ /* Dereference memory access function */
struct indirect_fetch_data { struct deref_fetch_param {
struct fetch_func orig; struct fetch_param orig;
long offset; long offset;
}; };
static __kprobes unsigned long fetch_indirect(struct pt_regs *regs, void *data) #define DEFINE_FETCH_deref(type) \
{ static __kprobes void FETCH_FUNC_NAME(deref, type)(struct pt_regs *regs,\
struct indirect_fetch_data *ind = data; void *data, void *dest) \
unsigned long addr; { \
struct deref_fetch_param *dprm = data; \
addr = call_fetch(&ind->orig, regs); unsigned long addr; \
if (addr) { call_fetch(&dprm->orig, regs, &addr); \
addr += ind->offset; if (addr) { \
return fetch_memory(regs, (void *)addr); addr += dprm->offset; \
} else fetch_memory_##type(regs, (void *)addr, dest); \
return 0; } else \
*(type *)dest = 0; \
} }
DEFINE_BASIC_FETCH_FUNCS(deref)
static __kprobes void free_indirect_fetch_data(struct indirect_fetch_data *data) static __kprobes void free_deref_fetch_param(struct deref_fetch_param *data)
{ {
if (data->orig.func == fetch_indirect) if (CHECK_BASIC_FETCH_FUNCS(deref, data->orig.fn))
free_indirect_fetch_data(data->orig.data); free_deref_fetch_param(data->orig.data);
else if (data->orig.func == fetch_symbol) else if (CHECK_BASIC_FETCH_FUNCS(symbol, data->orig.fn))
free_symbol_cache(data->orig.data); free_symbol_cache(data->orig.data);
kfree(data); kfree(data);
} }
/* Default (unsigned long) fetch type */
#define __DEFAULT_FETCH_TYPE(t) u##t
#define _DEFAULT_FETCH_TYPE(t) __DEFAULT_FETCH_TYPE(t)
#define DEFAULT_FETCH_TYPE _DEFAULT_FETCH_TYPE(BITS_PER_LONG)
#define DEFAULT_FETCH_TYPE_STR __stringify(DEFAULT_FETCH_TYPE)
#define ASSIGN_FETCH_FUNC(kind, type) \
.kind = FETCH_FUNC_NAME(kind, type)
#define ASSIGN_FETCH_TYPE(ptype, ftype, sign) \
{.name = #ptype, \
.size = sizeof(ftype), \
.is_signed = sign, \
.print = PRINT_TYPE_FUNC_NAME(ptype), \
.fmt = PRINT_TYPE_FMT_NAME(ptype), \
ASSIGN_FETCH_FUNC(reg, ftype), \
ASSIGN_FETCH_FUNC(stack, ftype), \
ASSIGN_FETCH_FUNC(retval, ftype), \
ASSIGN_FETCH_FUNC(memory, ftype), \
ASSIGN_FETCH_FUNC(symbol, ftype), \
ASSIGN_FETCH_FUNC(deref, ftype), \
}
/* Fetch type information table */
static const struct fetch_type {
const char *name; /* Name of type */
size_t size; /* Byte size of type */
int is_signed; /* Signed flag */
print_type_func_t print; /* Print functions */
const char *fmt; /* Fromat string */
/* Fetch functions */
fetch_func_t reg;
fetch_func_t stack;
fetch_func_t retval;
fetch_func_t memory;
fetch_func_t symbol;
fetch_func_t deref;
} fetch_type_table[] = {
ASSIGN_FETCH_TYPE(u8, u8, 0),
ASSIGN_FETCH_TYPE(u16, u16, 0),
ASSIGN_FETCH_TYPE(u32, u32, 0),
ASSIGN_FETCH_TYPE(u64, u64, 0),
ASSIGN_FETCH_TYPE(s8, u8, 1),
ASSIGN_FETCH_TYPE(s16, u16, 1),
ASSIGN_FETCH_TYPE(s32, u32, 1),
ASSIGN_FETCH_TYPE(s64, u64, 1),
};
static const struct fetch_type *find_fetch_type(const char *type)
{
int i;
if (!type)
type = DEFAULT_FETCH_TYPE_STR;
for (i = 0; i < ARRAY_SIZE(fetch_type_table); i++)
if (strcmp(type, fetch_type_table[i].name) == 0)
return &fetch_type_table[i];
return NULL;
}
/* Special function : only accept unsigned long */
static __kprobes void fetch_stack_address(struct pt_regs *regs,
void *dummy, void *dest)
{
*(unsigned long *)dest = kernel_stack_pointer(regs);
}
/** /**
* Kprobe event core functions * Kprobe event core functions
*/ */
struct probe_arg { struct probe_arg {
struct fetch_func fetch; struct fetch_param fetch;
const char *name; unsigned int offset; /* Offset from argument entry */
const char *name; /* Name of this argument */
const char *comm; /* Command of this argument */
const struct fetch_type *type; /* Type of this argument */
}; };
/* Flags for trace_probe */ /* Flags for trace_probe */
@ -204,6 +326,7 @@ struct trace_probe {
const char *symbol; /* symbol name */ const char *symbol; /* symbol name */
struct ftrace_event_call call; struct ftrace_event_call call;
struct trace_event event; struct trace_event event;
ssize_t size; /* trace entry size */
unsigned int nr_args; unsigned int nr_args;
struct probe_arg args[]; struct probe_arg args[];
}; };
@ -212,6 +335,7 @@ struct trace_probe {
(offsetof(struct trace_probe, args) + \ (offsetof(struct trace_probe, args) + \
(sizeof(struct probe_arg) * (n))) (sizeof(struct probe_arg) * (n)))
static __kprobes int probe_is_return(struct trace_probe *tp) static __kprobes int probe_is_return(struct trace_probe *tp)
{ {
return tp->rp.handler != NULL; return tp->rp.handler != NULL;
@ -222,49 +346,6 @@ static __kprobes const char *probe_symbol(struct trace_probe *tp)
return tp->symbol ? tp->symbol : "unknown"; return tp->symbol ? tp->symbol : "unknown";
} }
static int probe_arg_string(char *buf, size_t n, struct fetch_func *ff)
{
int ret = -EINVAL;
if (ff->func == fetch_register) {
const char *name;
name = regs_query_register_name((unsigned int)((long)ff->data));
ret = snprintf(buf, n, "%%%s", name);
} else if (ff->func == fetch_stack)
ret = snprintf(buf, n, "$stack%lu", (unsigned long)ff->data);
else if (ff->func == fetch_memory)
ret = snprintf(buf, n, "@0x%p", ff->data);
else if (ff->func == fetch_symbol) {
struct symbol_cache *sc = ff->data;
if (sc->offset)
ret = snprintf(buf, n, "@%s%+ld", sc->symbol,
sc->offset);
else
ret = snprintf(buf, n, "@%s", sc->symbol);
} else if (ff->func == fetch_retvalue)
ret = snprintf(buf, n, "$retval");
else if (ff->func == fetch_stack_address)
ret = snprintf(buf, n, "$stack");
else if (ff->func == fetch_indirect) {
struct indirect_fetch_data *id = ff->data;
size_t l = 0;
ret = snprintf(buf, n, "%+ld(", id->offset);
if (ret >= n)
goto end;
l += ret;
ret = probe_arg_string(buf + l, n - l, &id->orig);
if (ret < 0)
goto end;
l += ret;
ret = snprintf(buf + l, n - l, ")");
ret += l;
}
end:
if (ret >= n)
return -ENOSPC;
return ret;
}
static int register_probe_event(struct trace_probe *tp); static int register_probe_event(struct trace_probe *tp);
static void unregister_probe_event(struct trace_probe *tp); static void unregister_probe_event(struct trace_probe *tp);
@ -347,11 +428,12 @@ error:
static void free_probe_arg(struct probe_arg *arg) static void free_probe_arg(struct probe_arg *arg)
{ {
if (arg->fetch.func == fetch_symbol) if (CHECK_BASIC_FETCH_FUNCS(deref, arg->fetch.fn))
free_deref_fetch_param(arg->fetch.data);
else if (CHECK_BASIC_FETCH_FUNCS(symbol, arg->fetch.fn))
free_symbol_cache(arg->fetch.data); free_symbol_cache(arg->fetch.data);
else if (arg->fetch.func == fetch_indirect)
free_indirect_fetch_data(arg->fetch.data);
kfree(arg->name); kfree(arg->name);
kfree(arg->comm);
} }
static void free_trace_probe(struct trace_probe *tp) static void free_trace_probe(struct trace_probe *tp)
@ -457,28 +539,30 @@ static int split_symbol_offset(char *symbol, unsigned long *offset)
#define PARAM_MAX_ARGS 16 #define PARAM_MAX_ARGS 16
#define PARAM_MAX_STACK (THREAD_SIZE / sizeof(unsigned long)) #define PARAM_MAX_STACK (THREAD_SIZE / sizeof(unsigned long))
static int parse_probe_vars(char *arg, struct fetch_func *ff, int is_return) static int parse_probe_vars(char *arg, const struct fetch_type *t,
struct fetch_param *f, int is_return)
{ {
int ret = 0; int ret = 0;
unsigned long param; unsigned long param;
if (strcmp(arg, "retval") == 0) { if (strcmp(arg, "retval") == 0) {
if (is_return) { if (is_return)
ff->func = fetch_retvalue; f->fn = t->retval;
ff->data = NULL; else
} else
ret = -EINVAL; ret = -EINVAL;
} else if (strncmp(arg, "stack", 5) == 0) { } else if (strncmp(arg, "stack", 5) == 0) {
if (arg[5] == '\0') { if (arg[5] == '\0') {
ff->func = fetch_stack_address; if (strcmp(t->name, DEFAULT_FETCH_TYPE_STR) == 0)
ff->data = NULL; f->fn = fetch_stack_address;
else
ret = -EINVAL;
} else if (isdigit(arg[5])) { } else if (isdigit(arg[5])) {
ret = strict_strtoul(arg + 5, 10, &param); ret = strict_strtoul(arg + 5, 10, &param);
if (ret || param > PARAM_MAX_STACK) if (ret || param > PARAM_MAX_STACK)
ret = -EINVAL; ret = -EINVAL;
else { else {
ff->func = fetch_stack; f->fn = t->stack;
ff->data = (void *)param; f->data = (void *)param;
} }
} else } else
ret = -EINVAL; ret = -EINVAL;
@ -488,7 +572,8 @@ static int parse_probe_vars(char *arg, struct fetch_func *ff, int is_return)
} }
/* Recursive argument parser */ /* Recursive argument parser */
static int __parse_probe_arg(char *arg, struct fetch_func *ff, int is_return) static int __parse_probe_arg(char *arg, const struct fetch_type *t,
struct fetch_param *f, int is_return)
{ {
int ret = 0; int ret = 0;
unsigned long param; unsigned long param;
@ -497,13 +582,13 @@ static int __parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
switch (arg[0]) { switch (arg[0]) {
case '$': case '$':
ret = parse_probe_vars(arg + 1, ff, is_return); ret = parse_probe_vars(arg + 1, t, f, is_return);
break; break;
case '%': /* named register */ case '%': /* named register */
ret = regs_query_register_offset(arg + 1); ret = regs_query_register_offset(arg + 1);
if (ret >= 0) { if (ret >= 0) {
ff->func = fetch_register; f->fn = t->reg;
ff->data = (void *)(unsigned long)ret; f->data = (void *)(unsigned long)ret;
ret = 0; ret = 0;
} }
break; break;
@ -512,26 +597,22 @@ static int __parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
ret = strict_strtoul(arg + 1, 0, &param); ret = strict_strtoul(arg + 1, 0, &param);
if (ret) if (ret)
break; break;
ff->func = fetch_memory; f->fn = t->memory;
ff->data = (void *)param; f->data = (void *)param;
} else { } else {
ret = split_symbol_offset(arg + 1, &offset); ret = split_symbol_offset(arg + 1, &offset);
if (ret) if (ret)
break; break;
ff->data = alloc_symbol_cache(arg + 1, offset); f->data = alloc_symbol_cache(arg + 1, offset);
if (ff->data) if (f->data)
ff->func = fetch_symbol; f->fn = t->symbol;
else
ret = -EINVAL;
} }
break; break;
case '+': /* indirect memory */ case '+': /* deref memory */
case '-': case '-':
tmp = strchr(arg, '('); tmp = strchr(arg, '(');
if (!tmp) { if (!tmp)
ret = -EINVAL;
break; break;
}
*tmp = '\0'; *tmp = '\0';
ret = strict_strtol(arg + 1, 0, &offset); ret = strict_strtol(arg + 1, 0, &offset);
if (ret) if (ret)
@ -541,38 +622,58 @@ static int __parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
arg = tmp + 1; arg = tmp + 1;
tmp = strrchr(arg, ')'); tmp = strrchr(arg, ')');
if (tmp) { if (tmp) {
struct indirect_fetch_data *id; struct deref_fetch_param *dprm;
const struct fetch_type *t2 = find_fetch_type(NULL);
*tmp = '\0'; *tmp = '\0';
id = kzalloc(sizeof(struct indirect_fetch_data), dprm = kzalloc(sizeof(struct deref_fetch_param),
GFP_KERNEL); GFP_KERNEL);
if (!id) if (!dprm)
return -ENOMEM; return -ENOMEM;
id->offset = offset; dprm->offset = offset;
ret = __parse_probe_arg(arg, &id->orig, is_return); ret = __parse_probe_arg(arg, t2, &dprm->orig,
is_return);
if (ret) if (ret)
kfree(id); kfree(dprm);
else { else {
ff->func = fetch_indirect; f->fn = t->deref;
ff->data = (void *)id; f->data = (void *)dprm;
}
} }
} else
ret = -EINVAL;
break; break;
default:
/* TODO: support custom handler */
ret = -EINVAL;
} }
if (!ret && !f->fn)
ret = -EINVAL;
return ret; return ret;
} }
/* String length checking wrapper */ /* String length checking wrapper */
static int parse_probe_arg(char *arg, struct fetch_func *ff, int is_return) static int parse_probe_arg(char *arg, struct trace_probe *tp,
struct probe_arg *parg, int is_return)
{ {
const char *t;
if (strlen(arg) > MAX_ARGSTR_LEN) { if (strlen(arg) > MAX_ARGSTR_LEN) {
pr_info("Argument is too long.: %s\n", arg); pr_info("Argument is too long.: %s\n", arg);
return -ENOSPC; return -ENOSPC;
} }
return __parse_probe_arg(arg, ff, is_return); parg->comm = kstrdup(arg, GFP_KERNEL);
if (!parg->comm) {
pr_info("Failed to allocate memory for command '%s'.\n", arg);
return -ENOMEM;
}
t = strchr(parg->comm, ':');
if (t) {
arg[t - parg->comm] = '\0';
t++;
}
parg->type = find_fetch_type(t);
if (!parg->type) {
pr_info("Unsupported type: %s\n", t);
return -EINVAL;
}
parg->offset = tp->size;
tp->size += parg->type->size;
return __parse_probe_arg(arg, parg->type, &parg->fetch, is_return);
} }
/* Return 1 if name is reserved or already used by another argument */ /* Return 1 if name is reserved or already used by another argument */
@ -602,15 +703,18 @@ static int create_trace_probe(int argc, char **argv)
* @ADDR : fetch memory at ADDR (ADDR should be in kernel) * @ADDR : fetch memory at ADDR (ADDR should be in kernel)
* @SYM[+|-offs] : fetch memory at SYM +|- offs (SYM is a data symbol) * @SYM[+|-offs] : fetch memory at SYM +|- offs (SYM is a data symbol)
* %REG : fetch register REG * %REG : fetch register REG
* Indirect memory fetch: * Dereferencing memory fetch:
* +|-offs(ARG) : fetch memory at ARG +|- offs address. * +|-offs(ARG) : fetch memory at ARG +|- offs address.
* Alias name of args: * Alias name of args:
* NAME=FETCHARG : set NAME as alias of FETCHARG. * NAME=FETCHARG : set NAME as alias of FETCHARG.
* Type of args:
* FETCHARG:TYPE : use TYPE instead of unsigned long.
*/ */
struct trace_probe *tp; struct trace_probe *tp;
int i, ret = 0; int i, ret = 0;
int is_return = 0, is_delete = 0; int is_return = 0, is_delete = 0;
char *symbol = NULL, *event = NULL, *arg = NULL, *group = NULL; char *symbol = NULL, *event = NULL, *group = NULL;
char *arg, *tmp;
unsigned long offset = 0; unsigned long offset = 0;
void *addr = NULL; void *addr = NULL;
char buf[MAX_EVENT_NAME_LEN]; char buf[MAX_EVENT_NAME_LEN];
@ -723,13 +827,6 @@ static int create_trace_probe(int argc, char **argv)
else else
arg = argv[i]; arg = argv[i];
if (conflict_field_name(argv[i], tp->args, i)) {
pr_info("Argument%d name '%s' conflicts with "
"another field.\n", i, argv[i]);
ret = -EINVAL;
goto error;
}
tp->args[i].name = kstrdup(argv[i], GFP_KERNEL); tp->args[i].name = kstrdup(argv[i], GFP_KERNEL);
if (!tp->args[i].name) { if (!tp->args[i].name) {
pr_info("Failed to allocate argument%d name '%s'.\n", pr_info("Failed to allocate argument%d name '%s'.\n",
@ -737,9 +834,19 @@ static int create_trace_probe(int argc, char **argv)
ret = -ENOMEM; ret = -ENOMEM;
goto error; goto error;
} }
tmp = strchr(tp->args[i].name, ':');
if (tmp)
*tmp = '_'; /* convert : to _ */
if (conflict_field_name(tp->args[i].name, tp->args, i)) {
pr_info("Argument%d name '%s' conflicts with "
"another field.\n", i, argv[i]);
ret = -EINVAL;
goto error;
}
/* Parse fetch argument */ /* Parse fetch argument */
ret = parse_probe_arg(arg, &tp->args[i].fetch, is_return); ret = parse_probe_arg(arg, tp, &tp->args[i], is_return);
if (ret) { if (ret) {
pr_info("Parse error at argument%d. (%d)\n", i, ret); pr_info("Parse error at argument%d. (%d)\n", i, ret);
kfree(tp->args[i].name); kfree(tp->args[i].name);
@ -794,8 +901,7 @@ static void probes_seq_stop(struct seq_file *m, void *v)
static int probes_seq_show(struct seq_file *m, void *v) static int probes_seq_show(struct seq_file *m, void *v)
{ {
struct trace_probe *tp = v; struct trace_probe *tp = v;
int i, ret; int i;
char buf[MAX_ARGSTR_LEN + 1];
seq_printf(m, "%c", probe_is_return(tp) ? 'r' : 'p'); seq_printf(m, "%c", probe_is_return(tp) ? 'r' : 'p');
seq_printf(m, ":%s/%s", tp->call.system, tp->call.name); seq_printf(m, ":%s/%s", tp->call.system, tp->call.name);
@ -807,15 +913,10 @@ static int probes_seq_show(struct seq_file *m, void *v)
else else
seq_printf(m, " %s", probe_symbol(tp)); seq_printf(m, " %s", probe_symbol(tp));
for (i = 0; i < tp->nr_args; i++) { for (i = 0; i < tp->nr_args; i++)
ret = probe_arg_string(buf, MAX_ARGSTR_LEN, &tp->args[i].fetch); seq_printf(m, " %s=%s", tp->args[i].name, tp->args[i].comm);
if (ret < 0) {
pr_warning("Argument%d decoding error(%d).\n", i, ret);
return ret;
}
seq_printf(m, " %s=%s", tp->args[i].name, buf);
}
seq_printf(m, "\n"); seq_printf(m, "\n");
return 0; return 0;
} }
@ -945,9 +1046,10 @@ static const struct file_operations kprobe_profile_ops = {
static __kprobes void kprobe_trace_func(struct kprobe *kp, struct pt_regs *regs) static __kprobes void kprobe_trace_func(struct kprobe *kp, struct pt_regs *regs)
{ {
struct trace_probe *tp = container_of(kp, struct trace_probe, rp.kp); struct trace_probe *tp = container_of(kp, struct trace_probe, rp.kp);
struct kprobe_trace_entry *entry; struct kprobe_trace_entry_head *entry;
struct ring_buffer_event *event; struct ring_buffer_event *event;
struct ring_buffer *buffer; struct ring_buffer *buffer;
u8 *data;
int size, i, pc; int size, i, pc;
unsigned long irq_flags; unsigned long irq_flags;
struct ftrace_event_call *call = &tp->call; struct ftrace_event_call *call = &tp->call;
@ -957,7 +1059,7 @@ static __kprobes void kprobe_trace_func(struct kprobe *kp, struct pt_regs *regs)
local_save_flags(irq_flags); local_save_flags(irq_flags);
pc = preempt_count(); pc = preempt_count();
size = SIZEOF_KPROBE_TRACE_ENTRY(tp->nr_args); size = sizeof(*entry) + tp->size;
event = trace_current_buffer_lock_reserve(&buffer, call->id, size, event = trace_current_buffer_lock_reserve(&buffer, call->id, size,
irq_flags, pc); irq_flags, pc);
@ -965,10 +1067,10 @@ static __kprobes void kprobe_trace_func(struct kprobe *kp, struct pt_regs *regs)
return; return;
entry = ring_buffer_event_data(event); entry = ring_buffer_event_data(event);
entry->nargs = tp->nr_args;
entry->ip = (unsigned long)kp->addr; entry->ip = (unsigned long)kp->addr;
data = (u8 *)&entry[1];
for (i = 0; i < tp->nr_args; i++) for (i = 0; i < tp->nr_args; i++)
entry->args[i] = call_fetch(&tp->args[i].fetch, regs); call_fetch(&tp->args[i].fetch, regs, data + tp->args[i].offset);
if (!filter_current_check_discard(buffer, call, entry, event)) if (!filter_current_check_discard(buffer, call, entry, event))
trace_nowake_buffer_unlock_commit(buffer, event, irq_flags, pc); trace_nowake_buffer_unlock_commit(buffer, event, irq_flags, pc);
@ -979,9 +1081,10 @@ static __kprobes void kretprobe_trace_func(struct kretprobe_instance *ri,
struct pt_regs *regs) struct pt_regs *regs)
{ {
struct trace_probe *tp = container_of(ri->rp, struct trace_probe, rp); struct trace_probe *tp = container_of(ri->rp, struct trace_probe, rp);
struct kretprobe_trace_entry *entry; struct kretprobe_trace_entry_head *entry;
struct ring_buffer_event *event; struct ring_buffer_event *event;
struct ring_buffer *buffer; struct ring_buffer *buffer;
u8 *data;
int size, i, pc; int size, i, pc;
unsigned long irq_flags; unsigned long irq_flags;
struct ftrace_event_call *call = &tp->call; struct ftrace_event_call *call = &tp->call;
@ -989,7 +1092,7 @@ static __kprobes void kretprobe_trace_func(struct kretprobe_instance *ri,
local_save_flags(irq_flags); local_save_flags(irq_flags);
pc = preempt_count(); pc = preempt_count();
size = SIZEOF_KRETPROBE_TRACE_ENTRY(tp->nr_args); size = sizeof(*entry) + tp->size;
event = trace_current_buffer_lock_reserve(&buffer, call->id, size, event = trace_current_buffer_lock_reserve(&buffer, call->id, size,
irq_flags, pc); irq_flags, pc);
@ -997,11 +1100,11 @@ static __kprobes void kretprobe_trace_func(struct kretprobe_instance *ri,
return; return;
entry = ring_buffer_event_data(event); entry = ring_buffer_event_data(event);
entry->nargs = tp->nr_args;
entry->func = (unsigned long)tp->rp.kp.addr; entry->func = (unsigned long)tp->rp.kp.addr;
entry->ret_ip = (unsigned long)ri->ret_addr; entry->ret_ip = (unsigned long)ri->ret_addr;
data = (u8 *)&entry[1];
for (i = 0; i < tp->nr_args; i++) for (i = 0; i < tp->nr_args; i++)
entry->args[i] = call_fetch(&tp->args[i].fetch, regs); call_fetch(&tp->args[i].fetch, regs, data + tp->args[i].offset);
if (!filter_current_check_discard(buffer, call, entry, event)) if (!filter_current_check_discard(buffer, call, entry, event))
trace_nowake_buffer_unlock_commit(buffer, event, irq_flags, pc); trace_nowake_buffer_unlock_commit(buffer, event, irq_flags, pc);
@ -1011,13 +1114,14 @@ static __kprobes void kretprobe_trace_func(struct kretprobe_instance *ri,
enum print_line_t enum print_line_t
print_kprobe_event(struct trace_iterator *iter, int flags) print_kprobe_event(struct trace_iterator *iter, int flags)
{ {
struct kprobe_trace_entry *field; struct kprobe_trace_entry_head *field;
struct trace_seq *s = &iter->seq; struct trace_seq *s = &iter->seq;
struct trace_event *event; struct trace_event *event;
struct trace_probe *tp; struct trace_probe *tp;
u8 *data;
int i; int i;
field = (struct kprobe_trace_entry *)iter->ent; field = (struct kprobe_trace_entry_head *)iter->ent;
event = ftrace_find_event(field->ent.type); event = ftrace_find_event(field->ent.type);
tp = container_of(event, struct trace_probe, event); tp = container_of(event, struct trace_probe, event);
@ -1030,9 +1134,10 @@ print_kprobe_event(struct trace_iterator *iter, int flags)
if (!trace_seq_puts(s, ")")) if (!trace_seq_puts(s, ")"))
goto partial; goto partial;
for (i = 0; i < field->nargs; i++) data = (u8 *)&field[1];
if (!trace_seq_printf(s, " %s=%lx", for (i = 0; i < tp->nr_args; i++)
tp->args[i].name, field->args[i])) if (!tp->args[i].type->print(s, tp->args[i].name,
data + tp->args[i].offset))
goto partial; goto partial;
if (!trace_seq_puts(s, "\n")) if (!trace_seq_puts(s, "\n"))
@ -1046,13 +1151,14 @@ partial:
enum print_line_t enum print_line_t
print_kretprobe_event(struct trace_iterator *iter, int flags) print_kretprobe_event(struct trace_iterator *iter, int flags)
{ {
struct kretprobe_trace_entry *field; struct kretprobe_trace_entry_head *field;
struct trace_seq *s = &iter->seq; struct trace_seq *s = &iter->seq;
struct trace_event *event; struct trace_event *event;
struct trace_probe *tp; struct trace_probe *tp;
u8 *data;
int i; int i;
field = (struct kretprobe_trace_entry *)iter->ent; field = (struct kretprobe_trace_entry_head *)iter->ent;
event = ftrace_find_event(field->ent.type); event = ftrace_find_event(field->ent.type);
tp = container_of(event, struct trace_probe, event); tp = container_of(event, struct trace_probe, event);
@ -1071,9 +1177,10 @@ print_kretprobe_event(struct trace_iterator *iter, int flags)
if (!trace_seq_puts(s, ")")) if (!trace_seq_puts(s, ")"))
goto partial; goto partial;
for (i = 0; i < field->nargs; i++) data = (u8 *)&field[1];
if (!trace_seq_printf(s, " %s=%lx", for (i = 0; i < tp->nr_args; i++)
tp->args[i].name, field->args[i])) if (!tp->args[i].type->print(s, tp->args[i].name,
data + tp->args[i].offset))
goto partial; goto partial;
if (!trace_seq_puts(s, "\n")) if (!trace_seq_puts(s, "\n"))
@ -1129,29 +1236,43 @@ static int probe_event_raw_init(struct ftrace_event_call *event_call)
static int kprobe_event_define_fields(struct ftrace_event_call *event_call) static int kprobe_event_define_fields(struct ftrace_event_call *event_call)
{ {
int ret, i; int ret, i;
struct kprobe_trace_entry field; struct kprobe_trace_entry_head field;
struct trace_probe *tp = (struct trace_probe *)event_call->data; struct trace_probe *tp = (struct trace_probe *)event_call->data;
DEFINE_FIELD(unsigned long, ip, FIELD_STRING_IP, 0); DEFINE_FIELD(unsigned long, ip, FIELD_STRING_IP, 0);
DEFINE_FIELD(int, nargs, FIELD_STRING_NARGS, 1);
/* Set argument names as fields */ /* Set argument names as fields */
for (i = 0; i < tp->nr_args; i++) for (i = 0; i < tp->nr_args; i++) {
DEFINE_FIELD(unsigned long, args[i], tp->args[i].name, 0); ret = trace_define_field(event_call, tp->args[i].type->name,
tp->args[i].name,
sizeof(field) + tp->args[i].offset,
tp->args[i].type->size,
tp->args[i].type->is_signed,
FILTER_OTHER);
if (ret)
return ret;
}
return 0; return 0;
} }
static int kretprobe_event_define_fields(struct ftrace_event_call *event_call) static int kretprobe_event_define_fields(struct ftrace_event_call *event_call)
{ {
int ret, i; int ret, i;
struct kretprobe_trace_entry field; struct kretprobe_trace_entry_head field;
struct trace_probe *tp = (struct trace_probe *)event_call->data; struct trace_probe *tp = (struct trace_probe *)event_call->data;
DEFINE_FIELD(unsigned long, func, FIELD_STRING_FUNC, 0); DEFINE_FIELD(unsigned long, func, FIELD_STRING_FUNC, 0);
DEFINE_FIELD(unsigned long, ret_ip, FIELD_STRING_RETIP, 0); DEFINE_FIELD(unsigned long, ret_ip, FIELD_STRING_RETIP, 0);
DEFINE_FIELD(int, nargs, FIELD_STRING_NARGS, 1);
/* Set argument names as fields */ /* Set argument names as fields */
for (i = 0; i < tp->nr_args; i++) for (i = 0; i < tp->nr_args; i++) {
DEFINE_FIELD(unsigned long, args[i], tp->args[i].name, 0); ret = trace_define_field(event_call, tp->args[i].type->name,
tp->args[i].name,
sizeof(field) + tp->args[i].offset,
tp->args[i].type->size,
tp->args[i].type->is_signed,
FILTER_OTHER);
if (ret)
return ret;
}
return 0; return 0;
} }
@ -1176,8 +1297,8 @@ static int __set_print_fmt(struct trace_probe *tp, char *buf, int len)
pos += snprintf(buf + pos, LEN_OR_ZERO, "\"%s", fmt); pos += snprintf(buf + pos, LEN_OR_ZERO, "\"%s", fmt);
for (i = 0; i < tp->nr_args; i++) { for (i = 0; i < tp->nr_args; i++) {
pos += snprintf(buf + pos, LEN_OR_ZERO, " %s=%%lx", pos += snprintf(buf + pos, LEN_OR_ZERO, " %s=%s",
tp->args[i].name); tp->args[i].name, tp->args[i].type->fmt);
} }
pos += snprintf(buf + pos, LEN_OR_ZERO, "\", %s", arg); pos += snprintf(buf + pos, LEN_OR_ZERO, "\", %s", arg);
@ -1219,12 +1340,13 @@ static __kprobes void kprobe_perf_func(struct kprobe *kp,
{ {
struct trace_probe *tp = container_of(kp, struct trace_probe, rp.kp); struct trace_probe *tp = container_of(kp, struct trace_probe, rp.kp);
struct ftrace_event_call *call = &tp->call; struct ftrace_event_call *call = &tp->call;
struct kprobe_trace_entry *entry; struct kprobe_trace_entry_head *entry;
u8 *data;
int size, __size, i; int size, __size, i;
unsigned long irq_flags; unsigned long irq_flags;
int rctx; int rctx;
__size = SIZEOF_KPROBE_TRACE_ENTRY(tp->nr_args); __size = sizeof(*entry) + tp->size;
size = ALIGN(__size + sizeof(u32), sizeof(u64)); size = ALIGN(__size + sizeof(u32), sizeof(u64));
size -= sizeof(u32); size -= sizeof(u32);
if (WARN_ONCE(size > PERF_MAX_TRACE_SIZE, if (WARN_ONCE(size > PERF_MAX_TRACE_SIZE,
@ -1235,10 +1357,10 @@ static __kprobes void kprobe_perf_func(struct kprobe *kp,
if (!entry) if (!entry)
return; return;
entry->nargs = tp->nr_args;
entry->ip = (unsigned long)kp->addr; entry->ip = (unsigned long)kp->addr;
data = (u8 *)&entry[1];
for (i = 0; i < tp->nr_args; i++) for (i = 0; i < tp->nr_args; i++)
entry->args[i] = call_fetch(&tp->args[i].fetch, regs); call_fetch(&tp->args[i].fetch, regs, data + tp->args[i].offset);
perf_trace_buf_submit(entry, size, rctx, entry->ip, 1, irq_flags, regs); perf_trace_buf_submit(entry, size, rctx, entry->ip, 1, irq_flags, regs);
} }
@ -1249,12 +1371,13 @@ static __kprobes void kretprobe_perf_func(struct kretprobe_instance *ri,
{ {
struct trace_probe *tp = container_of(ri->rp, struct trace_probe, rp); struct trace_probe *tp = container_of(ri->rp, struct trace_probe, rp);
struct ftrace_event_call *call = &tp->call; struct ftrace_event_call *call = &tp->call;
struct kretprobe_trace_entry *entry; struct kretprobe_trace_entry_head *entry;
u8 *data;
int size, __size, i; int size, __size, i;
unsigned long irq_flags; unsigned long irq_flags;
int rctx; int rctx;
__size = SIZEOF_KRETPROBE_TRACE_ENTRY(tp->nr_args); __size = sizeof(*entry) + tp->size;
size = ALIGN(__size + sizeof(u32), sizeof(u64)); size = ALIGN(__size + sizeof(u32), sizeof(u64));
size -= sizeof(u32); size -= sizeof(u32);
if (WARN_ONCE(size > PERF_MAX_TRACE_SIZE, if (WARN_ONCE(size > PERF_MAX_TRACE_SIZE,
@ -1265,11 +1388,11 @@ static __kprobes void kretprobe_perf_func(struct kretprobe_instance *ri,
if (!entry) if (!entry)
return; return;
entry->nargs = tp->nr_args;
entry->func = (unsigned long)tp->rp.kp.addr; entry->func = (unsigned long)tp->rp.kp.addr;
entry->ret_ip = (unsigned long)ri->ret_addr; entry->ret_ip = (unsigned long)ri->ret_addr;
data = (u8 *)&entry[1];
for (i = 0; i < tp->nr_args; i++) for (i = 0; i < tp->nr_args; i++)
entry->args[i] = call_fetch(&tp->args[i].fetch, regs); call_fetch(&tp->args[i].fetch, regs, data + tp->args[i].offset);
perf_trace_buf_submit(entry, size, rctx, entry->ret_ip, 1, perf_trace_buf_submit(entry, size, rctx, entry->ret_ip, 1,
irq_flags, regs); irq_flags, regs);

22
kernel/trace/trace_ksym.c
View file

@ -34,12 +34,6 @@
#include <asm/atomic.h> #include <asm/atomic.h>
/*
* For now, let us restrict the no. of symbols traced simultaneously to number
* of available hardware breakpoint registers.
*/
#define KSYM_TRACER_MAX HBP_NUM
#define KSYM_TRACER_OP_LEN 3 /* rw- */ #define KSYM_TRACER_OP_LEN 3 /* rw- */
struct trace_ksym { struct trace_ksym {
@ -53,7 +47,6 @@ struct trace_ksym {
static struct trace_array *ksym_trace_array; static struct trace_array *ksym_trace_array;
static unsigned int ksym_filter_entry_count;
static unsigned int ksym_tracing_enabled; static unsigned int ksym_tracing_enabled;
static HLIST_HEAD(ksym_filter_head); static HLIST_HEAD(ksym_filter_head);
@ -181,13 +174,6 @@ int process_new_ksym_entry(char *ksymname, int op, unsigned long addr)
struct trace_ksym *entry; struct trace_ksym *entry;
int ret = -ENOMEM; int ret = -ENOMEM;
if (ksym_filter_entry_count >= KSYM_TRACER_MAX) {
printk(KERN_ERR "ksym_tracer: Maximum limit:(%d) reached. No"
" new requests for tracing can be accepted now.\n",
KSYM_TRACER_MAX);
return -ENOSPC;
}
entry = kzalloc(sizeof(struct trace_ksym), GFP_KERNEL); entry = kzalloc(sizeof(struct trace_ksym), GFP_KERNEL);
if (!entry) if (!entry)
return -ENOMEM; return -ENOMEM;
@ -203,13 +189,17 @@ int process_new_ksym_entry(char *ksymname, int op, unsigned long addr)
if (IS_ERR(entry->ksym_hbp)) { if (IS_ERR(entry->ksym_hbp)) {
ret = PTR_ERR(entry->ksym_hbp); ret = PTR_ERR(entry->ksym_hbp);
if (ret == -ENOSPC) {
printk(KERN_ERR "ksym_tracer: Maximum limit reached."
" No new requests for tracing can be accepted now.\n");
} else {
printk(KERN_INFO "ksym_tracer request failed. Try again" printk(KERN_INFO "ksym_tracer request failed. Try again"
" later!!\n"); " later!!\n");
}
goto err; goto err;
} }
hlist_add_head_rcu(&(entry->ksym_hlist), &ksym_filter_head); hlist_add_head_rcu(&(entry->ksym_hlist), &ksym_filter_head);
ksym_filter_entry_count++;
return 0; return 0;
@ -265,7 +255,6 @@ static void __ksym_trace_reset(void)
hlist_for_each_entry_safe(entry, node, node1, &ksym_filter_head, hlist_for_each_entry_safe(entry, node, node1, &ksym_filter_head,
ksym_hlist) { ksym_hlist) {
unregister_wide_hw_breakpoint(entry->ksym_hbp); unregister_wide_hw_breakpoint(entry->ksym_hbp);
ksym_filter_entry_count--;
hlist_del_rcu(&(entry->ksym_hlist)); hlist_del_rcu(&(entry->ksym_hlist));
synchronize_rcu(); synchronize_rcu();
kfree(entry); kfree(entry);
@ -338,7 +327,6 @@ static ssize_t ksym_trace_filter_write(struct file *file,
goto out_unlock; goto out_unlock;
} }
/* Error or "symbol:---" case: drop it */ /* Error or "symbol:---" case: drop it */
ksym_filter_entry_count--;
hlist_del_rcu(&(entry->ksym_hlist)); hlist_del_rcu(&(entry->ksym_hlist));
synchronize_rcu(); synchronize_rcu();
kfree(entry); kfree(entry);

57
kernel/trace/trace_selftest.c
View file

@ -17,7 +17,6 @@ static inline int trace_valid_entry(struct trace_entry *entry)
case TRACE_BRANCH: case TRACE_BRANCH:
case TRACE_GRAPH_ENT: case TRACE_GRAPH_ENT:
case TRACE_GRAPH_RET: case TRACE_GRAPH_RET:
case TRACE_HW_BRANCHES:
case TRACE_KSYM: case TRACE_KSYM:
return 1; return 1;
} }
@ -755,62 +754,6 @@ trace_selftest_startup_branch(struct tracer *trace, struct trace_array *tr)
} }
#endif /* CONFIG_BRANCH_TRACER */ #endif /* CONFIG_BRANCH_TRACER */
#ifdef CONFIG_HW_BRANCH_TRACER
int
trace_selftest_startup_hw_branches(struct tracer *trace,
struct trace_array *tr)
{
struct trace_iterator *iter;
struct tracer tracer;
unsigned long count;
int ret;
if (!trace->open) {
printk(KERN_CONT "missing open function...");
return -1;
}
ret = tracer_init(trace, tr);
if (ret) {
warn_failed_init_tracer(trace, ret);
return ret;
}
/*
* The hw-branch tracer needs to collect the trace from the various
* cpu trace buffers - before tracing is stopped.
*/
iter = kzalloc(sizeof(*iter), GFP_KERNEL);
if (!iter)
return -ENOMEM;
memcpy(&tracer, trace, sizeof(tracer));
iter->trace = &tracer;
iter->tr = tr;
iter->pos = -1;
mutex_init(&iter->mutex);
trace->open(iter);
mutex_destroy(&iter->mutex);
kfree(iter);
tracing_stop();
ret = trace_test_buffer(tr, &count);
trace->reset(tr);
tracing_start();
if (!ret && !count) {
printk(KERN_CONT "no entries found..");
ret = -1;
}
return ret;
}
#endif /* CONFIG_HW_BRANCH_TRACER */
#ifdef CONFIG_KSYM_TRACER #ifdef CONFIG_KSYM_TRACER
static int ksym_selftest_dummy; static int ksym_selftest_dummy;

41
mm/mlock.c
View file

@ -607,44 +607,3 @@ void user_shm_unlock(size_t size, struct user_struct *user)
spin_unlock(&shmlock_user_lock); spin_unlock(&shmlock_user_lock);
free_uid(user); free_uid(user);
} }
int account_locked_memory(struct mm_struct *mm, struct rlimit *rlim,
size_t size)
{
unsigned long lim, vm, pgsz;
int error = -ENOMEM;
pgsz = PAGE_ALIGN(size) >> PAGE_SHIFT;
down_write(&mm->mmap_sem);
lim = ACCESS_ONCE(rlim[RLIMIT_AS].rlim_cur) >> PAGE_SHIFT;
vm = mm->total_vm + pgsz;
if (lim < vm)
goto out;
lim = ACCESS_ONCE(rlim[RLIMIT_MEMLOCK].rlim_cur) >> PAGE_SHIFT;
vm = mm->locked_vm + pgsz;
if (lim < vm)
goto out;
mm->total_vm += pgsz;
mm->locked_vm += pgsz;
error = 0;
out:
up_write(&mm->mmap_sem);
return error;
}
void refund_locked_memory(struct mm_struct *mm, size_t size)
{
unsigned long pgsz = PAGE_ALIGN(size) >> PAGE_SHIFT;
down_write(&mm->mmap_sem);
mm->total_vm -= pgsz;
mm->locked_vm -= pgsz;
up_write(&mm->mmap_sem);
}

2
tools/perf/Documentation/perf-annotate.txt
View file

@ -1,5 +1,5 @@
perf-annotate(1) perf-annotate(1)
============== ================
NAME NAME
---- ----

6
tools/perf/Documentation/perf-bench.txt
View file

@ -1,5 +1,5 @@
perf-bench(1) perf-bench(1)
============ =============
NAME NAME
---- ----
@ -19,12 +19,12 @@ COMMON OPTIONS
-f:: -f::
--format=:: --format=::
Specify format style. Specify format style.
Current available format styles are, Current available format styles are:
'default':: 'default'::
Default style. This is mainly for human reading. Default style. This is mainly for human reading.
--------------------- ---------------------
% perf bench sched pipe # with no style specify % perf bench sched pipe # with no style specified
(executing 1000000 pipe operations between two tasks) (executing 1000000 pipe operations between two tasks)
Total time:5.855 sec Total time:5.855 sec
5.855061 usecs/op 5.855061 usecs/op

4
tools/perf/Documentation/perf-buildid-cache.txt
View file

@ -8,7 +8,7 @@ perf-buildid-cache - Manage build-id cache.
SYNOPSIS SYNOPSIS
-------- --------
[verse] [verse]
'perf buildid-list <options>' 'perf buildid-cache <options>'
DESCRIPTION DESCRIPTION
----------- -----------
@ -30,4 +30,4 @@ OPTIONS
SEE ALSO SEE ALSO
-------- --------
linkperf:perf-record[1], linkperf:perf-report[1] linkperf:perf-record[1], linkperf:perf-report[1], linkperf:perf-buildid-list[1]

2
tools/perf/Documentation/perf-diff.txt
View file

@ -1,5 +1,5 @@
perf-diff(1) perf-diff(1)
============== ============
NAME NAME
---- ----

35
tools/perf/Documentation/perf-inject.txt Normal file
View file

@ -0,0 +1,35 @@
perf-inject(1)
==============
NAME
----
perf-inject - Filter to augment the events stream with additional information
SYNOPSIS
--------
[verse]
'perf inject <options>'
DESCRIPTION
-----------
perf-inject reads a perf-record event stream and repipes it to stdout. At any
point the processing code can inject other events into the event stream - in
this case build-ids (-b option) are read and injected as needed into the event
stream.
Build-ids are just the first user of perf-inject - potentially anything that
needs userspace processing to augment the events stream with additional
information could make use of this facility.
OPTIONS
-------
-b::
--build-ids=::
Inject build-ids into the output stream
-v::
--verbose::
Be more verbose.
SEE ALSO
--------
linkperf:perf-record[1], linkperf:perf-report[1], linkperf:perf-archive[1]

2
tools/perf/Documentation/perf-kmem.txt
View file

@ -1,5 +1,5 @@
perf-kmem(1) perf-kmem(1)
============== ============
NAME NAME
---- ----

68
tools/perf/Documentation/perf-kvm.txt Normal file
View file

@ -0,0 +1,68 @@
perf-kvm(1)
===========
NAME
----
perf-kvm - Tool to trace/measure kvm guest os
SYNOPSIS
--------
[verse]
'perf kvm' [--host] [--guest] [--guestmount=<path>
[--guestkallsyms=<path> --guestmodules=<path> | --guestvmlinux=<path>]]
{top|record|report|diff|buildid-list}
'perf kvm' [--host] [--guest] [--guestkallsyms=<path> --guestmodules=<path>
| --guestvmlinux=<path>] {top|record|report|diff|buildid-list}
DESCRIPTION
-----------
There are a couple of variants of perf kvm:
'perf kvm [options] top <command>' to generates and displays
a performance counter profile of guest os in realtime
of an arbitrary workload.
'perf kvm record <command>' to record the performance couinter profile
of an arbitrary workload and save it into a perf data file. If both
--host and --guest are input, the perf data file name is perf.data.kvm.
If there is no --host but --guest, the file name is perf.data.guest.
If there is no --guest but --host, the file name is perf.data.host.
'perf kvm report' to display the performance counter profile information
recorded via perf kvm record.
'perf kvm diff' to displays the performance difference amongst two perf.data
files captured via perf record.
'perf kvm buildid-list' to display the buildids found in a perf data file,
so that other tools can be used to fetch packages with matching symbol tables
for use by perf report.
OPTIONS
-------
--host=::
Collect host side performance profile.
--guest=::
Collect guest side performance profile.
--guestmount=<path>::
Guest os root file system mount directory. Users mounts guest os
root directories under <path> by a specific filesystem access method,
typically, sshfs. For example, start 2 guest os. The one's pid is 8888
and the other's is 9999.
#mkdir ~/guestmount; cd ~/guestmount
#sshfs -o allow_other,direct_io -p 5551 localhost:/ 8888/
#sshfs -o allow_other,direct_io -p 5552 localhost:/ 9999/
#perf kvm --host --guest --guestmount=~/guestmount top
--guestkallsyms=<path>::
Guest os /proc/kallsyms file copy. 'perf' kvm' reads it to get guest
kernel symbols. Users copy it out from guest os.
--guestmodules=<path>::
Guest os /proc/modules file copy. 'perf' kvm' reads it to get guest
kernel module information. Users copy it out from guest os.
--guestvmlinux=<path>::
Guest os kernel vmlinux.
SEE ALSO
--------
linkperf:perf-top[1], linkperf:perf-record[1], linkperf:perf-report[1],
linkperf:perf-diff[1], linkperf:perf-buildid-list[1]

33
tools/perf/Documentation/perf-list.txt
View file

@ -15,6 +15,35 @@ DESCRIPTION
This command displays the symbolic event types which can be selected in the This command displays the symbolic event types which can be selected in the
various perf commands with the -e option. various perf commands with the -e option.
RAW HARDWARE EVENT DESCRIPTOR
-----------------------------
Even when an event is not available in a symbolic form within perf right now,
it can be encoded in a per processor specific way.
For instance For x86 CPUs NNN represents the raw register encoding with the
layout of IA32_PERFEVTSELx MSRs (see [Intel® 64 and IA-32 Architectures Software Developer's Manual Volume 3B: System Programming Guide] Figure 30-1 Layout
of IA32_PERFEVTSELx MSRs) or AMD's PerfEvtSeln (see [AMD64 Architecture Programmer’s Manual Volume 2: System Programming], Page 344,
Figure 13-7 Performance Event-Select Register (PerfEvtSeln)).
Example:
If the Intel docs for a QM720 Core i7 describe an event as:
Event Umask Event Mask
Num. Value Mnemonic Description Comment
A8H 01H LSD.UOPS Counts the number of micro-ops Use cmask=1 and
delivered by loop stream detector invert to count
cycles
raw encoding of 0x1A8 can be used:
perf stat -e r1a8 -a sleep 1
perf record -e r1a8 ...
You should refer to the processor specific documentation for getting these
details. Some of them are referenced in the SEE ALSO section below.
OPTIONS OPTIONS
------- -------
None None
@ -22,4 +51,6 @@ None
SEE ALSO SEE ALSO
-------- --------
linkperf:perf-stat[1], linkperf:perf-top[1], linkperf:perf-stat[1], linkperf:perf-top[1],
linkperf:perf-record[1] linkperf:perf-record[1],
http://www.intel.com/Assets/PDF/manual/253669.pdf[Intel® 64 and IA-32 Architectures Software Developer's Manual Volume 3B: System Programming Guide],
http://support.amd.com/us/Processor_TechDocs/24593.pdf[AMD64 Architecture Programmer’s Manual Volume 2: System Programming]

21
tools/perf/Documentation/perf-probe.txt
View file

@ -57,6 +57,14 @@ OPTIONS
--force:: --force::
Forcibly add events with existing name. Forcibly add events with existing name.
-n::
--dry-run::
Dry run. With this option, --add and --del doesn't execute actual
adding and removal operations.
--max-probes::
Set the maximum number of probe points for an event. Default is 128.
PROBE SYNTAX PROBE SYNTAX
------------ ------------
Probe points are defined by following syntax. Probe points are defined by following syntax.
@ -74,13 +82,22 @@ Probe points are defined by following syntax.
'EVENT' specifies the name of new event, if omitted, it will be set the name of the probed function. Currently, event group name is set as 'probe'. 'EVENT' specifies the name of new event, if omitted, it will be set the name of the probed function. Currently, event group name is set as 'probe'.
'FUNC' specifies a probed function name, and it may have one of the following options; '+OFFS' is the offset from function entry address in bytes, ':RLN' is the relative-line number from function entry line, and '%return' means that it probes function return. And ';PTN' means lazy matching pattern (see LAZY MATCHING). Note that ';PTN' must be the end of the probe point definition. In addition, '@SRC' specifies a source file which has that function. 'FUNC' specifies a probed function name, and it may have one of the following options; '+OFFS' is the offset from function entry address in bytes, ':RLN' is the relative-line number from function entry line, and '%return' means that it probes function return. And ';PTN' means lazy matching pattern (see LAZY MATCHING). Note that ';PTN' must be the end of the probe point definition. In addition, '@SRC' specifies a source file which has that function.
It is also possible to specify a probe point by the source line number or lazy matching by using 'SRC:ALN' or 'SRC;PTN' syntax, where 'SRC' is the source file path, ':ALN' is the line number and ';PTN' is the lazy matching pattern. It is also possible to specify a probe point by the source line number or lazy matching by using 'SRC:ALN' or 'SRC;PTN' syntax, where 'SRC' is the source file path, ':ALN' is the line number and ';PTN' is the lazy matching pattern.
'ARG' specifies the arguments of this probe point. You can use the name of local variable, or kprobe-tracer argument format (e.g. $retval, %ax, etc). 'ARG' specifies the arguments of this probe point, (see PROBE ARGUMENT).
PROBE ARGUMENT
--------------
Each probe argument follows below syntax.
[NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE]
'NAME' specifies the name of this argument (optional). You can use the name of local variable, local data structure member (e.g. var->field, var.field2), or kprobe-tracer argument format (e.g. $retval, %ax, etc). Note that the name of this argument will be set as the last member name if you specify a local data structure member (e.g. field2 for 'var->field1.field2'.)
'TYPE' casts the type of this argument (optional). If omitted, perf probe automatically set the type based on debuginfo.
LINE SYNTAX LINE SYNTAX
----------- -----------
Line range is descripted by following syntax. Line range is descripted by following syntax.
"FUNC[:RLN[+NUM|:RLN2]]|SRC:ALN[+NUM|:ALN2]" "FUNC[:RLN[+NUM|-RLN2]]|SRC:ALN[+NUM|-ALN2]"
FUNC specifies the function name of showing lines. 'RLN' is the start line FUNC specifies the function name of showing lines. 'RLN' is the start line
number from function entry line, and 'RLN2' is the end line number. As same as number from function entry line, and 'RLN2' is the end line number. As same as

8
tools/perf/Documentation/perf-record.txt
View file

@ -58,7 +58,7 @@ OPTIONS
-f:: -f::
--force:: --force::
Overwrite existing data file. Overwrite existing data file. (deprecated)
-c:: -c::
--count=:: --count=::
@ -69,8 +69,8 @@ OPTIONS
Output file name. Output file name.
-i:: -i::
--inherit:: --no-inherit::
Child tasks inherit counters. Child tasks do not inherit counters.
-F:: -F::
--freq=:: --freq=::
Profile at this frequency. Profile at this frequency.
@ -101,7 +101,7 @@ OPTIONS
-R:: -R::
--raw-samples:: --raw-samples::
Collect raw sample records from all opened counters (typically for tracepoint counters). Collect raw sample records from all opened counters (default for tracepoint counters).
SEE ALSO SEE ALSO
-------- --------

4
tools/perf/Documentation/perf-sched.txt
View file

@ -12,7 +12,7 @@ SYNOPSIS
DESCRIPTION DESCRIPTION
----------- -----------
There's four variants of perf sched: There are four variants of perf sched:
'perf sched record <command>' to record the scheduling events 'perf sched record <command>' to record the scheduling events
of an arbitrary workload. of an arbitrary workload.
@ -27,7 +27,7 @@ There's four variants of perf sched:
via perf sched record. (this is done by starting up mockup threads via perf sched record. (this is done by starting up mockup threads
that mimic the workload based on the events in the trace. These that mimic the workload based on the events in the trace. These
threads can then replay the timings (CPU runtime and sleep patterns) threads can then replay the timings (CPU runtime and sleep patterns)
of the workload as it occured when it was recorded - and can repeat of the workload as it occurred when it was recorded - and can repeat
it a number of times, measuring its performance.) it a number of times, measuring its performance.)
OPTIONS OPTIONS

4
tools/perf/Documentation/perf-stat.txt
View file

@ -31,8 +31,8 @@ OPTIONS
hexadecimal event descriptor. hexadecimal event descriptor.
-i:: -i::
--inherit:: --no-inherit::
child tasks inherit counters child tasks do not inherit counters
-p:: -p::
--pid=<pid>:: --pid=<pid>::
stat events on existing pid stat events on existing pid

22
tools/perf/Documentation/perf-test.txt Normal file
View file

@ -0,0 +1,22 @@
perf-test(1)
============
NAME
----
perf-test - Runs sanity tests.
SYNOPSIS
--------
[verse]
'perf test <options>'
DESCRIPTION
-----------
This command does assorted sanity tests, initially thru linked routines but
also will look for a directory with more tests in the form of scripts.
OPTIONS
-------
-v::
--verbose::
Be more verbose.

6
tools/perf/Documentation/perf-trace-perl.txt
View file

@ -49,12 +49,10 @@ available as calls back into the perf executable (see below).
As an example, the following perf record command can be used to record As an example, the following perf record command can be used to record
all sched_wakeup events in the system: all sched_wakeup events in the system:
# perf record -c 1 -f -a -M -R -e sched:sched_wakeup # perf record -a -e sched:sched_wakeup
Traces meant to be processed using a script should be recorded with Traces meant to be processed using a script should be recorded with
the above options: -c 1 says to sample every event, -a to enable the above option: -a to enable system-wide collection.
system-wide collection, -M to multiplex the output, and -R to collect
raw samples.
The format file for the sched_wakep event defines the following fields The format file for the sched_wakep event defines the following fields
(see /sys/kernel/debug/tracing/events/sched/sched_wakeup/format): (see /sys/kernel/debug/tracing/events/sched/sched_wakeup/format):

16
tools/perf/Documentation/perf-trace-python.txt
View file

@ -1,5 +1,5 @@
perf-trace-python(1) perf-trace-python(1)
================== ====================
NAME NAME
---- ----
@ -93,7 +93,7 @@ don't care how it exited, so we'll use 'perf record' to record only
the sys_enter events: the sys_enter events:
---- ----
# perf record -c 1 -f -a -M -R -e raw_syscalls:sys_enter # perf record -a -e raw_syscalls:sys_enter
^C[ perf record: Woken up 1 times to write data ] ^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 56.545 MB perf.data (~2470503 samples) ] [ perf record: Captured and wrote 56.545 MB perf.data (~2470503 samples) ]
@ -182,7 +182,7 @@ mean either that the record step recorded event types that it wasn't
really interested in, or the script was run against a trace file that really interested in, or the script was run against a trace file that
doesn't correspond to the script. doesn't correspond to the script.
The script generated by -g option option simply prints a line for each The script generated by -g option simply prints a line for each
event found in the trace stream i.e. it basically just dumps the event event found in the trace stream i.e. it basically just dumps the event
and its parameter values to stdout. The print_header() function is and its parameter values to stdout. The print_header() function is
simply a utility function used for that purpose. Let's rename the simply a utility function used for that purpose. Let's rename the
@ -359,7 +359,7 @@ your script:
# cat kernel-source/tools/perf/scripts/python/bin/syscall-counts-record # cat kernel-source/tools/perf/scripts/python/bin/syscall-counts-record
#!/bin/bash #!/bin/bash
perf record -c 1 -f -a -M -R -e raw_syscalls:sys_enter perf record -a -e raw_syscalls:sys_enter
---- ----
The 'report' script is also a shell script with the same base name as The 'report' script is also a shell script with the same base name as
@ -449,12 +449,10 @@ available as calls back into the perf executable (see below).
As an example, the following perf record command can be used to record As an example, the following perf record command can be used to record
all sched_wakeup events in the system: all sched_wakeup events in the system:
# perf record -c 1 -f -a -M -R -e sched:sched_wakeup # perf record -a -e sched:sched_wakeup
Traces meant to be processed using a script should be recorded with Traces meant to be processed using a script should be recorded with
the above options: -c 1 says to sample every event, -a to enable the above option: -a to enable system-wide collection.
system-wide collection, -M to multiplex the output, and -R to collect
raw samples.
The format file for the sched_wakep event defines the following fields The format file for the sched_wakep event defines the following fields
(see /sys/kernel/debug/tracing/events/sched/sched_wakeup/format): (see /sys/kernel/debug/tracing/events/sched/sched_wakeup/format):
@ -584,7 +582,7 @@ files:
flag_str(event_name, field_name, field_value) - returns the string represention corresponding to field_value for the flag field field_name of event event_name flag_str(event_name, field_name, field_value) - returns the string represention corresponding to field_value for the flag field field_name of event event_name
symbol_str(event_name, field_name, field_value) - returns the string represention corresponding to field_value for the symbolic field field_name of event event_name symbol_str(event_name, field_name, field_value) - returns the string represention corresponding to field_value for the symbolic field field_name of event event_name
The *autodict* function returns a special special kind of Python The *autodict* function returns a special kind of Python
dictionary that implements Perl's 'autovivifying' hashes in Python dictionary that implements Perl's 'autovivifying' hashes in Python
i.e. with autovivifying hashes, you can assign nested hash values i.e. with autovivifying hashes, you can assign nested hash values
without having to go to the trouble of creating intermediate levels if without having to go to the trouble of creating intermediate levels if

2
tools/perf/Documentation/perf-trace.txt
View file

@ -1,5 +1,5 @@
perf-trace(1) perf-trace(1)
============== =============
NAME NAME
---- ----

382
tools/perf/Makefile
View file

@ -1,3 +1,7 @@
ifeq ("$(origin O)", "command line")
OUTPUT := $(O)/
endif
# The default target of this Makefile is... # The default target of this Makefile is...
all:: all::
@ -150,10 +154,17 @@ all::
# Define LDFLAGS=-static to build a static binary. # Define LDFLAGS=-static to build a static binary.
# #
# Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds. # Define EXTRA_CFLAGS=-m64 or EXTRA_CFLAGS=-m32 as appropriate for cross-builds.
#
# Define NO_DWARF if you do not want debug-info analysis feature at all.
PERF-VERSION-FILE: .FORCE-PERF-VERSION-FILE $(shell sh -c 'mkdir -p $(OUTPUT)scripts/python/Perf-Trace-Util/' 2> /dev/null)
@$(SHELL_PATH) util/PERF-VERSION-GEN $(shell sh -c 'mkdir -p $(OUTPUT)scripts/perl/Perf-Trace-Util/' 2> /dev/null)
-include PERF-VERSION-FILE $(shell sh -c 'mkdir -p $(OUTPUT)util/scripting-engines/' 2> /dev/null)
$(shell sh -c 'mkdir $(OUTPUT)bench' 2> /dev/null)
$(OUTPUT)PERF-VERSION-FILE: .FORCE-PERF-VERSION-FILE
@$(SHELL_PATH) util/PERF-VERSION-GEN $(OUTPUT)
-include $(OUTPUT)PERF-VERSION-FILE
uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not') uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
uname_M := $(shell sh -c 'uname -m 2>/dev/null || echo not') uname_M := $(shell sh -c 'uname -m 2>/dev/null || echo not')
@ -162,6 +173,22 @@ uname_R := $(shell sh -c 'uname -r 2>/dev/null || echo not')
uname_P := $(shell sh -c 'uname -p 2>/dev/null || echo not') uname_P := $(shell sh -c 'uname -p 2>/dev/null || echo not')
uname_V := $(shell sh -c 'uname -v 2>/dev/null || echo not') uname_V := $(shell sh -c 'uname -v 2>/dev/null || echo not')
ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/i386/ -e s/sun4u/sparc64/ \
-e s/arm.*/arm/ -e s/sa110/arm/ \
-e s/s390x/s390/ -e s/parisc64/parisc/ \
-e s/ppc.*/powerpc/ -e s/mips.*/mips/ \
-e s/sh[234].*/sh/ )
# Additional ARCH settings for x86
ifeq ($(ARCH),i386)
ARCH := x86
endif
ifeq ($(ARCH),x86_64)
ARCH := x86
endif
$(shell sh -c 'mkdir -p $(OUTPUT)arch/$(ARCH)/util/' 2> /dev/null)
# CFLAGS and LDFLAGS are for the users to override from the command line. # CFLAGS and LDFLAGS are for the users to override from the command line.
# #
@ -274,7 +301,7 @@ endif
# Those must not be GNU-specific; they are shared with perl/ which may # Those must not be GNU-specific; they are shared with perl/ which may
# be built by a different compiler. (Note that this is an artifact now # be built by a different compiler. (Note that this is an artifact now
# but it still might be nice to keep that distinction.) # but it still might be nice to keep that distinction.)
BASIC_CFLAGS = -Iutil/include BASIC_CFLAGS = -Iutil/include -Iarch/$(ARCH)/include
BASIC_LDFLAGS = BASIC_LDFLAGS =
# Guard against environment variables # Guard against environment variables
@ -308,7 +335,7 @@ PROGRAMS += $(EXTRA_PROGRAMS)
# #
# Single 'perf' binary right now: # Single 'perf' binary right now:
# #
PROGRAMS += perf PROGRAMS += $(OUTPUT)perf
# List built-in command $C whose implementation cmd_$C() is not in # List built-in command $C whose implementation cmd_$C() is not in
# builtin-$C.o but is linked in as part of some other command. # builtin-$C.o but is linked in as part of some other command.
@ -318,7 +345,7 @@ PROGRAMS += perf
ALL_PROGRAMS = $(PROGRAMS) $(SCRIPTS) ALL_PROGRAMS = $(PROGRAMS) $(SCRIPTS)
# what 'all' will build but not install in perfexecdir # what 'all' will build but not install in perfexecdir
OTHER_PROGRAMS = perf$X OTHER_PROGRAMS = $(OUTPUT)perf$X
# Set paths to tools early so that they can be used for version tests. # Set paths to tools early so that they can be used for version tests.
ifndef SHELL_PATH ifndef SHELL_PATH
@ -330,7 +357,7 @@ endif
export PERL_PATH export PERL_PATH
LIB_FILE=libperf.a LIB_FILE=$(OUTPUT)libperf.a
LIB_H += ../../include/linux/perf_event.h LIB_H += ../../include/linux/perf_event.h
LIB_H += ../../include/linux/rbtree.h LIB_H += ../../include/linux/rbtree.h
@ -350,12 +377,13 @@ LIB_H += util/include/linux/rbtree.h
LIB_H += util/include/linux/string.h LIB_H += util/include/linux/string.h
LIB_H += util/include/linux/types.h LIB_H += util/include/linux/types.h
LIB_H += util/include/asm/asm-offsets.h LIB_H += util/include/asm/asm-offsets.h
LIB_H += util/include/asm/bitops.h
LIB_H += util/include/asm/bug.h LIB_H += util/include/asm/bug.h
LIB_H += util/include/asm/byteorder.h LIB_H += util/include/asm/byteorder.h
LIB_H += util/include/asm/hweight.h
LIB_H += util/include/asm/swab.h LIB_H += util/include/asm/swab.h
LIB_H += util/include/asm/system.h LIB_H += util/include/asm/system.h
LIB_H += util/include/asm/uaccess.h LIB_H += util/include/asm/uaccess.h
LIB_H += util/include/dwarf-regs.h
LIB_H += perf.h LIB_H += perf.h
LIB_H += util/cache.h LIB_H += util/cache.h
LIB_H += util/callchain.h LIB_H += util/callchain.h
@ -375,7 +403,6 @@ LIB_H += util/header.h
LIB_H += util/help.h LIB_H += util/help.h
LIB_H += util/session.h LIB_H += util/session.h
LIB_H += util/strbuf.h LIB_H += util/strbuf.h
LIB_H += util/string.h
LIB_H += util/strlist.h LIB_H += util/strlist.h
LIB_H += util/svghelper.h LIB_H += util/svghelper.h
LIB_H += util/run-command.h LIB_H += util/run-command.h
@ -389,79 +416,83 @@ LIB_H += util/thread.h
LIB_H += util/trace-event.h LIB_H += util/trace-event.h
LIB_H += util/probe-finder.h LIB_H += util/probe-finder.h
LIB_H += util/probe-event.h LIB_H += util/probe-event.h
LIB_H += util/pstack.h
LIB_H += util/cpumap.h LIB_H += util/cpumap.h
LIB_OBJS += util/abspath.o LIB_OBJS += $(OUTPUT)util/abspath.o
LIB_OBJS += util/alias.o LIB_OBJS += $(OUTPUT)util/alias.o
LIB_OBJS += util/build-id.o LIB_OBJS += $(OUTPUT)util/build-id.o
LIB_OBJS += util/config.o LIB_OBJS += $(OUTPUT)util/config.o
LIB_OBJS += util/ctype.o LIB_OBJS += $(OUTPUT)util/ctype.o
LIB_OBJS += util/debugfs.o LIB_OBJS += $(OUTPUT)util/debugfs.o
LIB_OBJS += util/environment.o LIB_OBJS += $(OUTPUT)util/environment.o
LIB_OBJS += util/event.o LIB_OBJS += $(OUTPUT)util/event.o
LIB_OBJS += util/exec_cmd.o LIB_OBJS += $(OUTPUT)util/exec_cmd.o
LIB_OBJS += util/help.o LIB_OBJS += $(OUTPUT)util/help.o
LIB_OBJS += util/levenshtein.o LIB_OBJS += $(OUTPUT)util/levenshtein.o
LIB_OBJS += util/parse-options.o LIB_OBJS += $(OUTPUT)util/parse-options.o
LIB_OBJS += util/parse-events.o LIB_OBJS += $(OUTPUT)util/parse-events.o
LIB_OBJS += util/path.o LIB_OBJS += $(OUTPUT)util/path.o
LIB_OBJS += util/rbtree.o LIB_OBJS += $(OUTPUT)util/rbtree.o
LIB_OBJS += util/bitmap.o LIB_OBJS += $(OUTPUT)util/bitmap.o
LIB_OBJS += util/hweight.o LIB_OBJS += $(OUTPUT)util/hweight.o
LIB_OBJS += util/find_next_bit.o LIB_OBJS += $(OUTPUT)util/run-command.o
LIB_OBJS += util/run-command.o LIB_OBJS += $(OUTPUT)util/quote.o
LIB_OBJS += util/quote.o LIB_OBJS += $(OUTPUT)util/strbuf.o
LIB_OBJS += util/strbuf.o LIB_OBJS += $(OUTPUT)util/string.o
LIB_OBJS += util/string.o LIB_OBJS += $(OUTPUT)util/strlist.o
LIB_OBJS += util/strlist.o LIB_OBJS += $(OUTPUT)util/usage.o
LIB_OBJS += util/usage.o LIB_OBJS += $(OUTPUT)util/wrapper.o
LIB_OBJS += util/wrapper.o LIB_OBJS += $(OUTPUT)util/sigchain.o
LIB_OBJS += util/sigchain.o LIB_OBJS += $(OUTPUT)util/symbol.o
LIB_OBJS += util/symbol.o LIB_OBJS += $(OUTPUT)util/color.o
LIB_OBJS += util/color.o LIB_OBJS += $(OUTPUT)util/pager.o
LIB_OBJS += util/pager.o LIB_OBJS += $(OUTPUT)util/header.o
LIB_OBJS += util/header.o LIB_OBJS += $(OUTPUT)util/callchain.o
LIB_OBJS += util/callchain.o LIB_OBJS += $(OUTPUT)util/values.o
LIB_OBJS += util/values.o LIB_OBJS += $(OUTPUT)util/debug.o
LIB_OBJS += util/debug.o LIB_OBJS += $(OUTPUT)util/map.o
LIB_OBJS += util/map.o LIB_OBJS += $(OUTPUT)util/pstack.o
LIB_OBJS += util/session.o LIB_OBJS += $(OUTPUT)util/session.o
LIB_OBJS += util/thread.o LIB_OBJS += $(OUTPUT)util/thread.o
LIB_OBJS += util/trace-event-parse.o LIB_OBJS += $(OUTPUT)util/trace-event-parse.o
LIB_OBJS += util/trace-event-read.o LIB_OBJS += $(OUTPUT)util/trace-event-read.o
LIB_OBJS += util/trace-event-info.o LIB_OBJS += $(OUTPUT)util/trace-event-info.o
LIB_OBJS += util/trace-event-scripting.o LIB_OBJS += $(OUTPUT)util/trace-event-scripting.o
LIB_OBJS += util/svghelper.o LIB_OBJS += $(OUTPUT)util/svghelper.o
LIB_OBJS += util/sort.o LIB_OBJS += $(OUTPUT)util/sort.o
LIB_OBJS += util/hist.o LIB_OBJS += $(OUTPUT)util/hist.o
LIB_OBJS += util/probe-event.o LIB_OBJS += $(OUTPUT)util/probe-event.o
LIB_OBJS += util/util.o LIB_OBJS += $(OUTPUT)util/util.o
LIB_OBJS += util/cpumap.o LIB_OBJS += $(OUTPUT)util/cpumap.o
BUILTIN_OBJS += builtin-annotate.o BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
BUILTIN_OBJS += builtin-bench.o BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
# Benchmark modules # Benchmark modules
BUILTIN_OBJS += bench/sched-messaging.o BUILTIN_OBJS += $(OUTPUT)bench/sched-messaging.o
BUILTIN_OBJS += bench/sched-pipe.o BUILTIN_OBJS += $(OUTPUT)bench/sched-pipe.o
BUILTIN_OBJS += bench/mem-memcpy.o BUILTIN_OBJS += $(OUTPUT)bench/mem-memcpy.o
BUILTIN_OBJS += builtin-diff.o BUILTIN_OBJS += $(OUTPUT)builtin-diff.o
BUILTIN_OBJS += builtin-help.o BUILTIN_OBJS += $(OUTPUT)builtin-help.o
BUILTIN_OBJS += builtin-sched.o BUILTIN_OBJS += $(OUTPUT)builtin-sched.o
BUILTIN_OBJS += builtin-buildid-list.o BUILTIN_OBJS += $(OUTPUT)builtin-buildid-list.o
BUILTIN_OBJS += builtin-buildid-cache.o BUILTIN_OBJS += $(OUTPUT)builtin-buildid-cache.o
BUILTIN_OBJS += builtin-list.o BUILTIN_OBJS += $(OUTPUT)builtin-list.o
BUILTIN_OBJS += builtin-record.o BUILTIN_OBJS += $(OUTPUT)builtin-record.o
BUILTIN_OBJS += builtin-report.o BUILTIN_OBJS += $(OUTPUT)builtin-report.o
BUILTIN_OBJS += builtin-stat.o BUILTIN_OBJS += $(OUTPUT)builtin-stat.o
BUILTIN_OBJS += builtin-timechart.o BUILTIN_OBJS += $(OUTPUT)builtin-timechart.o
BUILTIN_OBJS += builtin-top.o BUILTIN_OBJS += $(OUTPUT)builtin-top.o
BUILTIN_OBJS += builtin-trace.o BUILTIN_OBJS += $(OUTPUT)builtin-trace.o
BUILTIN_OBJS += builtin-probe.o BUILTIN_OBJS += $(OUTPUT)builtin-probe.o
BUILTIN_OBJS += builtin-kmem.o BUILTIN_OBJS += $(OUTPUT)builtin-kmem.o
BUILTIN_OBJS += builtin-lock.o BUILTIN_OBJS += $(OUTPUT)builtin-lock.o
BUILTIN_OBJS += $(OUTPUT)builtin-kvm.o
BUILTIN_OBJS += $(OUTPUT)builtin-test.o
BUILTIN_OBJS += $(OUTPUT)builtin-inject.o
PERFLIBS = $(LIB_FILE) PERFLIBS = $(LIB_FILE)
@ -476,6 +507,15 @@ PERFLIBS = $(LIB_FILE)
-include config.mak.autogen -include config.mak.autogen
-include config.mak -include config.mak
ifndef NO_DWARF
ifneq ($(shell sh -c "(echo '\#include <dwarf.h>'; echo '\#include <libdw.h>'; echo '\#include <version.h>'; echo '\#ifndef _ELFUTILS_PREREQ'; echo '\#error'; echo '\#endif'; echo 'int main(void) { Dwarf *dbg; dbg = dwarf_begin(0, DWARF_C_READ); return (long)dbg; }') | $(CC) -x c - $(ALL_CFLAGS) -I/usr/include/elfutils -ldw -lelf -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) "$(QUIET_STDERR)" && echo y"), y)
msg := $(warning No libdw.h found or old libdw.h found or elfutils is older than 0.138, disables dwarf support. Please install new elfutils-devel/libdw-dev);
NO_DWARF := 1
endif # Dwarf support
endif # NO_DWARF
-include arch/$(ARCH)/Makefile
ifeq ($(uname_S),Darwin) ifeq ($(uname_S),Darwin)
ifndef NO_FINK ifndef NO_FINK
ifeq ($(shell test -d /sw/lib && echo y),y) ifeq ($(shell test -d /sw/lib && echo y),y)
@ -492,6 +532,10 @@ ifeq ($(uname_S),Darwin)
PTHREAD_LIBS = PTHREAD_LIBS =
endif endif
ifneq ($(OUTPUT),)
BASIC_CFLAGS += -I$(OUTPUT)
endif
ifeq ($(shell sh -c "(echo '\#include <libelf.h>'; echo 'int main(void) { Elf * elf = elf_begin(0, ELF_C_READ, 0); return (long)elf; }') | $(CC) -x c - $(ALL_CFLAGS) -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) "$(QUIET_STDERR)" && echo y"), y) ifeq ($(shell sh -c "(echo '\#include <libelf.h>'; echo 'int main(void) { Elf * elf = elf_begin(0, ELF_C_READ, 0); return (long)elf; }') | $(CC) -x c - $(ALL_CFLAGS) -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) "$(QUIET_STDERR)" && echo y"), y)
ifneq ($(shell sh -c "(echo '\#include <gnu/libc-version.h>'; echo 'int main(void) { const char * version = gnu_get_libc_version(); return (long)version; }') | $(CC) -x c - $(ALL_CFLAGS) -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) "$(QUIET_STDERR)" && echo y"), y) ifneq ($(shell sh -c "(echo '\#include <gnu/libc-version.h>'; echo 'int main(void) { const char * version = gnu_get_libc_version(); return (long)version; }') | $(CC) -x c - $(ALL_CFLAGS) -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) "$(QUIET_STDERR)" && echo y"), y)
msg := $(error No gnu/libc-version.h found, please install glibc-dev[el]/glibc-static); msg := $(error No gnu/libc-version.h found, please install glibc-dev[el]/glibc-static);
@ -504,14 +548,29 @@ else
msg := $(error No libelf.h/libelf found, please install libelf-dev/elfutils-libelf-devel and glibc-dev[el]); msg := $(error No libelf.h/libelf found, please install libelf-dev/elfutils-libelf-devel and glibc-dev[el]);
endif endif
ifneq ($(shell sh -c "(echo '\#include <dwarf.h>'; echo '\#include <libdw.h>'; echo 'int main(void) { Dwarf *dbg; dbg = dwarf_begin(0, DWARF_C_READ); return (long)dbg; }') | $(CC) -x c - $(ALL_CFLAGS) -I/usr/include/elfutils -ldw -lelf -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) "$(QUIET_STDERR)" && echo y"), y) ifndef NO_DWARF
msg := $(warning No libdw.h found or old libdw.h found, disables dwarf support. Please install elfutils-devel/elfutils-dev); ifeq ($(origin PERF_HAVE_DWARF_REGS), undefined)
BASIC_CFLAGS += -DNO_DWARF_SUPPORT msg := $(warning DWARF register mappings have not been defined for architecture $(ARCH), DWARF support disabled);
else else
BASIC_CFLAGS += -I/usr/include/elfutils BASIC_CFLAGS += -I/usr/include/elfutils -DDWARF_SUPPORT
EXTLIBS += -lelf -ldw EXTLIBS += -lelf -ldw
LIB_OBJS += util/probe-finder.o LIB_OBJS += $(OUTPUT)util/probe-finder.o
endif # PERF_HAVE_DWARF_REGS
endif # NO_DWARF
ifdef NO_NEWT
BASIC_CFLAGS += -DNO_NEWT_SUPPORT
else
ifneq ($(shell sh -c "(echo '\#include <newt.h>'; echo 'int main(void) { newtInit(); newtCls(); return newtFinished(); }') | $(CC) -x c - $(ALL_CFLAGS) -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -lnewt -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) "$(QUIET_STDERR)" && echo y"), y)
msg := $(warning newt not found, disables TUI support. Please install newt-devel or libnewt-dev);
BASIC_CFLAGS += -DNO_NEWT_SUPPORT
else
# Fedora has /usr/include/slang/slang.h, but ubuntu /usr/include/slang.h
BASIC_CFLAGS += -I/usr/include/slang
EXTLIBS += -lnewt -lslang
LIB_OBJS += $(OUTPUT)util/newt.o
endif endif
endif # NO_NEWT
ifndef NO_LIBPERL ifndef NO_LIBPERL
PERL_EMBED_LDOPTS = `perl -MExtUtils::Embed -e ldopts 2>/dev/null` PERL_EMBED_LDOPTS = `perl -MExtUtils::Embed -e ldopts 2>/dev/null`
@ -522,8 +581,8 @@ ifneq ($(shell sh -c "(echo '\#include <EXTERN.h>'; echo '\#include <perl.h>'; e
BASIC_CFLAGS += -DNO_LIBPERL BASIC_CFLAGS += -DNO_LIBPERL
else else
ALL_LDFLAGS += $(PERL_EMBED_LDOPTS) ALL_LDFLAGS += $(PERL_EMBED_LDOPTS)
LIB_OBJS += util/scripting-engines/trace-event-perl.o LIB_OBJS += $(OUTPUT)util/scripting-engines/trace-event-perl.o
LIB_OBJS += scripts/perl/Perf-Trace-Util/Context.o LIB_OBJS += $(OUTPUT)scripts/perl/Perf-Trace-Util/Context.o
endif endif
ifndef NO_LIBPYTHON ifndef NO_LIBPYTHON
@ -531,16 +590,19 @@ PYTHON_EMBED_LDOPTS = `python-config --ldflags 2>/dev/null`
PYTHON_EMBED_CCOPTS = `python-config --cflags 2>/dev/null` PYTHON_EMBED_CCOPTS = `python-config --cflags 2>/dev/null`
endif endif
ifneq ($(shell sh -c "(echo '\#include <Python.h>'; echo 'int main(void) { Py_Initialize(); return 0; }') | $(CC) -x c - $(PYTHON_EMBED_CCOPTS) -o /dev/null $(PYTHON_EMBED_LDOPTS) > /dev/null 2>&1 && echo y"), y) ifneq ($(shell sh -c "(echo '\#include <Python.h>'; echo 'int main(void) { Py_Initialize(); return 0; }') | $(CC) -x c - $(PYTHON_EMBED_CCOPTS) -o $(BITBUCKET) $(PYTHON_EMBED_LDOPTS) > /dev/null 2>&1 && echo y"), y)
BASIC_CFLAGS += -DNO_LIBPYTHON BASIC_CFLAGS += -DNO_LIBPYTHON
else else
ALL_LDFLAGS += $(PYTHON_EMBED_LDOPTS) ALL_LDFLAGS += $(PYTHON_EMBED_LDOPTS)
LIB_OBJS += util/scripting-engines/trace-event-python.o LIB_OBJS += $(OUTPUT)util/scripting-engines/trace-event-python.o
LIB_OBJS += scripts/python/Perf-Trace-Util/Context.o LIB_OBJS += $(OUTPUT)scripts/python/Perf-Trace-Util/Context.o
endif endif
ifdef NO_DEMANGLE ifdef NO_DEMANGLE
BASIC_CFLAGS += -DNO_DEMANGLE BASIC_CFLAGS += -DNO_DEMANGLE
else ifdef HAVE_CPLUS_DEMANGLE
EXTLIBS += -liberty
BASIC_CFLAGS += -DHAVE_CPLUS_DEMANGLE
else else
has_bfd := $(shell sh -c "(echo '\#include <bfd.h>'; echo 'int main(void) { bfd_demangle(0, 0, 0); return 0; }') | $(CC) -x c - $(ALL_CFLAGS) -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) -lbfd "$(QUIET_STDERR)" && echo y") has_bfd := $(shell sh -c "(echo '\#include <bfd.h>'; echo 'int main(void) { bfd_demangle(0, 0, 0); return 0; }') | $(CC) -x c - $(ALL_CFLAGS) -o $(BITBUCKET) $(ALL_LDFLAGS) $(EXTLIBS) -lbfd "$(QUIET_STDERR)" && echo y")
@ -607,53 +669,53 @@ ifdef NO_C99_FORMAT
endif endif
ifdef SNPRINTF_RETURNS_BOGUS ifdef SNPRINTF_RETURNS_BOGUS
COMPAT_CFLAGS += -DSNPRINTF_RETURNS_BOGUS COMPAT_CFLAGS += -DSNPRINTF_RETURNS_BOGUS
COMPAT_OBJS += compat/snprintf.o COMPAT_OBJS += $(OUTPUT)compat/snprintf.o
endif endif
ifdef FREAD_READS_DIRECTORIES ifdef FREAD_READS_DIRECTORIES
COMPAT_CFLAGS += -DFREAD_READS_DIRECTORIES COMPAT_CFLAGS += -DFREAD_READS_DIRECTORIES
COMPAT_OBJS += compat/fopen.o COMPAT_OBJS += $(OUTPUT)compat/fopen.o
endif endif
ifdef NO_SYMLINK_HEAD ifdef NO_SYMLINK_HEAD
BASIC_CFLAGS += -DNO_SYMLINK_HEAD BASIC_CFLAGS += -DNO_SYMLINK_HEAD
endif endif
ifdef NO_STRCASESTR ifdef NO_STRCASESTR
COMPAT_CFLAGS += -DNO_STRCASESTR COMPAT_CFLAGS += -DNO_STRCASESTR
COMPAT_OBJS += compat/strcasestr.o COMPAT_OBJS += $(OUTPUT)compat/strcasestr.o
endif endif
ifdef NO_STRTOUMAX ifdef NO_STRTOUMAX
COMPAT_CFLAGS += -DNO_STRTOUMAX COMPAT_CFLAGS += -DNO_STRTOUMAX
COMPAT_OBJS += compat/strtoumax.o COMPAT_OBJS += $(OUTPUT)compat/strtoumax.o
endif endif
ifdef NO_STRTOULL ifdef NO_STRTOULL
COMPAT_CFLAGS += -DNO_STRTOULL COMPAT_CFLAGS += -DNO_STRTOULL
endif endif
ifdef NO_SETENV ifdef NO_SETENV
COMPAT_CFLAGS += -DNO_SETENV COMPAT_CFLAGS += -DNO_SETENV
COMPAT_OBJS += compat/setenv.o COMPAT_OBJS += $(OUTPUT)compat/setenv.o
endif endif
ifdef NO_MKDTEMP ifdef NO_MKDTEMP
COMPAT_CFLAGS += -DNO_MKDTEMP COMPAT_CFLAGS += -DNO_MKDTEMP
COMPAT_OBJS += compat/mkdtemp.o COMPAT_OBJS += $(OUTPUT)compat/mkdtemp.o
endif endif
ifdef NO_UNSETENV ifdef NO_UNSETENV
COMPAT_CFLAGS += -DNO_UNSETENV COMPAT_CFLAGS += -DNO_UNSETENV
COMPAT_OBJS += compat/unsetenv.o COMPAT_OBJS += $(OUTPUT)compat/unsetenv.o
endif endif
ifdef NO_SYS_SELECT_H ifdef NO_SYS_SELECT_H
BASIC_CFLAGS += -DNO_SYS_SELECT_H BASIC_CFLAGS += -DNO_SYS_SELECT_H
endif endif
ifdef NO_MMAP ifdef NO_MMAP
COMPAT_CFLAGS += -DNO_MMAP COMPAT_CFLAGS += -DNO_MMAP
COMPAT_OBJS += compat/mmap.o COMPAT_OBJS += $(OUTPUT)compat/mmap.o
else else
ifdef USE_WIN32_MMAP ifdef USE_WIN32_MMAP
COMPAT_CFLAGS += -DUSE_WIN32_MMAP COMPAT_CFLAGS += -DUSE_WIN32_MMAP
COMPAT_OBJS += compat/win32mmap.o COMPAT_OBJS += $(OUTPUT)compat/win32mmap.o
endif endif
endif endif
ifdef NO_PREAD ifdef NO_PREAD
COMPAT_CFLAGS += -DNO_PREAD COMPAT_CFLAGS += -DNO_PREAD
COMPAT_OBJS += compat/pread.o COMPAT_OBJS += $(OUTPUT)compat/pread.o
endif endif
ifdef NO_FAST_WORKING_DIRECTORY ifdef NO_FAST_WORKING_DIRECTORY
BASIC_CFLAGS += -DNO_FAST_WORKING_DIRECTORY BASIC_CFLAGS += -DNO_FAST_WORKING_DIRECTORY
@ -675,10 +737,10 @@ else
endif endif
endif endif
ifdef NO_INET_NTOP ifdef NO_INET_NTOP
LIB_OBJS += compat/inet_ntop.o LIB_OBJS += $(OUTPUT)compat/inet_ntop.o
endif endif
ifdef NO_INET_PTON ifdef NO_INET_PTON
LIB_OBJS += compat/inet_pton.o LIB_OBJS += $(OUTPUT)compat/inet_pton.o
endif endif
ifdef NO_ICONV ifdef NO_ICONV
@ -695,15 +757,15 @@ endif
ifdef PPC_SHA1 ifdef PPC_SHA1
SHA1_HEADER = "ppc/sha1.h" SHA1_HEADER = "ppc/sha1.h"
LIB_OBJS += ppc/sha1.o ppc/sha1ppc.o LIB_OBJS += $(OUTPUT)ppc/sha1.o ppc/sha1ppc.o
else else
ifdef ARM_SHA1 ifdef ARM_SHA1
SHA1_HEADER = "arm/sha1.h" SHA1_HEADER = "arm/sha1.h"
LIB_OBJS += arm/sha1.o arm/sha1_arm.o LIB_OBJS += $(OUTPUT)arm/sha1.o $(OUTPUT)arm/sha1_arm.o
else else
ifdef MOZILLA_SHA1 ifdef MOZILLA_SHA1
SHA1_HEADER = "mozilla-sha1/sha1.h" SHA1_HEADER = "mozilla-sha1/sha1.h"
LIB_OBJS += mozilla-sha1/sha1.o LIB_OBJS += $(OUTPUT)mozilla-sha1/sha1.o
else else
SHA1_HEADER = <openssl/sha.h> SHA1_HEADER = <openssl/sha.h>
EXTLIBS += $(LIB_4_CRYPTO) EXTLIBS += $(LIB_4_CRYPTO)
@ -715,15 +777,15 @@ ifdef NO_PERL_MAKEMAKER
endif endif
ifdef NO_HSTRERROR ifdef NO_HSTRERROR
COMPAT_CFLAGS += -DNO_HSTRERROR COMPAT_CFLAGS += -DNO_HSTRERROR
COMPAT_OBJS += compat/hstrerror.o COMPAT_OBJS += $(OUTPUT)compat/hstrerror.o
endif endif
ifdef NO_MEMMEM ifdef NO_MEMMEM
COMPAT_CFLAGS += -DNO_MEMMEM COMPAT_CFLAGS += -DNO_MEMMEM
COMPAT_OBJS += compat/memmem.o COMPAT_OBJS += $(OUTPUT)compat/memmem.o
endif endif
ifdef INTERNAL_QSORT ifdef INTERNAL_QSORT
COMPAT_CFLAGS += -DINTERNAL_QSORT COMPAT_CFLAGS += -DINTERNAL_QSORT
COMPAT_OBJS += compat/qsort.o COMPAT_OBJS += $(OUTPUT)compat/qsort.o
endif endif
ifdef RUNTIME_PREFIX ifdef RUNTIME_PREFIX
COMPAT_CFLAGS += -DRUNTIME_PREFIX COMPAT_CFLAGS += -DRUNTIME_PREFIX
@ -803,7 +865,7 @@ export TAR INSTALL DESTDIR SHELL_PATH
SHELL = $(SHELL_PATH) SHELL = $(SHELL_PATH)
all:: .perf.dev.null shell_compatibility_test $(ALL_PROGRAMS) $(BUILT_INS) $(OTHER_PROGRAMS) PERF-BUILD-OPTIONS all:: .perf.dev.null shell_compatibility_test $(ALL_PROGRAMS) $(BUILT_INS) $(OTHER_PROGRAMS) $(OUTPUT)PERF-BUILD-OPTIONS
ifneq (,$X) ifneq (,$X)
$(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_PROGRAMS) $(BUILT_INS) perf$X)), test '$p' -ef '$p$X' || $(RM) '$p';) $(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_PROGRAMS) $(BUILT_INS) perf$X)), test '$p' -ef '$p$X' || $(RM) '$p';)
endif endif
@ -815,39 +877,39 @@ please_set_SHELL_PATH_to_a_more_modern_shell:
shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
strip: $(PROGRAMS) perf$X strip: $(PROGRAMS) $(OUTPUT)perf$X
$(STRIP) $(STRIP_OPTS) $(PROGRAMS) perf$X $(STRIP) $(STRIP_OPTS) $(PROGRAMS) $(OUTPUT)perf$X
perf.o: perf.c common-cmds.h PERF-CFLAGS $(OUTPUT)perf.o: perf.c $(OUTPUT)common-cmds.h $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -DPERF_VERSION='"$(PERF_VERSION)"' \ $(QUIET_CC)$(CC) -DPERF_VERSION='"$(PERF_VERSION)"' \
'-DPERF_HTML_PATH="$(htmldir_SQ)"' \ '-DPERF_HTML_PATH="$(htmldir_SQ)"' \
$(ALL_CFLAGS) -c $(filter %.c,$^) $(ALL_CFLAGS) -c $(filter %.c,$^) -o $@
perf$X: perf.o $(BUILTIN_OBJS) $(PERFLIBS) $(OUTPUT)perf$X: $(OUTPUT)perf.o $(BUILTIN_OBJS) $(PERFLIBS)
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ perf.o \ $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(OUTPUT)perf.o \
$(BUILTIN_OBJS) $(ALL_LDFLAGS) $(LIBS) $(BUILTIN_OBJS) $(ALL_LDFLAGS) $(LIBS)
builtin-help.o: builtin-help.c common-cmds.h PERF-CFLAGS $(OUTPUT)builtin-help.o: builtin-help.c $(OUTPUT)common-cmds.h $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o $*.o -c $(ALL_CFLAGS) \ $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) \
'-DPERF_HTML_PATH="$(htmldir_SQ)"' \ '-DPERF_HTML_PATH="$(htmldir_SQ)"' \
'-DPERF_MAN_PATH="$(mandir_SQ)"' \ '-DPERF_MAN_PATH="$(mandir_SQ)"' \
'-DPERF_INFO_PATH="$(infodir_SQ)"' $< '-DPERF_INFO_PATH="$(infodir_SQ)"' $<
builtin-timechart.o: builtin-timechart.c common-cmds.h PERF-CFLAGS $(OUTPUT)builtin-timechart.o: builtin-timechart.c $(OUTPUT)common-cmds.h $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o $*.o -c $(ALL_CFLAGS) \ $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) \
'-DPERF_HTML_PATH="$(htmldir_SQ)"' \ '-DPERF_HTML_PATH="$(htmldir_SQ)"' \
'-DPERF_MAN_PATH="$(mandir_SQ)"' \ '-DPERF_MAN_PATH="$(mandir_SQ)"' \
'-DPERF_INFO_PATH="$(infodir_SQ)"' $< '-DPERF_INFO_PATH="$(infodir_SQ)"' $<
$(BUILT_INS): perf$X $(BUILT_INS): $(OUTPUT)perf$X
$(QUIET_BUILT_IN)$(RM) $@ && \ $(QUIET_BUILT_IN)$(RM) $@ && \
ln perf$X $@ 2>/dev/null || \ ln perf$X $@ 2>/dev/null || \
ln -s perf$X $@ 2>/dev/null || \ ln -s perf$X $@ 2>/dev/null || \
cp perf$X $@ cp perf$X $@
common-cmds.h: util/generate-cmdlist.sh command-list.txt $(OUTPUT)common-cmds.h: util/generate-cmdlist.sh command-list.txt
common-cmds.h: $(wildcard Documentation/perf-*.txt) $(OUTPUT)common-cmds.h: $(wildcard Documentation/perf-*.txt)
$(QUIET_GEN). util/generate-cmdlist.sh > $@+ && mv $@+ $@ $(QUIET_GEN). util/generate-cmdlist.sh > $@+ && mv $@+ $@
$(patsubst %.sh,%,$(SCRIPT_SH)) : % : %.sh $(patsubst %.sh,%,$(SCRIPT_SH)) : % : %.sh
@ -859,7 +921,7 @@ $(patsubst %.sh,%,$(SCRIPT_SH)) : % : %.sh
-e 's/@@NO_CURL@@/$(NO_CURL)/g' \ -e 's/@@NO_CURL@@/$(NO_CURL)/g' \
$@.sh >$@+ && \ $@.sh >$@+ && \
chmod +x $@+ && \ chmod +x $@+ && \
mv $@+ $@ mv $@+ $(OUTPUT)$@
configure: configure.ac configure: configure.ac
$(QUIET_GEN)$(RM) $@ $<+ && \ $(QUIET_GEN)$(RM) $@ $<+ && \
@ -869,60 +931,50 @@ configure: configure.ac
$(RM) $<+ $(RM) $<+
# These can record PERF_VERSION # These can record PERF_VERSION
perf.o perf.spec \ $(OUTPUT)perf.o perf.spec \
$(patsubst %.sh,%,$(SCRIPT_SH)) \ $(patsubst %.sh,%,$(SCRIPT_SH)) \
$(patsubst %.perl,%,$(SCRIPT_PERL)) \ $(patsubst %.perl,%,$(SCRIPT_PERL)) \
: PERF-VERSION-FILE : $(OUTPUT)PERF-VERSION-FILE
%.o: %.c PERF-CFLAGS $(OUTPUT)%.o: %.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o $*.o -c $(ALL_CFLAGS) $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) $<
%.s: %.c PERF-CFLAGS $(OUTPUT)%.s: %.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -S $(ALL_CFLAGS) $< $(QUIET_CC)$(CC) -S $(ALL_CFLAGS) $<
%.o: %.S $(OUTPUT)%.o: %.S
$(QUIET_CC)$(CC) -o $*.o -c $(ALL_CFLAGS) $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) $<
util/exec_cmd.o: util/exec_cmd.c PERF-CFLAGS $(OUTPUT)util/exec_cmd.o: util/exec_cmd.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o $*.o -c $(ALL_CFLAGS) \ $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) \
'-DPERF_EXEC_PATH="$(perfexecdir_SQ)"' \ '-DPERF_EXEC_PATH="$(perfexecdir_SQ)"' \
'-DBINDIR="$(bindir_relative_SQ)"' \ '-DBINDIR="$(bindir_relative_SQ)"' \
'-DPREFIX="$(prefix_SQ)"' \ '-DPREFIX="$(prefix_SQ)"' \
$< $<
builtin-init-db.o: builtin-init-db.c PERF-CFLAGS $(OUTPUT)builtin-init-db.o: builtin-init-db.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o $*.o -c $(ALL_CFLAGS) -DDEFAULT_PERF_TEMPLATE_DIR='"$(template_dir_SQ)"' $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -DDEFAULT_PERF_TEMPLATE_DIR='"$(template_dir_SQ)"' $<
util/config.o: util/config.c PERF-CFLAGS $(OUTPUT)util/config.o: util/config.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o $*.o -c $(ALL_CFLAGS) -DETC_PERFCONFIG='"$(ETC_PERFCONFIG_SQ)"' $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -DETC_PERFCONFIG='"$(ETC_PERFCONFIG_SQ)"' $<
util/rbtree.o: ../../lib/rbtree.c PERF-CFLAGS $(OUTPUT)util/newt.o: util/newt.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o util/rbtree.o -c $(ALL_CFLAGS) -DETC_PERFCONFIG='"$(ETC_PERFCONFIG_SQ)"' $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -DENABLE_SLFUTURE_CONST $<
# some perf warning policies can't fit to lib/bitmap.c, eg: it warns about variable shadowing $(OUTPUT)util/rbtree.o: ../../lib/rbtree.c $(OUTPUT)PERF-CFLAGS
# from <string.h> that comes from kernel headers wrapping. $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) -DETC_PERFCONFIG='"$(ETC_PERFCONFIG_SQ)"' $<
KBITMAP_FLAGS=`echo $(ALL_CFLAGS) | sed s/-Wshadow// | sed s/-Wswitch-default// | sed s/-Wextra//`
util/bitmap.o: ../../lib/bitmap.c PERF-CFLAGS $(OUTPUT)util/scripting-engines/trace-event-perl.o: util/scripting-engines/trace-event-perl.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o util/bitmap.o -c $(KBITMAP_FLAGS) -DETC_PERFCONFIG='"$(ETC_PERFCONFIG_SQ)"' $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) $(PERL_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-shadow $<
util/hweight.o: ../../lib/hweight.c PERF-CFLAGS $(OUTPUT)scripts/perl/Perf-Trace-Util/Context.o: scripts/perl/Perf-Trace-Util/Context.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o util/hweight.o -c $(ALL_CFLAGS) -DETC_PERFCONFIG='"$(ETC_PERFCONFIG_SQ)"' $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) $(PERL_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-nested-externs $<
util/find_next_bit.o: ../../lib/find_next_bit.c PERF-CFLAGS $(OUTPUT)util/scripting-engines/trace-event-python.o: util/scripting-engines/trace-event-python.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o util/find_next_bit.o -c $(ALL_CFLAGS) -DETC_PERFCONFIG='"$(ETC_PERFCONFIG_SQ)"' $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) $(PYTHON_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-shadow $<
util/scripting-engines/trace-event-perl.o: util/scripting-engines/trace-event-perl.c PERF-CFLAGS $(OUTPUT)scripts/python/Perf-Trace-Util/Context.o: scripts/python/Perf-Trace-Util/Context.c $(OUTPUT)PERF-CFLAGS
$(QUIET_CC)$(CC) -o util/scripting-engines/trace-event-perl.o -c $(ALL_CFLAGS) $(PERL_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-shadow $< $(QUIET_CC)$(CC) -o $@ -c $(ALL_CFLAGS) $(PYTHON_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-nested-externs $<
scripts/perl/Perf-Trace-Util/Context.o: scripts/perl/Perf-Trace-Util/Context.c PERF-CFLAGS $(OUTPUT)perf-%$X: %.o $(PERFLIBS)
$(QUIET_CC)$(CC) -o scripts/perl/Perf-Trace-Util/Context.o -c $(ALL_CFLAGS) $(PERL_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-nested-externs $<
util/scripting-engines/trace-event-python.o: util/scripting-engines/trace-event-python.c PERF-CFLAGS
$(QUIET_CC)$(CC) -o util/scripting-engines/trace-event-python.o -c $(ALL_CFLAGS) $(PYTHON_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-shadow $<
scripts/python/Perf-Trace-Util/Context.o: scripts/python/Perf-Trace-Util/Context.c PERF-CFLAGS
$(QUIET_CC)$(CC) -o scripts/python/Perf-Trace-Util/Context.o -c $(ALL_CFLAGS) $(PYTHON_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-nested-externs $<
perf-%$X: %.o $(PERFLIBS)
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS)
$(LIB_OBJS) $(BUILTIN_OBJS): $(LIB_H) $(LIB_OBJS) $(BUILTIN_OBJS): $(LIB_H)
@ -963,17 +1015,17 @@ cscope:
TRACK_CFLAGS = $(subst ','\'',$(ALL_CFLAGS)):\ TRACK_CFLAGS = $(subst ','\'',$(ALL_CFLAGS)):\
$(bindir_SQ):$(perfexecdir_SQ):$(template_dir_SQ):$(prefix_SQ) $(bindir_SQ):$(perfexecdir_SQ):$(template_dir_SQ):$(prefix_SQ)
PERF-CFLAGS: .FORCE-PERF-CFLAGS $(OUTPUT)PERF-CFLAGS: .FORCE-PERF-CFLAGS
@FLAGS='$(TRACK_CFLAGS)'; \ @FLAGS='$(TRACK_CFLAGS)'; \
if test x"$$FLAGS" != x"`cat PERF-CFLAGS 2>/dev/null`" ; then \ if test x"$$FLAGS" != x"`cat $(OUTPUT)PERF-CFLAGS 2>/dev/null`" ; then \
echo 1>&2 " * new build flags or prefix"; \ echo 1>&2 " * new build flags or prefix"; \
echo "$$FLAGS" >PERF-CFLAGS; \ echo "$$FLAGS" >$(OUTPUT)PERF-CFLAGS; \
fi fi
# We need to apply sq twice, once to protect from the shell # We need to apply sq twice, once to protect from the shell
# that runs PERF-BUILD-OPTIONS, and then again to protect it # that runs $(OUTPUT)PERF-BUILD-OPTIONS, and then again to protect it
# and the first level quoting from the shell that runs "echo". # and the first level quoting from the shell that runs "echo".
PERF-BUILD-OPTIONS: .FORCE-PERF-BUILD-OPTIONS $(OUTPUT)PERF-BUILD-OPTIONS: .FORCE-PERF-BUILD-OPTIONS
@echo SHELL_PATH=\''$(subst ','\'',$(SHELL_PATH_SQ))'\' >$@ @echo SHELL_PATH=\''$(subst ','\'',$(SHELL_PATH_SQ))'\' >$@
@echo TAR=\''$(subst ','\'',$(subst ','\'',$(TAR)))'\' >>$@ @echo TAR=\''$(subst ','\'',$(subst ','\'',$(TAR)))'\' >>$@
@echo NO_CURL=\''$(subst ','\'',$(subst ','\'',$(NO_CURL)))'\' >>$@ @echo NO_CURL=\''$(subst ','\'',$(subst ','\'',$(NO_CURL)))'\' >>$@
@ -994,7 +1046,7 @@ all:: $(TEST_PROGRAMS)
export NO_SVN_TESTS export NO_SVN_TESTS
check: common-cmds.h check: $(OUTPUT)common-cmds.h
if sparse; \ if sparse; \
then \ then \
for i in *.c */*.c; \ for i in *.c */*.c; \
@ -1028,10 +1080,10 @@ export perfexec_instdir
install: all install: all
$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(bindir_SQ)' $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(bindir_SQ)'
$(INSTALL) perf$X '$(DESTDIR_SQ)$(bindir_SQ)' $(INSTALL) $(OUTPUT)perf$X '$(DESTDIR_SQ)$(bindir_SQ)'
$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl/Perf-Trace-Util/lib/Perf/Trace' $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl/Perf-Trace-Util/lib/Perf/Trace'
$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl/bin' $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl/bin'
$(INSTALL) perf-archive -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)' $(INSTALL) $(OUTPUT)perf-archive -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)'
$(INSTALL) scripts/perl/Perf-Trace-Util/lib/Perf/Trace/* -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl/Perf-Trace-Util/lib/Perf/Trace' $(INSTALL) scripts/perl/Perf-Trace-Util/lib/Perf/Trace/* -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl/Perf-Trace-Util/lib/Perf/Trace'
$(INSTALL) scripts/perl/*.pl -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl' $(INSTALL) scripts/perl/*.pl -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl'
$(INSTALL) scripts/perl/bin/* -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl/bin' $(INSTALL) scripts/perl/bin/* -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/perl/bin'
@ -1045,7 +1097,7 @@ ifdef BUILT_INS
$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)' $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)'
$(INSTALL) $(BUILT_INS) '$(DESTDIR_SQ)$(perfexec_instdir_SQ)' $(INSTALL) $(BUILT_INS) '$(DESTDIR_SQ)$(perfexec_instdir_SQ)'
ifneq (,$X) ifneq (,$X)
$(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_PROGRAMS) $(BUILT_INS) perf$X)), $(RM) '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$p';) $(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_PROGRAMS) $(BUILT_INS) $(OUTPUT)perf$X)), $(RM) '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/$p';)
endif endif
endif endif
@ -1129,14 +1181,14 @@ clean:
$(RM) *.o */*.o */*/*.o */*/*/*.o $(LIB_FILE) $(RM) *.o */*.o */*/*.o */*/*/*.o $(LIB_FILE)
$(RM) $(ALL_PROGRAMS) $(BUILT_INS) perf$X $(RM) $(ALL_PROGRAMS) $(BUILT_INS) perf$X
$(RM) $(TEST_PROGRAMS) $(RM) $(TEST_PROGRAMS)
$(RM) *.spec *.pyc *.pyo */*.pyc */*.pyo common-cmds.h TAGS tags cscope* $(RM) *.spec *.pyc *.pyo */*.pyc */*.pyo $(OUTPUT)common-cmds.h TAGS tags cscope*
$(RM) -r autom4te.cache $(RM) -r autom4te.cache
$(RM) config.log config.mak.autogen config.mak.append config.status config.cache $(RM) config.log config.mak.autogen config.mak.append config.status config.cache
$(RM) -r $(PERF_TARNAME) .doc-tmp-dir $(RM) -r $(PERF_TARNAME) .doc-tmp-dir
$(RM) $(PERF_TARNAME).tar.gz perf-core_$(PERF_VERSION)-*.tar.gz $(RM) $(PERF_TARNAME).tar.gz perf-core_$(PERF_VERSION)-*.tar.gz
$(RM) $(htmldocs).tar.gz $(manpages).tar.gz $(RM) $(htmldocs).tar.gz $(manpages).tar.gz
$(MAKE) -C Documentation/ clean $(MAKE) -C Documentation/ clean
$(RM) PERF-VERSION-FILE PERF-CFLAGS PERF-BUILD-OPTIONS $(RM) $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)PERF-CFLAGS $(OUTPUT)PERF-BUILD-OPTIONS
.PHONY: all install clean strip .PHONY: all install clean strip
.PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell

4
tools/perf/arch/powerpc/Makefile Normal file
View file

@ -0,0 +1,4 @@
ifndef NO_DWARF
PERF_HAVE_DWARF_REGS := 1
LIB_OBJS += $(OUTPUT)arch/$(ARCH)/util/dwarf-regs.o
endif

88
tools/perf/arch/powerpc/util/dwarf-regs.c Normal file
View file

@ -0,0 +1,88 @@
/*
* Mapping of DWARF debug register numbers into register names.
*
* Copyright (C) 2010 Ian Munsie, IBM Corporation.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
#include <libio.h>
#include <dwarf-regs.h>
struct pt_regs_dwarfnum {
const char *name;
unsigned int dwarfnum;
};
#define STR(s) #s
#define REG_DWARFNUM_NAME(r, num) {.name = r, .dwarfnum = num}
#define GPR_DWARFNUM_NAME(num) \
{.name = STR(%gpr##num), .dwarfnum = num}
#define REG_DWARFNUM_END {.name = NULL, .dwarfnum = 0}
/*
* Reference:
* http://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi-1.9.html
*/
static const struct pt_regs_dwarfnum regdwarfnum_table[] = {
GPR_DWARFNUM_NAME(0),
GPR_DWARFNUM_NAME(1),
GPR_DWARFNUM_NAME(2),
GPR_DWARFNUM_NAME(3),
GPR_DWARFNUM_NAME(4),
GPR_DWARFNUM_NAME(5),
GPR_DWARFNUM_NAME(6),
GPR_DWARFNUM_NAME(7),
GPR_DWARFNUM_NAME(8),
GPR_DWARFNUM_NAME(9),
GPR_DWARFNUM_NAME(10),
GPR_DWARFNUM_NAME(11),
GPR_DWARFNUM_NAME(12),
GPR_DWARFNUM_NAME(13),
GPR_DWARFNUM_NAME(14),
GPR_DWARFNUM_NAME(15),
GPR_DWARFNUM_NAME(16),
GPR_DWARFNUM_NAME(17),
GPR_DWARFNUM_NAME(18),
GPR_DWARFNUM_NAME(19),
GPR_DWARFNUM_NAME(20),
GPR_DWARFNUM_NAME(21),
GPR_DWARFNUM_NAME(22),
GPR_DWARFNUM_NAME(23),
GPR_DWARFNUM_NAME(24),
GPR_DWARFNUM_NAME(25),
GPR_DWARFNUM_NAME(26),
GPR_DWARFNUM_NAME(27),
GPR_DWARFNUM_NAME(28),
GPR_DWARFNUM_NAME(29),
GPR_DWARFNUM_NAME(30),
GPR_DWARFNUM_NAME(31),
REG_DWARFNUM_NAME("%msr", 66),
REG_DWARFNUM_NAME("%ctr", 109),
REG_DWARFNUM_NAME("%link", 108),
REG_DWARFNUM_NAME("%xer", 101),
REG_DWARFNUM_NAME("%dar", 119),
REG_DWARFNUM_NAME("%dsisr", 118),
REG_DWARFNUM_END,
};
/**
* get_arch_regstr() - lookup register name from it's DWARF register number
* @n: the DWARF register number
*
* get_arch_regstr() returns the name of the register in struct
* regdwarfnum_table from it's DWARF register number. If the register is not
* found in the table, this returns NULL;
*/
const char *get_arch_regstr(unsigned int n)
{
const struct pt_regs_dwarfnum *roff;
for (roff = regdwarfnum_table; roff->name != NULL; roff++)
if (roff->dwarfnum == n)
return roff->name;
return NULL;
}

4
tools/perf/arch/x86/Makefile Normal file
View file

@ -0,0 +1,4 @@
ifndef NO_DWARF
PERF_HAVE_DWARF_REGS := 1
LIB_OBJS += $(OUTPUT)arch/$(ARCH)/util/dwarf-regs.o
endif

75
tools/perf/arch/x86/util/dwarf-regs.c Normal file
View file

@ -0,0 +1,75 @@
/*
* dwarf-regs.c : Mapping of DWARF debug register numbers into register names.
* Extracted from probe-finder.c
*
* Written by Masami Hiramatsu <mhiramat@redhat.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*
*/
#include <libio.h>
#include <dwarf-regs.h>
/*
* Generic dwarf analysis helpers
*/
#define X86_32_MAX_REGS 8
const char *x86_32_regs_table[X86_32_MAX_REGS] = {
"%ax",
"%cx",
"%dx",
"%bx",
"$stack", /* Stack address instead of %sp */
"%bp",
"%si",
"%di",
};
#define X86_64_MAX_REGS 16
const char *x86_64_regs_table[X86_64_MAX_REGS] = {
"%ax",
"%dx",
"%cx",
"%bx",
"%si",
"%di",
"%bp",
"%sp",
"%r8",
"%r9",
"%r10",
"%r11",
"%r12",
"%r13",
"%r14",
"%r15",
};
/* TODO: switching by dwarf address size */
#ifdef __x86_64__
#define ARCH_MAX_REGS X86_64_MAX_REGS
#define arch_regs_table x86_64_regs_table
#else
#define ARCH_MAX_REGS X86_32_MAX_REGS
#define arch_regs_table x86_32_regs_table
#endif
/* Return architecture dependent register string (for kprobe-tracer) */
const char *get_arch_regstr(unsigned int n)
{
return (n <= ARCH_MAX_REGS) ? arch_regs_table[n] : NULL;
}

3
tools/perf/bench/mem-memcpy.c
View file

@ -10,7 +10,6 @@
#include "../perf.h" #include "../perf.h"
#include "../util/util.h" #include "../util/util.h"
#include "../util/parse-options.h" #include "../util/parse-options.h"
#include "../util/string.h"
#include "../util/header.h" #include "../util/header.h"
#include "bench.h" #include "bench.h"
@ -24,7 +23,7 @@
static const char *length_str = "1MB"; static const char *length_str = "1MB";
static const char *routine = "default"; static const char *routine = "default";
static int use_clock = 0; static bool use_clock = false;
static int clock_fd; static int clock_fd;
static const struct option options[] = { static const struct option options[] = {

10
tools/perf/bench/sched-messaging.c
View file

@ -31,9 +31,9 @@
#define DATASIZE 100 #define DATASIZE 100
static int use_pipes = 0; static bool use_pipes = false;
static unsigned int loops = 100; static unsigned int loops = 100;
static unsigned int thread_mode = 0; static bool thread_mode = false;
static unsigned int num_groups = 10; static unsigned int num_groups = 10;
struct sender_context { struct sender_context {
@ -256,10 +256,8 @@ static const struct option options[] = {
"Use pipe() instead of socketpair()"), "Use pipe() instead of socketpair()"),
OPT_BOOLEAN('t', "thread", &thread_mode, OPT_BOOLEAN('t', "thread", &thread_mode,
"Be multi thread instead of multi process"), "Be multi thread instead of multi process"),
OPT_INTEGER('g', "group", &num_groups, OPT_UINTEGER('g', "group", &num_groups, "Specify number of groups"),
"Specify number of groups"), OPT_UINTEGER('l', "loop", &loops, "Specify number of loops"),
OPT_INTEGER('l', "loop", &loops,
"Specify number of loops"),
OPT_END() OPT_END()
}; };

2
tools/perf/bench/sched-pipe.c
View file

@ -93,7 +93,7 @@ int bench_sched_pipe(int argc, const char **argv,
switch (bench_format) { switch (bench_format) {
case BENCH_FORMAT_DEFAULT: case BENCH_FORMAT_DEFAULT:
printf("# Extecuted %d pipe operations between two tasks\n\n", printf("# Executed %d pipe operations between two tasks\n\n",
loops); loops);
result_usec = diff.tv_sec * 1000000; result_usec = diff.tv_sec * 1000000;

245
tools/perf/builtin-annotate.c
View file

@ -14,7 +14,6 @@
#include "util/cache.h" #include "util/cache.h"
#include <linux/rbtree.h> #include <linux/rbtree.h>
#include "util/symbol.h" #include "util/symbol.h"
#include "util/string.h"
#include "perf.h" #include "perf.h"
#include "util/debug.h" #include "util/debug.h"
@ -29,80 +28,16 @@
static char const *input_name = "perf.data"; static char const *input_name = "perf.data";
static int force; static bool force;
static int full_paths; static bool full_paths;
static int print_line; static bool print_line;
struct sym_hist {
u64 sum;
u64 ip[0];
};
struct sym_ext {
struct rb_node node;
double percent;
char *path;
};
struct sym_priv {
struct sym_hist *hist;
struct sym_ext *ext;
};
static const char *sym_hist_filter; static const char *sym_hist_filter;
static int sym__alloc_hist(struct symbol *self) static int hists__add_entry(struct hists *self, struct addr_location *al)
{ {
struct sym_priv *priv = symbol__priv(self);
const int size = (sizeof(*priv->hist) +
(self->end - self->start) * sizeof(u64));
priv->hist = zalloc(size);
return priv->hist == NULL ? -1 : 0;
}
/*
* collect histogram counts
*/
static int annotate__hist_hit(struct hist_entry *he, u64 ip)
{
unsigned int sym_size, offset;
struct symbol *sym = he->sym;
struct sym_priv *priv;
struct sym_hist *h;
he->count++;
if (!sym || !he->map)
return 0;
priv = symbol__priv(sym);
if (priv->hist == NULL && sym__alloc_hist(sym) < 0)
return -ENOMEM;
sym_size = sym->end - sym->start;
offset = ip - sym->start;
pr_debug3("%s: ip=%#Lx\n", __func__, he->map->unmap_ip(he->map, ip));
if (offset >= sym_size)
return 0;
h = priv->hist;
h->sum++;
h->ip[offset]++;
pr_debug3("%#Lx %s: count++ [ip: %#Lx, %#Lx] => %Ld\n", he->sym->start,
he->sym->name, ip, ip - he->sym->start, h->ip[offset]);
return 0;
}
static int perf_session__add_hist_entry(struct perf_session *self,
struct addr_location *al, u64 count)
{
bool hit;
struct hist_entry *he; struct hist_entry *he;
if (sym_hist_filter != NULL && if (sym_hist_filter != NULL &&
@ -116,11 +51,11 @@ static int perf_session__add_hist_entry(struct perf_session *self,
return 0; return 0;
} }
he = __perf_session__add_hist_entry(&self->hists, al, NULL, count, &hit); he = __hists__add_entry(self, al, NULL, 1);
if (he == NULL) if (he == NULL)
return -ENOMEM; return -ENOMEM;
return annotate__hist_hit(he, al->addr); return hist_entry__inc_addr_samples(he, al->addr);
} }
static int process_sample_event(event_t *event, struct perf_session *session) static int process_sample_event(event_t *event, struct perf_session *session)
@ -136,7 +71,7 @@ static int process_sample_event(event_t *event, struct perf_session *session)
return -1; return -1;
} }
if (!al.filtered && perf_session__add_hist_entry(session, &al, 1)) { if (!al.filtered && hists__add_entry(&session->hists, &al)) {
pr_warning("problem incrementing symbol count, " pr_warning("problem incrementing symbol count, "
"skipping event\n"); "skipping event\n");
return -1; return -1;
@ -145,106 +80,11 @@ static int process_sample_event(event_t *event, struct perf_session *session)
return 0; return 0;
} }
struct objdump_line {
struct list_head node;
s64 offset;
char *line;
};
static struct objdump_line *objdump_line__new(s64 offset, char *line)
{
struct objdump_line *self = malloc(sizeof(*self));
if (self != NULL) {
self->offset = offset;
self->line = line;
}
return self;
}
static void objdump_line__free(struct objdump_line *self)
{
free(self->line);
free(self);
}
static void objdump__add_line(struct list_head *head, struct objdump_line *line)
{
list_add_tail(&line->node, head);
}
static struct objdump_line *objdump__get_next_ip_line(struct list_head *head,
struct objdump_line *pos)
{
list_for_each_entry_continue(pos, head, node)
if (pos->offset >= 0)
return pos;
return NULL;
}
static int parse_line(FILE *file, struct hist_entry *he,
struct list_head *head)
{
struct symbol *sym = he->sym;
struct objdump_line *objdump_line;
char *line = NULL, *tmp, *tmp2;
size_t line_len;
s64 line_ip, offset = -1;
char *c;
if (getline(&line, &line_len, file) < 0)
return -1;
if (!line)
return -1;
c = strchr(line, '\n');
if (c)
*c = 0;
line_ip = -1;
/*
* Strip leading spaces:
*/
tmp = line;
while (*tmp) {
if (*tmp != ' ')
break;
tmp++;
}
if (*tmp) {
/*
* Parse hexa addresses followed by ':'
*/
line_ip = strtoull(tmp, &tmp2, 16);
if (*tmp2 != ':')
line_ip = -1;
}
if (line_ip != -1) {
u64 start = map__rip_2objdump(he->map, sym->start);
offset = line_ip - start;
}
objdump_line = objdump_line__new(offset, line);
if (objdump_line == NULL) {
free(line);
return -1;
}
objdump__add_line(head, objdump_line);
return 0;
}
static int objdump_line__print(struct objdump_line *self, static int objdump_line__print(struct objdump_line *self,
struct list_head *head, struct list_head *head,
struct hist_entry *he, u64 len) struct hist_entry *he, u64 len)
{ {
struct symbol *sym = he->sym; struct symbol *sym = he->ms.sym;
static const char *prev_line; static const char *prev_line;
static const char *prev_color; static const char *prev_color;
@ -327,7 +167,7 @@ static void insert_source_line(struct sym_ext *sym_ext)
static void free_source_line(struct hist_entry *he, int len) static void free_source_line(struct hist_entry *he, int len)
{ {
struct sym_priv *priv = symbol__priv(he->sym); struct sym_priv *priv = symbol__priv(he->ms.sym);
struct sym_ext *sym_ext = priv->ext; struct sym_ext *sym_ext = priv->ext;
int i; int i;
@ -346,7 +186,7 @@ static void free_source_line(struct hist_entry *he, int len)
static void static void
get_source_line(struct hist_entry *he, int len, const char *filename) get_source_line(struct hist_entry *he, int len, const char *filename)
{ {
struct symbol *sym = he->sym; struct symbol *sym = he->ms.sym;
u64 start; u64 start;
int i; int i;
char cmd[PATH_MAX * 2]; char cmd[PATH_MAX * 2];
@ -361,7 +201,7 @@ get_source_line(struct hist_entry *he, int len, const char *filename)
if (!priv->ext) if (!priv->ext)
return; return;
start = he->map->unmap_ip(he->map, sym->start); start = he->ms.map->unmap_ip(he->ms.map, sym->start);
for (i = 0; i < len; i++) { for (i = 0; i < len; i++) {
char *path = NULL; char *path = NULL;
@ -425,7 +265,7 @@ static void print_summary(const char *filename)
static void hist_entry__print_hits(struct hist_entry *self) static void hist_entry__print_hits(struct hist_entry *self)
{ {
struct symbol *sym = self->sym; struct symbol *sym = self->ms.sym;
struct sym_priv *priv = symbol__priv(sym); struct sym_priv *priv = symbol__priv(sym);
struct sym_hist *h = priv->hist; struct sym_hist *h = priv->hist;
u64 len = sym->end - sym->start, offset; u64 len = sym->end - sym->start, offset;
@ -439,23 +279,17 @@ static void hist_entry__print_hits(struct hist_entry *self)
static void annotate_sym(struct hist_entry *he) static void annotate_sym(struct hist_entry *he)
{ {
struct map *map = he->map; struct map *map = he->ms.map;
struct dso *dso = map->dso; struct dso *dso = map->dso;
struct symbol *sym = he->sym; struct symbol *sym = he->ms.sym;
const char *filename = dso->long_name, *d_filename; const char *filename = dso->long_name, *d_filename;
u64 len; u64 len;
char command[PATH_MAX*2];
FILE *file;
LIST_HEAD(head); LIST_HEAD(head);
struct objdump_line *pos, *n; struct objdump_line *pos, *n;
if (!filename) if (hist_entry__annotate(he, &head) < 0)
return; return;
pr_debug("%s: filename=%s, sym=%s, start=%#Lx, end=%#Lx\n", __func__,
filename, sym->name, map->unmap_ip(map, sym->start),
map->unmap_ip(map, sym->end));
if (full_paths) if (full_paths)
d_filename = filename; d_filename = filename;
else else
@ -472,29 +306,6 @@ static void annotate_sym(struct hist_entry *he)
printf(" Percent | Source code & Disassembly of %s\n", d_filename); printf(" Percent | Source code & Disassembly of %s\n", d_filename);
printf("------------------------------------------------\n"); printf("------------------------------------------------\n");
if (verbose >= 2)
printf("annotating [%p] %30s : [%p] %30s\n",
dso, dso->long_name, sym, sym->name);
sprintf(command, "objdump --start-address=0x%016Lx --stop-address=0x%016Lx -dS %s|grep -v %s",
map__rip_2objdump(map, sym->start),
map__rip_2objdump(map, sym->end),
filename, filename);
if (verbose >= 3)
printf("doing: %s\n", command);
file = popen(command, "r");
if (!file)
return;
while (!feof(file)) {
if (parse_line(file, he, &head) < 0)
break;
}
pclose(file);
if (verbose) if (verbose)
hist_entry__print_hits(he); hist_entry__print_hits(he);
@ -508,25 +319,25 @@ static void annotate_sym(struct hist_entry *he)
free_source_line(he, len); free_source_line(he, len);
} }
static void perf_session__find_annotations(struct perf_session *self) static void hists__find_annotations(struct hists *self)
{ {
struct rb_node *nd; struct rb_node *nd;
for (nd = rb_first(&self->hists); nd; nd = rb_next(nd)) { for (nd = rb_first(&self->entries); nd; nd = rb_next(nd)) {
struct hist_entry *he = rb_entry(nd, struct hist_entry, rb_node); struct hist_entry *he = rb_entry(nd, struct hist_entry, rb_node);
struct sym_priv *priv; struct sym_priv *priv;
if (he->sym == NULL) if (he->ms.sym == NULL)
continue; continue;
priv = symbol__priv(he->sym); priv = symbol__priv(he->ms.sym);
if (priv->hist == NULL) if (priv->hist == NULL)
continue; continue;
annotate_sym(he); annotate_sym(he);
/* /*
* Since we have a hist_entry per IP for the same symbol, free * Since we have a hist_entry per IP for the same symbol, free
* he->sym->hist to signal we already processed this symbol. * he->ms.sym->hist to signal we already processed this symbol.
*/ */
free(priv->hist); free(priv->hist);
priv->hist = NULL; priv->hist = NULL;
@ -545,7 +356,7 @@ static int __cmd_annotate(void)
int ret; int ret;
struct perf_session *session; struct perf_session *session;
session = perf_session__new(input_name, O_RDONLY, force); session = perf_session__new(input_name, O_RDONLY, force, false);
if (session == NULL) if (session == NULL)
return -ENOMEM; return -ENOMEM;
@ -554,7 +365,7 @@ static int __cmd_annotate(void)
goto out_delete; goto out_delete;
if (dump_trace) { if (dump_trace) {
event__print_totals(); perf_session__fprintf_nr_events(session, stdout);
goto out_delete; goto out_delete;
} }
@ -562,11 +373,11 @@ static int __cmd_annotate(void)
perf_session__fprintf(session, stdout); perf_session__fprintf(session, stdout);
if (verbose > 2) if (verbose > 2)
dsos__fprintf(stdout); perf_session__fprintf_dsos(session, stdout);
perf_session__collapse_resort(&session->hists); hists__collapse_resort(&session->hists);
perf_session__output_resort(&session->hists, session->event_total[0]); hists__output_resort(&session->hists);
perf_session__find_annotations(session); hists__find_annotations(&session->hists);
out_delete: out_delete:
perf_session__delete(session); perf_session__delete(session);
@ -581,10 +392,12 @@ static const char * const annotate_usage[] = {
static const struct option options[] = { static const struct option options[] = {
OPT_STRING('i', "input", &input_name, "file", OPT_STRING('i', "input", &input_name, "file",
"input file name"), "input file name"),
OPT_STRING('d', "dsos", &symbol_conf.dso_list_str, "dso[,dso...]",
"only consider symbols in these dsos"),
OPT_STRING('s', "symbol", &sym_hist_filter, "symbol", OPT_STRING('s', "symbol", &sym_hist_filter, "symbol",
"symbol to annotate"), "symbol to annotate"),
OPT_BOOLEAN('f', "force", &force, "don't complain, do it"), OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
OPT_BOOLEAN('v', "verbose", &verbose, OPT_INCR('v', "verbose", &verbose,
"be more verbose (show symbol address, etc)"), "be more verbose (show symbol address, etc)"),
OPT_BOOLEAN('D', "dump-raw-trace", &dump_trace, OPT_BOOLEAN('D', "dump-raw-trace", &dump_trace,
"dump raw trace in ASCII"), "dump raw trace in ASCII"),

Some files were not shown because too many files have changed in this diff Show more