OProfile manual

Older versions of OProfile were not capable of attributing samples to symbols from dynamically compiled code, i.e. "just-in-time (JIT) code". Typical JIT compilers load the JIT code into anonymous memory regions. OProfile reported the samples from such code, but the attribution provided was simply:

     anon: <tgid><address range>

Due to this limitation, it wasn't possible to profile applications executed by virtual machines (VMs) like the Java Virtual Machine. OProfile now contains an infrastructure to support JITed code. A development library is provided to allow developers to add support for any VM that produces dynamically compiled code (see the OProfile JIT agent developer guide). In addition, built-in support is included for the following:

For information on how to use OProfile's JIT support, see Section 3, “Setting up the JIT profiling feature”.

OProfile currently does not support event-based profiling (i.e, using hardware events like cache misses, branch mispredicts) on virtual machine guests running under systems such as VMware. The list of supported events displayed by ophelp or 'opcontrol --list-events' is based on CPU type and does not take into account whether the running system is a guest system or real system. To use OProfile on such guest systems, you can use timer mode (see Section 5.2, “OProfile in timer interrupt mode”).

Add this to the startup parameters of the JVM (for JVMTI):

-agentpath:<libdir>/libjvmti_oprofile.so[=<options>] 

or

-agentlib:jvmti_oprofile[=<options>] 

The JVMPI agent implementation is enabled with the command line option

-Xrunjvmpi_oprofile[:<options>] 

Currently, there is just one option available -- debug. For JVMPI, the convention for specifying an option is option_name=[yes|no]. For JVMTI, the option specification is simply the option name, implying "yes"; no option specified implies "no".

The agent library (installed in <oprof_install_dir>/lib/oprofile) needs to be in the library search path (e.g. add the library directory to LD_LIBRARY_PATH). If the command line of the JVM is not accessible, it may be buried within shell scripts or a launcher program. It may also be possible to set an environment variable to add the instrumentation. For Sun JVMs this is JAVA_TOOL_OPTIONS. Please check your JVM documentation for further information on the agent startup options.

Most processor models include performance monitor units that can be configured to monitor (count) various types of hardware events. This section is where you can find architecture-specific information to help you use these events for profiling. You do not really need to read this section unless you are interested in using events other than the default event chosen by OProfile.

The Intel hardware performance counters are detailed in the Intel IA-32 Architecture Manual, Volume 3, available from http://developer.intel.com/. The AMD Athlon/Opteron/Phenom/Turion implementation is detailed in http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/22007.pdf. For IBM PowerPC processors, documentation is available at https://www.power.org/. For example, https://www.power.org/events/Power7 contains specific information on the performance monitor unit for the IBM POWER7.

These processors are capable of delivering an interrupt when a counter overflows. This is the basic mechanism on which OProfile is based. The delivery mode is NMI, so blocking interrupts in the kernel does not prevent profiling. When the interrupt handler is called, the current PC value and the current task are recorded into the profiling structure. This allows the overflow event to be attached to a specific assembly instruction in a binary image. OProfile receives this data from the kernel and writes it to the sample files.

If we use an event such as CPU_CLK_UNHALTED or INST_RETIRED (GLOBAL_POWER_EVENTS or INSTR_RETIRED, respectively, on the Pentium 4), we can use the overflow counts as an estimate of actual time spent in each part of code. Alternatively we can profile interesting data such as the cache behaviour of routines with the other available counters.

However there are several caveats. First, there are those issues listed in the Intel manual. There is a delay between the counter overflow and the interrupt delivery that can skew results on a small scale - this means you cannot rely on the profiles at the instruction level as being perfectly accurate. If you are using an "event-mode" counter such as the cache counters, a count registered against it doesn't mean that it is responsible for that event. However, it implies that the counter overflowed in the dynamic vicinity of that instruction, to within a few instructions. Further details on this problem can be found in Chapter 5, Interpreting profiling results and also in the Digital paper "ProfileMe: A Hardware Performance Counter".

Each counter has several configuration parameters. First, there is the unit mask: this simply further specifies what to count. Second, there is the counter value, discussed below. Third, there is a parameter whether to increment counts whilst in kernel or user space. You can configure these separately for each counter.

After each overflow event, the counter will be re-initialized such that another overflow will occur after this many events have been counted. Thus, higher values mean less-detailed profiling, and lower values mean more detail, but higher overhead. Picking a good value for this parameter is, unfortunately, somewhat of a black art. It is of course dependent on the event you have chosen. Specifying too large a value will mean not enough interrupts are generated to give a realistic profile (though this problem can be ameliorated by profiling for longer). Specifying too small a value can lead to higher performance overhead.

Some CPU types do not provide the needed hardware support to use the hardware performance counters. This includes some laptops, classic Pentiums, and other CPU types not yet supported by OProfile (such as Cyrix). On these machines, OProfile falls back to using the timer interrupt for profiling, back to using the real-time clock interrupt to collect samples. In timer mode, OProfile is not able to profile code that has interrupts disabled.

You can force use of the timer interrupt by using the timer=1 module parameter (or oprofile.timer=1 on the boot command line if OProfile is built-in). If OProfile was built as a kernel module, then you must pass the 'timer=1' parameter with the modprobe command. Do this before executing 'opcontrol --init' or edit the opcontrol command's invocation of modprobe to pass the 'timer=1' parameter.

The Pentium 4 / Xeon performance counters are organized around 3 types of model specific registers (MSRs): 45 event selection control registers (ESCRs), 18 counter configuration control registers (CCCRs) and 18 counters. ESCRs describe a particular set of events which are to be recorded, and CCCRs bind ESCRs to counters and configure their operation. Unfortunately the relationship between these registers is quite complex; they cannot all be used with one another at any time. There is, however, a subset of 8 counters, 8 ESCRs, and 8 CCCRs which can be used independently of one another, so OProfile only accesses those registers, treating them as a bank of 8 "normal" counters, similar to those in the P6 or Athlon/Opteron/Phenom/Turion families of CPU.

There is currently no support for Precision Event-Based Sampling (PEBS), nor any advanced uses of the Debug Store (DS). Current support is limited to the conservative extension of OProfile's existing interrupt-based model described above.

The Itanium 2 performance monitoring unit (PMU) organizes the counters as four pairs of performance event monitoring registers. Each pair is composed of a Performance Monitoring Configuration (PMC) register and Performance Monitoring Data (PMD) register. The PMC selects the performance event being monitored and the PMD determines the sampling interval. The IA64 Performance Monitoring Unit (PMU) triggers sampling with maskable interrupts. Thus, samples will not occur in sections of the IA64 kernel where interrupts are disabled.

None of the advance features of the Itanium 2 performance monitoring unit such as opcode matching, address range matching, or precise event sampling are supported by this version of OProfile. The Itanium 2 support only maps OProfile's existing interrupt-based model to the PMU hardware.

The performance monitoring unit (PMU) for the IBM PowerPC 64-bit processors consists of between 4 and 8 counters (depending on the model), plus three special purpose registers used for programming the counters -- MMCR0, MMCR1, and MMCRA. Advanced features such as instruction matching and thresholding are not supported by this version of OProfile.

The Cell Broadband Engine (CBE) processor core consists of a PowerPC Processing Element (PPE) and 8 Synergistic Processing Elements (SPE). PPEs and SPEs each consist of a processing unit (PPU and SPU, respectively) and other hardware components, such as memory controllers.

A PPU has two hardware threads (aka "virtual CPUs"). The performance monitor unit of the CBE collects event information on one hardware thread at a time. Therefore, when profiling PPE events, OProfile collects the profile based on the selected events by time slicing the performance counter hardware between the two threads. The user must ensure the collection interval is long enough so that the time spent collecting data for each PPU is sufficient to obtain a good profile.

To profile an SPU application, the user should specify the SPU_CYCLES event. When starting OProfile with SPU_CYCLES, the opcontrol script enforces certain separation parameters (separate=cpu,lib) to ensure that sufficient information is collected in the sample data in order to generate a complete report. The --merge=cpu option can be used to obtain a more readable report if analyzing the performance of each separate SPU is not necessary.

Profiling with an SPU event (events 4100 through 4163) is not compatible with any other event. Further more, only one SPU event can be specified at a time. The hardware only supports profiling on one SPU per node at a time. The OProfile kernel code time slices between the eight SPUs to collect data on all SPUs.

SPU profile reports have some unique characteristics compared to reports for standard architectures:

Instruction-Based Sampling (IBS) is a new performance measurement technique available on AMD Family 10h processors. Traditional performance counter sampling is not precise enough to isolate performance issues to individual instructions. IBS, however, precisely identifies instructions which are not making the best use of the processor pipeline and memory hierarchy. For more information, please refer to the "Instruction-Based Sampling: A New Performance Analysis Technique for AMD Family 10h Processors" ( http://developer.amd.com/assets/AMD_IBS_paper_EN.pdf). There are two types of IBS profile types, described in the following sections.

Enabling IBS profiling is done simply by specifying IBS performance events through the "--event=" options. These events are listed in the opcontrol --list-events.

opcontrol --event=IBS_FETCH_XXX:<count>:<um>:<kernel>:<user>
opcontrol --event=IBS_OP_XXX:<count>:<um>:<kernel>:<user>

Note: * All IBS fetch event must have the same event count and unitmask,
        as do those for IBS op.

IBM System z provides a facility which does instruction sampling as part of the CPU. This has great advantages over the timer based sampling approach like better sampling resolution with less overhead and the possibility to get samples within code sections where interrupts are disabled (useful especially for Linux kernel code).

A public description of the System z CPU-Measurement Facilities can be found here: The Load-Program-Parameter and CPU-Measurement Facilities

System z hardware sampling can be used for Linux instances in LPAR mode. The hardware sampling support used by OProfile was introduced for System z10 in October 2008.

To enable hardware sampling for an LPAR you must activate the LPAR with authorization for basic sampling control. See the "Support Element Operations Guide" for your mainframe system for more information.

The hardware sampling facility can be enabled and disabled using the event interface. A `virtual' counter 0 has been defined that only supports a single event, HWSAMPLING. By default the HWSAMPLING event is enabled on machines providing the facility. For both events only the `count', `kernel' and `user' options are evaluated by the kernel module.

The `count' value is the sampling rate as it is passed to the CPU measurement facility. A sample will be taken by the hardware every `count' cycles. Using low values here will quickly fill up the sampling buffers and will generate CPU load on the OProfile daemon and the kernel module being busy flushing the hardware buffers. This might considerably impact the workload to be profiled.

The unit mask `um' is required to be zero.

The opcontrol tool provides a new option specific to System z hardware sampling:

A special counter /dev/oprofile/timer is provided by the kernel module allowing to switch back to timer mode sampling dynamically. The TIMER event is limited to be used only with this counter. The TIMER event can be specified using the --event= as with every other event.

opcontrol --event=TIMER:1

On z10 or later machines the default event is set to TIMER in case the hardware sampling facility is not available.

Although required, the 'count' parameter of the TIMER event is ignored. The value may eventually be used for timer based sampling with a configurable sampling frequency, but this is currently not supported.

OProfile is a low-level profiler which allows continuous profiling with a low-overhead cost. When using OProfile legacy mode profiling, it may be possible to configure such a low a counter reset value (i.e., high sampling rate) that the system can become overloaded with counter interrupts and your system's responsiveness may be severely impacted. Whilst some validation is done on the count values you pass to opcontrol with your event specification, it is not foolproof.

If this happens, it will be impossible to bring the system back to a workable state. There is no way to provide real security against this happening, other than making sure to use a reasonable value for the counter reset. For example, setting CPU_CLK_UNHALTED event type with a ridiculously low reset count (e.g. 500) is likely to freeze the system.

In short : Don't try a foolish sample count value. Unfortunately the definition of a foolish value is really dependent on the event type. If ever in doubt, post a message to

Image summaries for all profiles with DATA_MEM_REFS samples in the saved session called "stresstest" :

# opreport session:stresstest event:DATA_MEM_REFS

Symbol summary for the application called "test_sym53c8xx,9xx". Note the escaping is necessary as image: takes a comma-separated list.

# opreport -l ./test/test_sym53c8xx\,9xx

Image summaries for all binaries in the test directory, excepting boring-test :

# opreport image:./test/\* image-exclude:./test/boring-test

Differential profile of a binary stored in two archives :

# opreport -l /bin/bash { archive:./orig } { archive:./new }

Differential profile of an archived binary with the current session :

# opreport -l /bin/bash { archive:./orig } { }

Each session's sample files can be found in the $SESSION_DIR/samples/ directory (default when using legacy mode: /var/lib/oprofile/samples/; default when using operf: <cur_dir>/oprofile_data/samples/). These are used, along with the binary image files, to produce human-readable data. In some circumstances (e.g., kernel modules), OProfile will not be able to find the binary images. All the tools have an --image-path option to which you can pass a comma-separated list of alternate paths to search. For example, I can let OProfile find my 2.6 modules by using --image-path /lib/modules/2.6.0/kernel/. It is your responsibility to ensure that the correct images are found when using this option.

Note that if a binary image changes after the sample file was created, you won't be able to get useful symbol-based data out. This situation is detected for you. If you replace a binary, you should make sure to save the old binary if you need to do comparative profiles.

When attempting to get output, you may see the error :

error: no sample files found: profile specification too strict ?

What this is saying is that the profile specification you passed in, when matched against the available sample files, resulted in no matches. There are a number of reasons this might happen:

This section provides details on how to use the OProfile callgraph feature.

#include <string.h>
#include <stdlib.h>
#include <stdio.h>

#define SIZE 500000

static int compare(const void *s1, const void *s2)
{
        return strcmp(s1, s2);
}

static void repeat(void)
{
        int i;
        char *strings[SIZE];
        char str[] = "abcdefghijklmnopqrstuvwxyz";

        for (i = 0; i < SIZE; ++i) {
                strings[i] = strdup(str);
                strfry(strings[i]);
        }

        qsort(strings, SIZE, sizeof(char *), compare);
}

int main()
{
        while (1)
                repeat();
}

samples  %        image name               symbol name
  197       0.1548  cg                       main
  127036   99.8452  cg                       repeat
84590    42.5084  libc-2.3.2.so            strfry
  84590    66.4838  libc-2.3.2.so            strfry [self]
  39169    30.7850  libc-2.3.2.so            random_r
  3475      2.7312  libc-2.3.2.so            __i686.get_pc_thunk.bx
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
  1         2.2727  libj9ute23.so            java.bin                 traceV
  2         4.5455  libj9ute23.so            java.bin                 utsTraceV
  4         9.0909  libj9trc23.so            java.bin                 fillInUTInterfaces
  37       84.0909  libj9trc23.so            java.bin                 twGetSequenceCounter
8         0.0154  libj9prt23.so            java.bin                 j9time_hires_clock
  27       61.3636  anon (tgid:10014 range:0x100000-0x103000) java.bin                 (no symbols)
  9        20.4545  libc-2.4.so              java.bin                 gettimeofday
  8        18.1818  libj9prt23.so            java.bin                 j9time_hires_clock [self]
-------------------------------------------------------------------------------

Often, we'd like to be able to compare two profiles. For example, when analysing the performance of an application, we'd like to make code changes and examine the effect of the change. This is supported in opreport by giving a profile specification that identifies two different profiles. The general form is of:

$ opreport <shared-spec> { <first-profile> } { <second-profile> }

For each of the profiles, the shared section is prefixed, and then the specification is analysed. The usual parameters work both within the shared section, and in the sub-specification within the curly braces.

A typical way to use this feature is with archives created with oparchive. Let's look at an example:

$ ./a
$ oparchive -o orig ./a
$ opcontrol --reset
  # edit and recompile a
$ ./a
  # now compare the current profile of a with the archived profile
$ opreport -xl ./a { archive:./orig } { }
CPU: PIII, speed 863.233 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a
unit mask of 0x00 (No unit mask) count 100000
samples  %        diff %    symbol name
92435    48.5366  +0.4999   a
54226    ---      ---       c
49222    25.8459  +++       d
48787    25.6175  -2.2e-01  b

Note that we specified an empty second profile in the curly braces, as we wanted to use the current session; alternatively, we could have specified another archive, or a tgid etc. We specified the binary a in the shared section, so we matched that in both the profiles we're diffing.

As in the normal output, the results are sorted by the number of samples, and the percentage field represents the relative percentage of the symbol's samples in the second profile.

Notice the new column in the output. This value represents the percentage change of the relative percent between the first and the second profile: roughly, "how much more important this symbol is". Looking at the symbol a(), we can see that it took roughly the same amount of the total profile in both the first and the second profile. The function c() was not in the new profile, so has been marked with ---. Note that the sample value is the number of samples in the first profile; since we're displaying results for the second profile, we don't list a percentage value for it, as it would be meaningless. d() is new in the second profile, and consequently marked with +++.

When comparing profiles between different binaries, it should be clear that functions can change in terms of VMA and size. To avoid this problem, opreport considers a symbol to be the same if the symbol name, image name, and owning application name all match; any other factors are ignored. Note that the check for application name means that trying to compare library profiles between two different applications will not work as you might expect: each symbol will be considered different.

Many applications, typically ones involving dynamic compilation into machine code (just-in-time, or "JIT", compilation), have executable mappings that are not backed by an ELF file. opreport has basic support for showing the samples taken in these regions; for example:

$ opreport /usr/bin/mono -l
CPU: ppc64 POWER5, speed 1654.34 MHz (estimated)
Counted CYCLES events (Processor Cycles using continuous sampling) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name    		                symbol name
47       58.7500  mono                     			(no symbols)
14       17.5000  anon (tgid:3189 range:0xf72aa000-0xf72fa000)  (no symbols)
9        11.2500  anon (tgid:3189 range:0xf6cca000-0xf6dd9000)  (no symbols)
.	 .	  .						.

Note that, since such mappings are dependent upon individual invocations of a binary, these mappings are always listed as a dependent image, even when using the legacy mode opcontrol --separate=none command. Equally, the results are not affected by the --merge option.

As shown in the opreport output above, OProfile is unable to attribute the samples to any symbol(s) because there is no ELF file for this code. Enhanced support for JITed code is now available for some virtual machines; e.g., the Java Virtual Machine. For details about OProfile output for JITed code, see Section 4, “OProfile results with JIT samples”.

For more information about JIT support in OProfile, see Section 4.1, “Support for dynamically compiled (JIT) code”.

The --xml option can be used to generate XML instead of the usual text format. This allows opreport to eliminate some of the constraints dictated by the two dimensional text format. For example, it is possible to separate the sample data across multiple events, cpus and threads. The XML schema implemented by opreport is found in doc/opreport.xsd. It contains more detailed comments about the structure of the XML generated by opreport.

Since XML is consumed by a client program rather than a user, its structure is fairly static. In particular, the --sort option is incompatible with the --xml option. Percentages are not dislayed in the XML so the options related to percentages will have no effect. Full pathnames are always displayed in the XML so --long-filenames is not necessary. The --details option will cause all of the individual sample data to be included in the XML as well as the instruction byte stream for each symbol (for doing disassembly) and can result in very large XML files.

Of course, opannotate needs to be able to locate the source files for the binary image(s) in order to produce output. Some binary images have debug information where the given source file paths are relative, not absolute. You can specify search paths to look for these files (similar to gdb's dir command) with the --search-dirs option.

Sometimes you may have a binary image which gives absolute paths for the source files, but you have the actual sources elsewhere (commonly, you've installed an SRPM for a binary on your system and you want annotation from an existing profile). You can use the --base-dirs option to redirect OProfile to look somewhere else for source files. For example, imagine we have a binary generated from a source file that is given in the debug information as /tmp/build/libfoo/foo.c, and you have the source tree matching that binary installed in /home/user/libfoo/. You can redirect OProfile to find foo.c correctly like this :

$ opannotate --source --base-dirs=/tmp/build/libfoo/ --search-dirs=/home/user/libfoo/ --output-dir=annotated/ /lib/libfoo.so

You can specify multiple (comma-separated) paths to both options.

OProfile uses non-maskable interrupts (NMI) on the P6 generation, Pentium 4, Athlon, Opteron, Phenom, and Turion processors. These interrupts can occur even in sections of the kernel where interrupts are disabled, allowing collection of samples in virtually all executable code. The timer interrupt mode and Itanium 2 collection mechanisms use maskable interrupts; therefore, these profiling mechanisms have "sample shadows", or blind spots: regions where no samples will be collected. Typically, the samples will be attributed to the code immediately after the interrupts are re-enabled.

Your kernel is likely to support halting the processor when a CPU is idle. As the typical hardware events like CPU_CLK_UNHALTED do not count when the CPU is halted, the kernel profile will not reflect the actual amount of time spent idle. You can change this behaviour by booting with the idle=poll option, which uses a different idle routine. This will appear as poll_idle() in your kernel profile.

OProfile profiles kernel modules by default. However, there are a couple of problems you may have when trying to get results. First, you may have booted via an initrd; this means that the actual path for the module binaries cannot be determined automatically. To get around this, you can use the -p option to the profiling tools to specify where to look for the kernel modules.

In kernel version 2.6, the information on where kernel module binaries are located was removed. This means OProfile needs guiding with the -p option to find your modules. Normally, you can just use your standard module top-level directory for this. Note that due to this problem, OProfile cannot check that the modification times match; it is your responsibility to make sure you do not modify a binary after a profile has been created.

If you have run insmod or modprobe to insert a module in a particular directory, it is important that you specify this directory with the -p option first, so that it over-rides an older module binary that might exist in other directories you've specified with -p. It is up to you to make sure that these values are correct: the kernel simply does not provide enough information for OProfile to get this information.

The compiler can introduce some pitfalls in the annotated source output. The optimizer can move pieces of code in such manner that two line of codes are interlaced (instruction scheduling). Also debug info generated by the compiler can show strange behavior. This is especially true for complex expressions e.g. inside an if statement:

	if (a && ..
	    b && ..
	    c &&)

here the problem come from the position of line number. The available debug info does not give enough details for the if condition, so all samples are accumulated at the position of the right brace of the expression. Using opannotate -a can help to show the real samples at an assembly level.

The compiler generally needs to generate "glue" code across function calls, dependent on the particular function call conventions used. Additionally other things need to happen, like stack pointer adjustment for the local variables; this code is known as the function prologue. Similar code is needed at function return, and is known as the function epilogue. This will show up in annotations as samples at the very start and end of a function, where there is no apparent executable code in the source.

You may see that a function is credited with a certain number of samples, but the listing does not add up to the correct total. To pick a real example :

               :internal_sk_buff_alloc_security(struct sk_buff *skb)
 353 2.342%    :{ /* internal_sk_buff_alloc_security total: 1882 12.48% */
               :
               :        sk_buff_security_t *sksec;
  15 0.0995%   :        int rc = 0;
               :
  10 0.06633%  :        sksec = skb->lsm_security;
 468 3.104%    :        if (sksec && sksec->magic == DSI_MAGIC) {
               :                goto out;
               :        }
               :
               :        sksec = (sk_buff_security_t *) get_sk_buff_memory(skb);
   3 0.0199%   :        if (!sksec) {
  38 0.2521%   :                rc = -ENOMEM;
               :                goto out;
  10 0.06633%  :        }
               :        memset(sksec, 0, sizeof (sk_buff_security_t));
  44 0.2919%   :        sksec->magic = DSI_MAGIC;
  32 0.2123%   :        sksec->skb = skb;
  45 0.2985%   :        sksec->sid = DSI_SID_NORMAL;
  31 0.2056%   :        skb->lsm_security = sksec;
               :
               :      out:
               :
 146 0.9685%   :        return rc;
               :
  98 0.6501%   :}

Here, the function is credited with 1,882 samples, but the annotations below do not account for this. This is usually because of inline functions - the compiler marks such code with debug entries for the inline function definition, and this is where opannotate annotates such samples. In the case above, memset is the most likely candidate for this problem. Examining the mixed source/assembly output can help identify such results.

This problem is more visible when there is no source file available, in the following example it's trivially visible the sums of symbols samples is less than the number of the samples for this file. The difference must be accounted to inline functions.

/*
 * Total samples for file : "arch/i386/kernel/process.c"
 *
 *    109  2.4616
 */

 /* default_idle total:     84  1.8970 */
 /* cpu_idle total:         21  0.4743 */
 /* flush_thread total:      1  0.0226 */
 /* prepare_to_copy total:   1  0.0226 */
 /* __switch_to total:      18  0.4065 */

The missing samples are not lost, they will be credited to another source location where the inlined function is defined. The inlined function will be credited from multiple call site and merged in one place in the annotated source file so there is no way to see from what call site are coming the samples for an inlined function.

When running opannotate, you may get a warning "some functions compiled without debug information may have incorrect source line attributions". In some rare cases, OProfile is not able to verify that the derived source line is correct (when some parts of the binary image are compiled without debugging information). Be wary of results if this warning appears.

Furthermore, for some languages the compiler can implicitly generate functions, such as default copy constructors. Such functions are labelled by the compiler as having a line number of 0, which means the source annotation can be confusing.

Depending on your compiler you can fall into the following problem:

struct big_object { int a[500]; };

int main()
{
	big_object a, b;
	for (int i = 0 ; i != 1000 * 1000; ++i)
		b = a;
	return 0;
}

Compiled with gcc 3.0.4 the annotated source is clearly inaccurate:

               :int main()
               :{  /* main total: 7871 100% */
               :        big_object a, b;
               :        for (int i = 0 ; i != 1000 * 1000; ++i)
               :                b = a;
 7871 100%     :        return 0;
               :}

The problem here is distinct from the IRQ latency problem; the debug line number information is not precise enough; again, looking at output of opannoatate -as can help.

               :int main()
               :{
               :        big_object a, b;
               :        for (int i = 0 ; i != 1000 * 1000; ++i)
               : 80484c0:       push   %ebp
               : 80484c1:       mov    %esp,%ebp
               : 80484c3:       sub    $0xfac,%esp
               : 80484c9:       push   %edi
               : 80484ca:       push   %esi
               : 80484cb:       push   %ebx
               :                b = a;
               : 80484cc:       lea    0xfffff060(%ebp),%edx
               : 80484d2:       lea    0xfffff830(%ebp),%eax
               : 80484d8:       mov    $0xf423f,%ebx
               : 80484dd:       lea    0x0(%esi),%esi
               :        return 0;
    3 0.03811% : 80484e0:       mov    %edx,%edi
               : 80484e2:       mov    %eax,%esi
    1 0.0127%  : 80484e4:       cld
    8 0.1016%  : 80484e5:       mov    $0x1f4,%ecx
 7850 99.73%   : 80484ea:       repz movsl %ds:(%esi),%es:(%edi)
    9 0.1143%  : 80484ec:       dec    %ebx
               : 80484ed:       jns    80484e0
               : 80484ef:       xor    %eax,%eax
               : 80484f1:       pop    %ebx
               : 80484f2:       pop    %esi
               : 80484f3:       pop    %edi
               : 80484f4:       leave
               : 80484f5:       ret

So here it's clear that copying is correctly credited with of all the samples, but the line number information is misplaced. objdump -dS exposes the same problem. Note that maintaining accurate debug information for compilers when optimizing is difficult, so this problem is not suprising. The problem of debug information accuracy is also dependent on the binutils version used; some BFD library versions contain a work-around for known problems of gcc, some others do not. This is unfortunate but we must live with that, since profiling is pointless when you disable optimisation (which would give better debugging entries).