The Digger: additional notes. DTrace output can be shufled in multi-CPU environment.
Below are important additions about the Digger tool.
1. DTrace output can be shufled in multi-CPU environment.
This means that output can be in not chronological order. It is not something Digger specific, it is how DTrace works.
When DTrace script is being executed there are two parts to DTrace: the in kernel part and the DTrace script process in userland. When a dtrace probe fires, the data traced is placed in a per CPU buffer. Then, periodically, the DTrace script reads the buffers (in round-robin style) and continues processing the data for final output. While the data from any single CPU was entered into its buffer in order the probes are firing asynchronously with respect to all the CPUs.
If probe1 fired on CPU3 and after that probe2 fired on CPU2, and after that probe3 fired on CPU1 it is possible to get output as, for example, as following:
The most common case where we can see this are the cases when a process(thread) migrated from one CPU to another and DTrace consumer reads and prints current CPU-buffer before previous. It does not happen always but sometimes you can see it.
So, do not panic if some part of the output is not chronological in multi-CPU environment. It is expected.
If the output order is important, then it is strongly recommended to print such columns as timestamp or relative timestamp in order to be sure that the output is chronological, otherwise to fix it manually or using tools like sort (1). Column CPU# also is helpful because allows to see places where a process(thread) migrated from one CPU to another.
2. How much is performance impact of the Digger?
It depends on the usage. When you trace everything – all application functions or all library calls – performance impact can be significant and amount of printed info can be huge.
The Digger allows to restrict
a) traced area – where tracing is enabled
b) traced contents – which functions are traced
For example, in order to trace kcb* functions and all syscalls *read* inside function qertbFetch
command line should be:
digger.sh -p PID -Fcdeo -f qertbFetch -t kcb* -s *read*
Then generated dtrace script will be containing only required probes:
pid$target:a.out:qertbFetch:entry pid$target:a.out:qertbFetch:return + pid$target:a.out:kcb*:entry pid$target:a.out:kcb*:return + syscall::*read*:entry syscall::*read*:entry
Note: “a.out” is synonym of executing binary.
It helps to trace only relevant information, only required functions. It decreases performance impact and decreases probability of drops.
So, if you need to decrease impact of the tracing just try to restrict traced area and contents.
3. Important bug is fixed.
There was an issue related with replacement tab symbols on spaces during generating html wordpress page with source code (thanks Zhenx Li for let me know). The issue led to possible error “syntax error at line n: `end of file’ unexpected” when the source of the tool was copied/pasted. Now links on html pages with source are replaced on direct download links.