• If this parameter is set to 0, there are no time calculations in rowsource statistics. The functions qerstSnapStats()/qerstUpdateStats()do not get timestamp
• If this parameter is set to 1, time is calculated always. Every pair qerstSnapStats()/qerstUpdateStats() gets timestamp (this is the case in the excerpts above).
• If this parameter is set to N (as default value 128) then timestamp will be got every N tuples. It means that only N call of qerstSnapStats()/qerstUpdateStats() on some rowsource level will get timestamp.
• There is special value 3 which enables gathering only rowcounts.

Pay attention on the case when _rowsource_statistics_sampfreq = N.
If this parameter is set to N (as default value 128) then timestamp will be got every N tuples. It means that only N call of qerstSnapStats()/qerstUpdateStats() on some rowsource level will get timestamp. Consequenced calls (for example next 127 times) of qerstSnapStats()/qerstUpdateStats() use that calculated time. In this case reported time of some rowsources is not real, but something like forecasted. As a result in this case we can find inconsistent time of some steps of an execution plan, when reported time of child steps more then time of parent steps.

From the other hand the most accurate (_rowsource_statistics_sampfreq = 1) leads to increased number of additional calls of the function getting timestamp, as a result increased time for execution of a query.

These are examples of the Digger’s output of the Digger for the same query

select --+ index(tbl)
       count(pad)
  from tbl;

-----------------------------------------------------
| Id  | Operation                    | Name | Rows  |
-----------------------------------------------------
|   0 | SELECT STATEMENT             |      |     1 |
|   1 |  SORT AGGREGATE              |      |     1 |
|   2 |   TABLE ACCESS BY INDEX ROWID| TBL  |    50 |
|   3 |    INDEX FULL SCAN           | IDX  |    50 |
-----------------------------------------------------

but with different _rowsource_statistics_sampfreq: 0 (timing disabled), 1 (the most accurate timing), 2, 4, 8, 16, 128
special case: _rowsource_statistics_sampfreq = 3 (only rowcounts)

Thus, we can separate three levels of details of rowsource statistics managed by the parameter _rowsource_statistics_sampfreq:
1: number of rows returned by rowsource
2: (1) + statistics as consistent gets, current gets, starts (number of times an execution plan step has been executed), etc
3: (1) + (2) + timing

As we know from 10g there are 3 ways how to enable gathering of rowsource statistics:
1) set parameter statistics_level to “all” on session or system level

alter session set statistics_level=all

By this another hidden parameter parameter _rowsource_execution_statistics is set into TRUE, it enables gathering of rowsource statistics.

• statistics_level=all also sets parameter timed_os_statistics.
• This is the only way to enable rowsource statistics gathering in 9i (except actual rows which are populated by enabled sql trace without statistics_level=all).
  Setting the parameter without sql trace enables gathering of rowsource statistics only for CBO (optimizer modes all_rows/first_rows), and the only way to get the statistics – to query V$SQL_PLAN_STATISTICS[_ALL].
  Setting the parameter with sql trace enables gathering of rowsource statistics for queries with any optimizer mode. In this case the subset statistics also can be printed in trace file. SQL trace without parameter statistics_level=all gathers only rowcounts.

In this case _rowsource_statistics_sampfreq is used only if the parameter has been set explicitly.
Otherwise using value is 1.

SQL> select ksppinm name,
  2         ksppstdvl value,
  3         ksppstdf IsDefault,
  4         decode(bitand(ksppstvf, 5), 1, 'TRUE', 0, 'FALSE') IsSetExplicitly
  5    from x$ksppi x, x$ksppcv y
  6   where (x.indx = y.indx)
  7     and ksppinm = '_rowsource_statistics_sampfreq';

NAME                                VALUE      ISDEFAULT  ISSETEXPLICITLY
----------------------------------- ---------- ---------- --------------------
_rowsource_statistics_sampfreq      128        TRUE       FALSE

SQL> select count(*) from v$parameter where name ='_rowsource_statistics_sampfreq';

  COUNT(*)
----------
         0

Here although the value of the parameter still be equal 128, the value using for frequency of getting timestamps will be 1.

If _rowsource_statistics_sampfreq is set manually then using value is _rowsource_statistics_sampfreq.

SQL> alter session set "_rowsource_statistics_sampfreq"=128;

Session altered.

SQL> select ksppinm name,
  2         ksppstdvl value,
  3         ksppstdf IsDefault,
  4         decode(bitand(ksppstvf, 5), 1, 'TRUE', 0, 'FALSE') IsSetExplicitly
  5    from x$ksppi x, x$ksppcv y
  6   where (x.indx = y.indx)
  7     and ksppinm = '_rowsource_statistics_sampfreq';

NAME                                VALUE      ISDEFAULT  ISSETEXPLICITLY
----------------------------------- ---------- ---------- --------------------
_rowsource_statistics_sampfreq      128        TRUE       TRUE

SQL> select count(*) from v$parameter where name ='_rowsource_statistics_sampfreq';

  COUNT(*)
----------
         1

In this case the value using for getting timestamps will be 128.

• Notice that hidden parameters set explicitly appear in v$parameter

It is possible to set _rowsource_execution_statistics=true inside a session with statistics_level=typical manually.
In such case (and in cases below) real frequence of getting timestamps is equal to parameter _rowsource_statistics_sampfreq regardless how it was set (explicitly or not).

2) hint gather_plan_statistics
it enables rowsource statistics for a query.
In this case just _rowsource_statistics_sampfreq is used.

3) sql trace (10046 event)
In this case just _rowsource_statistics_sampfreq is used.
Keep in mind that in this case it is possible to see rowsource statistics not only via trace file or tkprof report, but also using dbms_xplan.display_cursor, it is much more convenient.

• In each of these three cases optimizer environment (v$[sql/ses/sys]_optimizer_env) property sqlstat_enabled set to true.

Timing: overhead

Let’s see how and why different precise of timing calculation (_rowsource_statistics_sampfreq) affects amount of CPU and execution time of a query.
I will run a query with different _rowsource_statistics_sampfreq: default, 0, 128, 16, 8, 4, 2, 1.

In testcases below I will use simple table in 10 millions of rows and with an index

create table test(id not null, pad) as
  select rownum as id,
         '*' as pad
    from dual
 connect by level <= 10000000;

create index idx on test(id);

SQL> exec dbms_stats.gather_table_stats(user, 'TEST', estimate_percent => 100, cascade => true);

PL/SQL procedure successfully completed

and two queries over the table: one with Full Table Scan and another with Index Full Scan.
All runs on two versions – 10g and 11g.

In all cases I will use SQL tracing, because of easy access to consumed CPU instead of just rely on elapsed time.
In 10g sql trace for the session, in 11g SQL trace for exact SQL statement

alter session set events 'sql_trace[sql:9pzmsyh5t14bb]';

After all queries will be run and traced, I again run the same queries, but during each run of the SQL query I measured number of timer calls (gethrtime). I did it after previous run in order to avoid measurement (dtrace) overhead, so CPU time in the trace results in the table below does not contain dtrace overhead.

So, the first query with Full Table Scan:

select count(pad)
  from test;

--------------------------------------------
| Id  | Operation          | Name |   Rows |
--------------------------------------------
|   0 | SELECT STATEMENT   |      |      1 |
|   1 |  SORT AGGREGATE    |      |      1 |
|   2 |   TABLE ACCESS FULL| TEST |     10M|
--------------------------------------------

Result of the test are below: (all tkprof reports are made with option aggregate=no)
10g Full Table Scan: full script, output, raw trace, tkprof report

11g Full Table Scan: full script, output, raw trace, tkprof report

• Notice interesting detail: such form of SQL tracing:

  alter session set events 'sql_trace[sql:9pzmsyh5t14bb]'

  does not trace parse calls

And the same actions for changed query and Index Full Scan

select --+ index(test)
       count(pad)
  from test;

-----------------------------------------------------
| Id  | Operation                    | Name | Rows  |
-----------------------------------------------------
|   0 | SELECT STATEMENT             |      |     1 |
|   1 |  SORT AGGREGATE              |      |     1 |
|   2 |   TABLE ACCESS BY INDEX ROWID| TEST |    10M|
|   3 |    INDEX FULL SCAN           | IDX  |    10M|
-----------------------------------------------------

10g Index Full Scan: full script, output, raw trace, tkprof report

11g Index Full Scan: full script, output, raw trace, tkprof report

We can see that _rowsource_statistics_sampfreq increases number of additional timer calls which obviously increases CPU consumption.
Also we can see that default value 128 does not lead to significant overhead

Pay attention aslo on:
- in the Full Table Scan case number of additional timer calls in 10g is in 1.5 times more than in 11g. We saw the reason above in section “How it works” – qerstUpdateStats() calls gethrtime() twice
- in the Index Full Scan case number of additional timer calls in 10g is in 3(!) times more than in 11g. Why?
Let’s try to take a look at where, in what call stack, gethrtime() were called.

dtrace -p <PID> -n 'pid$target:libc.so.1:gethrtime:entry{@tim[ustack()]=count();}' > report.txt

Excerpts:
10g:

...
              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x2fd
              oracle`qerstFetch+0x152
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
              oracle`0xdba2dc
          9999999

              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x2fd
              oracle`qerstRowP+0x3d
              oracle`qerixtFetch+0x47d
              oracle`qerstFetch+0xf0
              oracle`qerpfFetch+0x102
              oracle`qerstFetch+0x12a
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
          9999999

              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x2fd
              oracle`qerstRowP+0x3d
              oracle`qertbFetchByRowID+0x88f
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
              oracle`0xdba2dc
          9999999

              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x31e
              oracle`qerstFetch+0x152
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
              oracle`0xdba2dc
          9999999

              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x31e
              oracle`qerstRowP+0x3d
              oracle`qerixtFetch+0x47d
              oracle`qerstFetch+0xf0
              oracle`qerpfFetch+0x102
              oracle`qerstFetch+0x12a
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
          9999999

              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x31e
              oracle`qerstRowP+0x3d
              oracle`qertbFetchByRowID+0x88f
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
              oracle`0xdba2dc
          9999999

              libc.so.1`gethrtime
              oracle`qerstSnapStats+0x44
              oracle`qerstRowP+0x29b
              oracle`qerixtFetch+0x47d
              oracle`qerstFetch+0xf0
              oracle`qerpfFetch+0x102
              oracle`qerstFetch+0x12a
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
         10000000

              libc.so.1`gethrtime
              oracle`qerstSnapStats+0x44
              oracle`qerstRowP+0x29b
              oracle`qertbFetchByRowID+0x88f
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
              oracle`0xdba2dc
         10000000

              libc.so.1`gethrtime
              oracle`qerstSnapStats+0x44
              oracle`qerstFetch+0x48
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
              oracle`0xdba2dc
         10000001

              libc.so.1`gethrtime
              oracle`qerstSnapStats+0x44
              oracle`qerstFetch+0x48
              oracle`qerpfFetch+0x102
              oracle`qerstFetch+0x12a
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
         10000001

              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x134
              oracle`qerstFetch+0x152
              oracle`qerpfFetch+0x102
              oracle`qerstFetch+0x12a
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
         10000001

              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x155
              oracle`qerstFetch+0x152
              oracle`qerpfFetch+0x102
              oracle`qerstFetch+0x12a
              oracle`qertbFetchByRowID+0x271
              oracle`qerstFetch+0xf0
              oracle`qergsFetch+0x16c
              oracle`qerstFetch+0xf0
              oracle`opifch2+0xa4e
              oracle`kpoal8+0xdb7
              oracle`opiodr+0x433
              oracle`ttcpip+0x46a
              oracle`opitsk+0x52d
              oracle`opiino+0x3f0
              oracle`opiodr+0x433
              oracle`opidrv+0x2f1
              oracle`sou2o+0x5b
              oracle`opimai_real+0x84
              oracle`main+0x64
         10000001

11g:

...
              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x206
              oracle`qerstRowP+0x4e
              oracle`qertbFetchByRowID+0xa4e
              oracle`qerstFetch+0x3ea
              oracle`qergsFetch+0x7c3
              oracle`qerstFetch+0x3ea
              oracle`opifch2+0x8a4
              oracle`kpoal8+0xfb9
              oracle`opiodr+0x3a7
              oracle`ttcpip+0x4f7
              oracle`opitsk+0x608
              oracle`opiino+0x335
              oracle`opiodr+0x3a7
              oracle`opidrv+0x2e6
              oracle`sou2o+0x58
              oracle`opimai_real+0x20a
              oracle`ssthrdmain+0x125
              oracle`main+0xd4
              oracle`0x15d18bc
          9999999

              libc.so.1`gethrtime
              oracle`qerstUpdateStats+0x206
              oracle`qerstFetch+0x1a2
              oracle`qertbFetchByRowID+0x4c7
              oracle`qerstFetch+0x3ea
              oracle`qergsFetch+0x7c3
              oracle`qerstFetch+0x3ea
              oracle`opifch2+0x8a4
              oracle`kpoal8+0xfb9
              oracle`opiodr+0x3a7
              oracle`ttcpip+0x4f7
              oracle`opitsk+0x608
              oracle`opiino+0x335
              oracle`opiodr+0x3a7
              oracle`opidrv+0x2e6
              oracle`sou2o+0x58
              oracle`opimai_real+0x20a
              oracle`ssthrdmain+0x125
              oracle`main+0xd4
              oracle`0x15d18bc
          9999999

              libc.so.1`gethrtime
              oracle`qerstRowP+0x25d
              oracle`qertbFetchByRowID+0xa4e
              oracle`qerstFetch+0x3ea
              oracle`qergsFetch+0x7c3
              oracle`qerstFetch+0x3ea
              oracle`opifch2+0x8a4
              oracle`kpoal8+0xfb9
              oracle`opiodr+0x3a7
              oracle`ttcpip+0x4f7
              oracle`opitsk+0x608
              oracle`opiino+0x335
              oracle`opiodr+0x3a7
              oracle`opidrv+0x2e6
              oracle`sou2o+0x58
              oracle`opimai_real+0x20a
              oracle`ssthrdmain+0x125
              oracle`main+0xd4
              oracle`0x15d18bc
         10000000

              libc.so.1`gethrtime
              oracle`qerstFetch+0x234
              oracle`qertbFetchByRowID+0x4c7
              oracle`qerstFetch+0x3ea
              oracle`qergsFetch+0x7c3
              oracle`qerstFetch+0x3ea
              oracle`opifch2+0x8a4
              oracle`kpoal8+0xfb9
              oracle`opiodr+0x3a7
              oracle`ttcpip+0x4f7
              oracle`opitsk+0x608
              oracle`opiino+0x335
              oracle`opiodr+0x3a7
              oracle`opidrv+0x2e6
              oracle`sou2o+0x58
              oracle`opimai_real+0x20a
              oracle`ssthrdmain+0x125
              oracle`main+0xd4
              oracle`0x15d18bc
         10000001

As we can see in 10g additional rowsource level appears – qerpfFetch (prefetch), which also wrapped by qerstSnapStats/qerstUpdateStats functions. As the result number of calls in three times (1.5*2) more than in 11g.

Amount of overhead depends on number of these additional calls of the function getting timestamp and also it depends on implementation of this function, that is in different OS overhead can be different.
Mentioned calls gethrtime() in Solaris or gettimeofday() in Linux are syscalls. But these are special type of syscalls, these syscalls are implemented as syscalls with minimal overhead. Such syscalls in Solaris are called “fast trap syscalls” in Linux vsyscalls/VDSO (only 64 bit architectures).

• more about vsyscall/VDSO in Linux:
  - vsyscalls and the vDSO,
  - gettimeofday speedup

It is why we do not see it as real syscalls. In this case in order to catch these calls we should trace library calls which are wrappers for the real syscalls. For example, in Linux it is possible to use ltrace (library trace).

Thus, performance overhead depends on
- number of additional timer calls: which itself depends on number of processed rows in each rowsource and number of rowsources
- implementation of the function getting timestamp, so in different OS overhead can be different
- and how exactly it works inside OS, even in the same OS it can work in different modes

Timing: Overhead: VDSO on Linux

• gettimeofday speedup:
  A Virtual Dynamic Shared Object (VDSO), is a shared library that allows application in user space to perform some kernel actions without as much overhead as a system call. The VDSO is often used to provide fast access to the gettimeofday system call data.
  Enabling the VDSO instructs the kernel to use its definition of the symbols in the VDSO, rather than the ones found in any user-space shared libraries, particularly the glibc. The effects of enabling the VDSO are system-wide – either all processes use it or none do.
  When enabled, the VDSO overrides the glibc definition of gettimeofday with it’s own. This removes the overhead of a system call, as the call is made direct to the kernel memory, rather than going through the glibc.

The behavior of gethrtime() can be changed (system-wide):

• The VDSO boot parameter has three possible values:
  0
  Provides the most accurate time intervals at μs (microsecond) resolution, but also produces the highest call overhead, as it uses a regular system call
  1
  Slightly less accurate, although still at μs resolution, with a lower call overhead
  2
  The least accurate, with time intervals at the ms (millisecond) level, but offers the lowest call overhead

VDSO behavior is enabled by default. The value used to enable the VDSO affects the behavior of gettimeofday. It can be enabled by echoing the desired value to /proc/sys/kernel/syscall64.

echo 1 > /proc/sys/kernel/vsyscall64

It allows us to compare performance of gehrtime() as VDSO and as usual syscall.
Environment: VirtualBox with Oracle 11.2.0.3 on Oracle Linux 6.3 inside

I will change VDSO behavior and run the same query as above, consequentially in three modes: 2,1,0 (gethrtime() becames a real syscall).
Again I use sql trace instead of statistics_level or hint gather_plan_statistics in order to easy access to cpu time.
The tracefile: orcl_ora_3017.trc

# echo 2 > /proc/sys/kernel/vsyscall64

-----------------------------------------------------------------------------------------------
| Id  | Operation                    | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |      |      1 |        |      1 |00:00:05.45 |   41400 |
|   1 |  SORT AGGREGATE              |      |      1 |      1 |      1 |00:00:05.45 |   41400 |
|   2 |   TABLE ACCESS BY INDEX ROWID| TEST |      1 |     10M|     10M|00:00:04.54 |   41400 |
|   3 |    INDEX FULL SCAN           | IDX  |      1 |     10M|     10M|00:00:01.74 |   23555 |
-----------------------------------------------------------------------------------------------

FETCH #139665587261584:c=5403178,e=5453897,p=0,cr=41400,cu=0,mis=0,r=1,dep=0,og=1,plh=4182611088,tim=1350789769054152

cpu=5,40

# echo 1 > /proc/sys/kernel/vsyscall64

-----------------------------------------------------------------------------------------------
| Id  | Operation                    | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |      |      1 |        |      1 |00:00:05.17 |   41400 |
|   1 |  SORT AGGREGATE              |      |      1 |      1 |      1 |00:00:05.17 |   41400 |
|   2 |   TABLE ACCESS BY INDEX ROWID| TEST |      1 |     10M|     10M|00:00:04.29 |   41400 |
|   3 |    INDEX FULL SCAN           | IDX  |      1 |     10M|     10M|00:00:01.63 |   23555 |
-----------------------------------------------------------------------------------------------

FETCH #139665578749400:c=5119221,e=5169173,p=0,cr=41400,cu=0,mis=0,r=1,dep=0,og=1,plh=4182611088,tim=1350789789329746

cpu=5,12

# echo 0 > /proc/sys/kernel/vsyscall64

-----------------------------------------------------------------------------------------------
| Id  | Operation                    | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |      |      1 |        |      1 |00:00:11.67 |   41400 |
|   1 |  SORT AGGREGATE              |      |      1 |      1 |      1 |00:00:11.67 |   41400 |
|   2 |   TABLE ACCESS BY INDEX ROWID| TEST |      1 |     10M|     10M|00:00:09.15 |   41400 |
|   3 |    INDEX FULL SCAN           | IDX  |      1 |     10M|     10M|00:00:03.28 |   23555 |
-----------------------------------------------------------------------------------------------

FETCH #139665585290416:c=11504250,e=11668447,p=0,cr=41400,cu=0,mis=0,r=1,dep=0,og=1,plh=4182611088,tim=1350789819491035

cpu=11,50

Summary

In order to measure number of times gettimeofday() has been called on Linux it is possible either use ltrace (in any mode) or any syscalls measurement method (as strace or dtrace on OL6) in mode 0 (when gettimeofday() is real syscall).

I have used dtrace and the simplest script:

# dtrace -p 3017 -n 'syscall::gettimeofday:entry/pid == $target/{@calls=count()}'
dtrace: description 'syscall::gettimeofday:entry' matched 1 probe
^C
40000640

Thus, number of timer calls in this case is 40 mln – the same number as in Solaris.

Timing: known issues

In some environments it is possible to get execution plans with absolutely crazy timings, as below.
This case is Solaris 10 on VirtualBox:

select count(*)
  from (select rownum id from dual connect by level <= 999999) t1,
       (select rownum id from dual connect by level <= 999999) t2
 where t1.id = t2.id;

--------------------------------------------------------------------------------------------------------------------
| Id  | Operation                        | Name | Starts | E-Rows | A-Rows |   A-Time   |  OMem |  1Mem | Used-Mem |
--------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                 |      |      1 |        |      1 |00:00:02.56 |       |       |          |
|   1 |  SORT AGGREGATE                  |      |      1 |      1 |      1 |00:00:02.56 |       |       |          |
|*  2 |   HASH JOIN                      |      |      1 |      1 |  99999 |07:15:32.57 |  3749K|  1936K| 5594K (0)|
|   3 |    VIEW                          |      |      1 |      1 |  99999 |03:28:07.19 |       |       |          |
|   4 |     COUNT                        |      |      1 |        |  99999 |02:31:21.65 |       |       |          |
|   5 |      CONNECT BY WITHOUT FILTERING|      |      1 |        |  99999 |01:53:30.97 |       |       |          |
|   6 |       FAST DUAL                  |      |      1 |      1 |      1 |00:00:00.01 |       |       |          |
|   7 |    VIEW                          |      |      1 |      1 |  99999 |03:28:17.39 |       |       |          |
|   8 |     COUNT                        |      |      1 |        |  99999 |02:50:24.37 |       |       |          |
|   9 |      CONNECT BY WITHOUT FILTERING|      |      1 |        |  99999 |00:56:48.52 |       |       |          |
|  10 |       FAST DUAL                  |      |      1 |      1 |      1 |00:00:00.01 |       |       |          |
--------------------------------------------------------------------------------------------------------------------

Pay attention that the query has been done in ~3 seconds, but reported time of some steps more than 7 hours.
The reason is non-monotonic timer. It means that subsequent call of timestamp can return value less that previous call (time from the past).
More details of this problem and the tool allowing to check it is described here: http://www.doerzbach.com/index.php?entry=entry120113-124240

P.S.
By the way Real-Time SQL monitoring works absolutely another way… but this is another story. I am going to explain it also in one of next posts.

P.P.S.
Seems that’s all. I hope didn’t forget anything
Any additions, corrections, questions are welcome!

Here I am explaining

• why reported actual time (“A-Time” or “time=” in sql trace) can be inconsistent,
• why execution time with statistics_level=all can be longer,
• and how exactly it works.

This stuff covers versions 10g and later if another is not mentioned.

Merry Christmas and Happy New Year 2013!

Introduction

The rowsource statistics are figures as time spent during execution of rowsource (a step of execution plan), number of returned rows, number of buffer gets and physical reads and writes and some statistics of usage of workareas. These statistics are populated in V$SQL_PLAN_STATISTICS and V$SQL_PLAN_STATISTICS_ALL (based on X$QESRSTAT and X$QESRSTATALL) when statistics gathering is enabled. Also if sql_trace (10046 event) is enabled then these statistics are printed in raw trace file as details inside rows “STAT #” and can be seen in tkprof report as execution plan.

The statistics gathering is enabled for a query only if it is turned on before a query execution and can be seen only when the query is finished (event with error) or canceled.

Rowsource statistics were introduced in Oracle 9i, but there was no useful interface to see they in convenient form. In 10g very important tool has been introduced – the function DBMS_XPLAN.DISPLAY_CURSOR having many useful options. The tool has become an indispensable tool for SQL tuning.

How to use

 Gathering of rowsource statistics can be enabled by one of following options:

1. set parameter statistics_level to all (at session or system level)
2. run a query with hint gather_plan_statistics
3. enable sql trace

• these methods explained more detailed below in the section “Timing sampling frequency” (part 2).

Gathered statistics populate V$SQL_PLAN_STATISTICS and V$SQL_PLAN_STATISTICS_ALL and can bee seen either
- by very useful form using function DBMS_XPLAN.DISPLAY_CURSOR (preferable).
- directly query V$SQL_PLAN_STATISTICS_ALL
- in trace file as STAT# rows or in tkprof report as figures inside brackets in execution plan, if a query was traced by sql trace

Using DBMS_XPLAN.DISPLAY_CURSOR with SQL trace
It is helpful to emphasize that even you used sql trace in order to investigate what is going on with a query, it is possible to use dbms_xplan.display_cursor to see rowsource statistics.
Pros are obvious:
It is easier, because it does not require access to the trace file on the host.
Rowsource statistics printed in tracefile contain only minimum of figures: cr (consistent reads), pr (physical reads), pw (physical writes), time. At the same time complete rowsource statistics collected in v$sql_plan_statistics[_all] and printed by dbms_xplan.display_cursor() contain more useful information then in trace, in particular
- starts (number of times a rowsource has been executed),
- current gets. Trace shows cu= only for sql call level (parse, exec, fetch) as “cu=” stat, but not for rowsource level. In the same time DISPLAY_CURSOR() shows “buffer gets” that is sum of cr and cu, but it is possible to query v$sql_plan_statistics[_all] to see cr and cu as separated values;
- workareas and tempspace usage
So, rowsource statistics printed in tracefile is only restricted subset of gathered statistics.
Also display_cursor() is able to provide much helpful information related with execution plan as predicates, column projection, notes, etc.

Cons:
V$SQL_PLAN_STATISTICS[_ALL] contains two sets of figures: one is cumulated values for ALL executions of a cursor and one for LAST execution of the cursor. It means that if a query should be executed many times and only one of execution in the middle is object of interest, then V$SQL_PLAN_STATISTICS[_ALL] and DISPLAY_CURSOR() can be useless, SQL trace is more appropriate.

The gathering of the rowsource statistics by sql trace also means that if you need to get rowsource statistics and session settings cannot be changed (so, parameter statistics_level cannot be used), the query cannot be changed (so, hint gather_plan_statistics cannot be used), then it is possible to just enable sql trace in the session, or enable sql trace for exact query using 11g synthax:

SQL> alter system set events 'sql_trace[sql:gpdmdntvzjgcr]';

where “gpdmdntvzjgcr” is sql_id which we are going to trace

• See also: sql_trace event in 11g:
  Miladin Modrakovic: ORACLE EVENT SQL_TRACE IN 11G
  Tanel Poder: Diagnostic events. ORADEBUG DOC and 11g improvements.
  

• See also: Randolf Geist – How to use dbms_cursor with parallel queries

• upd: There is also an option create SQL profile with hint gather_plan_statistics inside, can be useful especially in cases when sql_id is unknown for example when sql text is always changed/unique due to used literals, details in comments below the page

Rowsources and rowsource statistics architecture

Let’s look how rowsource statistics work. Let’s prepare data, this is a table with 50 rows and an index

SQL> create table tbl(id not null, pad) as
  2      select rownum as id, rpad('X',100,'X') as pad
  4        from all_objects
  5       where rownum <= 50;
Table created

SQL> create index idx on tbl(id);

Index created

SQL> exec dbms_stats.gather_table_stats(user, 'TBL', estimate_percent => 100, cascade => true);

PL/SQL procedure successfully completed

execute the Digger

digger.sh -p PID -FSmdeoz -f opifch* -t *,libc.so.1:gethrtime > output.trc

It means:
-f - opifch* - tracing is enabled inside function which name is beginning from opifch
-t - *,libc.so.1:gethrtime - to trace all calls of Oracle and calls gethrtime from library libc
-F - to print flow indents
-S - trace all syscalls
-m - print module
-d - print relative timestamps
-z - print user stack before enter into qergsFetch

and execute a query

select --+ index(tbl)
       count(pad)
  from tbl;

-----------------------------------------------------
| Id  | Operation                    | Name | Rows  |
-----------------------------------------------------
|   0 | SELECT STATEMENT             |      |     1 |
|   1 |  SORT AGGREGATE              |      |     1 |
|   2 |   TABLE ACCESS BY INDEX ROWID| TBL  |    50 |
|   3 |    INDEX FULL SCAN           | IDX  |    50 |
-----------------------------------------------------

twice – once without rowsource statistics: stats_disabled_full.txt
and once with them: freq1_full.txt

• The files stats_disabled_full.txt and freq1_full.txt mentioned above contain complete output of function calls. Instead of show complete tree of calls I will use the output of only qer* and qesa* functions and gethrtime() library call, just for better readability.
  In order to do get this output I execute Digger as above, but with parameters
  -f qerstFetch
  -t qer*,qesa*,libc.so.1:gethrtime

  digger.sh -p PID -FSmdeoz -f qerstFetch -t qer*,qesa*,libc.so.1:gethrtime > output.trc
  

  what means to trace only qer* and qesa* function calls and function gethrtime from library libc inside function qerstFetch. So, the output below is from files containing these calls.
  Why qer* and qesa* ? Functions qer* are Query Execution Rowsource layer and qesa* is just to have function qesaFastAggNonDistSS in the output, I will explain about it in the section Rowsource Arcitechture

We can see that with enabled rowsource statistics every call of rowsource function is wrapped in qerst* correlated functions.

The output without rowsource statistics: stats_disabled.txt

module call(args)             = return
------ ---------------------------------
oracle -> qergsFetch(0xC1A2D828, 0xFFFFFD7FFDC353E8, 0x4B6D830)
oracle   -> qertbFetchByRowID(0xC1A2D9F8, 0xFFFFFD7FFDC350C0, 0x39983B8)
oracle     -> qerixtFetch(0xC1A2DC80, 0xFFFFFD7FFDC349B8, 0x0)
oracle     <- qerixtFetch = 0x0
oracle     -> qertbGetPartitionNumber(0xFFFFFD7FFFDFB264, 0xC1A2DB08, 0xFFFFFD7FFDC35188)
oracle     <- qertbGetPartitionNumber = 0x1
oracle     -> qerixtFetch(0xC1A2DC80, 0xFFFFFD7FFDC349B8, 0x0)
oracle     <- qerixtFetch = 0x0

The output with rowsource statistics: freq1.txt

   module call(args)             = return
--------- ---------------------------------
   oracle -> qerstFetch(0xC19A32B8, 0xFFFFFD7FFDC349B8, 0x4B6D830)
libc.so.1   -> gethrtime(0xFFFFFD7FFFDFB510, 0x0, 0x43)
libc.so.1   <- gethrtime = 0xB3789663
   oracle   -> qergsFetch(0xC19A3428, 0xFFFFFD7FFDC346D8, 0x3570158)
   oracle     -> qerstFetch(0xC19A3520, 0xFFFFFD7FFDC34438, 0x39983B8)
libc.so.1       -> gethrtime(0xFFFFFD7FFFDFB290, 0x0, 0x14)
libc.so.1       <- gethrtime = 0xB37B2B9C
   oracle       -> qertbFetchByRowID(0xC19A3688, 0xFFFFFD7FFDC34120, 0x3570158)
   oracle         -> qerstFetch(0xC19A3810, 0xFFFFFD7FFDC34088, 0x0)
libc.so.1           -> gethrtime(0xFFFFFD7FFFDFAF90, 0x0, 0x51)
libc.so.1           <- gethrtime = 0xB37D610A
   oracle           -> qerixtFetch(0xC19A39A0, 0xFFFFFD7FFDC33980, 0x0)
   oracle           <- qerixtFetch = 0x0
   oracle           -> qerstUpdateStats(0xFFFFFD7FFDC34088, 0xFFFFFD7FFFDFAF90, 0x1)
libc.so.1             -> gethrtime(0x59, 0xCA6C3AE0, 0x5A)
libc.so.1             <- gethrtime = 0xB3812DCB
   oracle           <- qerstUpdateStats = 0xE9E8513F
   oracle         <- qerstFetch = 0x0
   oracle         -> qertbGetPartitionNumber(0xFFFFFD7FFFDFB084, 0xC19A3798, 0xFFFFFD7FFDC341E8)
   oracle         <- qertbGetPartitionNumber = 0x1
   oracle         -> qerstRowP(0xFFFFFD7FFFDFB250, 0x7FFF, 0x3570158)
   oracle           -> qerstUpdateStats(0xFFFFFD7FFDC34438, 0xFFFFFD7FFFDFB290, 0x1)
libc.so.1             -> gethrtime(0x59, 0xCA6C3AE0, 0x5A)
libc.so.1             <- gethrtime = 0xB38957DB
   oracle           <- qerstUpdateStats = 0xE9E850AF
libc.so.1           -> gethrtime(0xFFFFFD7FFFDFB450, 0x7FFF, 0xFFFFFD7FFDC4BBC8)
libc.so.1           <- gethrtime = 0xB38B4E8F
   oracle         <- qerstRowP = 0x7FFF

• Pay attention that parameter -S has been used in Digger, it means to trace syscalls, but we do not see such syscalls as gethrtime(), only library calls. It is because gethrtime() is implemented as special type of system call – fast trap syscall. On Linux Oracle calls gettimeofday(), which implemented as vsyscall/VDSO, it is also special type of syscall, which allows application in user space to perform some kernel actions without as much overhead as a system call.
  

Each real rowsource function as qergsFetch, qerixtFetch, qertbFetchByRowID now is wrapped by qerstFetch function with calling of qerstSnapStats() before real rowsource function and qerstUpdateStats() after it. Inside functions qerstSnapStats() and qerstUpdateStats() Oracle calls getting timestamp in order to calculate time.

a construction

-> qersomeFetch()
<- qersomeFetch()

becames

-> qerstFetch()
  -> qerstSnapStats()
    -> gethrtime()
    <- gethrtime
  <- qerstSnapStats
  -> qersomeFetch()
  <- qersomeFetch
  -> qerstUpdateStats()
    -> gethrtime()
    <- gethrtime
  <- qerstUpdateStats
<- qerstFetch

Inside this qerst* function before and after rowsource function call there are calls of a functions qerstSnapStats()/qerstUpdateStats() getting timestamp, gethrtime() in this case (Solaris). In Linux Oracle calls gettimeofday().

• Notice that I said qerstSnapStats(), but there is no function qerstSnapStats() in the output above. I suggest that it is because compilator inline optimizations. In the same case in 10g we can see that qerst* functions do not call gethrtime() directly.
  An excerpt from 10.2.0.5:

  -> qergsFetch(0x389777988, 0x332EF70, 0xFFFFFD7FFFDFC4B0)
    -> qerstFetch(0x389777A68, 0x1C84DE0, 0x389777988)
      -> qerstSnapStats(0xFFFFFD7FFDB77520, 0xFFFFFD7FFFDFC2C8, 0x389777988)
        -> gethrtime(0xFFFFFD7FFDB77520, 0xFFFFFD7FFFDFC2C8, 0x389777988)
        <- gethrtime = 0x55E25FEA
      <- qerstSnapStats = 0x3CD57897
      -> qertbFetchByRowID(0x389777BD0, 0x332EF70, 0xFFFFFD7FFFDFC320)
        -> qerstFetch(0x389777D48, 0x0, 0xFFFFFD7FFFDFC218)
          -> qerstSnapStats(0xFFFFFD7FFDB772C8, 0xFFFFFD7FFFDFBEC8, 0xFFFFFD7FFFDFC218)
            -> gethrtime(0xFFFFFD7FFDB772C8, 0xFFFFFD7FFFDFBEC8, 0xFFFFFD7FFFDFC218)
            <- gethrtime = 0x55E65426
          <- qerstSnapStats = 0x3CD57995
          -> qerixtFetch(0x389777ED8, 0x0, 0xFFFFFD7FFFDFC218)
          <- qerixtFetch = 0x0
          -> qerstUpdateStats(0xFFFFFD7FFDB772C8, 0xFFFFFD7FFFDFBEC8, 0x1)
            -> gethrtime(0xFFFFFD7FFDB772C8, 0xFFFFFD7FFFDFBEC8, 0x1)
            <- gethrtime = 0x55EAB35F
            -> gethrtime(0xFFFFFD7FFDB772C8, 0xFFFFFD7FFFDFBEC8, 0xF3)
            <- gethrtime = 0x55EBAAB1
          <- qerstUpdateStats = 0x3CD57AEA
        <- qerstFetch = 0x0
  

  But here in 10g we can see another problem – qerstUpdateStats calls gethrtime() twice! We will see effects below.
  Just keep this in mind, when see the output from 11.2 above, that in cases which looks like qerstFetch() or qerstRowP calls gethrtime() directly, in fact between them there is qerstSnapStats()

Keep in mind that an execution plan is a virtual flow of execution.
Example, let’s look at our execution plan:

-----------------------------------------------------
| Id  | Operation                    | Name | Rows  |
-----------------------------------------------------
|   0 | SELECT STATEMENT             |      |     1 |
|   1 |  SORT AGGREGATE              |      |     1 |
|   2 |   TABLE ACCESS BY INDEX ROWID| TBL  |    20 |
|   3 |    INDEX FULL SCAN           | IDX  |    20 |
-----------------------------------------------------

Virtual flow is:
SELECT STATEMENT (fetch) calls SORT AGGREGATE
SORT AGGREGATE calls TABLE ACCESS BY INDEX ROWID
TABLE ACCESS BY INDEX ROWID calls INDEX FULL SCAN

It almost matches with real flow, but not completely. In reality some actions related with rowsource SORT AGGREGATE are executed inside TABLE ACCESS BY INDEX ROWID.
stats_disabled.txt:

 -> qergsFetch(0xC12FEB20, 0xFFFFFD7FFDC3B490, 0x4B6D830)
   -> qertbFetchByRowID(0xC12FECF0, 0xFFFFFD7FFDC3B168, 0x39983B8)
     -> qerixtFetch(0xC12FEF50, 0xFFFFFD7FFDC3AA60, 0x0)
     <- qerixtFetch = 0x0
     -> qertbGetPartitionNumber(0xFFFFFD7FFFDFB264, 0xC12FEE00, 0xFFFFFD7FFDC3B230)
     <- qertbGetPartitionNumber = 0x1
     -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0x39983B8)
     <- qesaFastAggNonDistSS = 0x7FFF
     -> qerixtFetch(0xC12FEF50, 0xFFFFFD7FFDC3AA60, 0x0)
     <- qerixtFetch = 0x0
     -> qertbGetPartitionNumber(0xFFFFFD7FFFDFB264, 0xC12FEE00, 0xFFFFFD7FFDC3B230)
     <- qertbGetPartitionNumber = 0x1
     -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0x39983B8)
     <- qesaFastAggNonDistSS = 0x7FFF

Pay attention that the function qesaFastAggNonDistSS from SORT AGGREGATE rowsource is called inside qertbFetchByRowID.

Or another example: let’s take a slightly changed query with Hash Join:

select --+ index(t) leading(t t2) use_hash(t)
       count(pad)
  from tbl t, dual t2
 where substr(t.pad, 1, 1) = t2.dummy;

----------------------------------------------
| Id  | Operation                     | Name |
----------------------------------------------
|   0 | SELECT STATEMENT              |      |
|   1 |  SORT AGGREGATE               |      |
|*  2 |   HASH JOIN                   |      |
|   3 |    TABLE ACCESS BY INDEX ROWID| TBL  |
|   4 |     INDEX FULL SCAN           | IDX  |
|   5 |    TABLE ACCESS FULL          | DUAL |
----------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("T2"."DUMMY"=SUBSTR("T"."PAD",1,1))

Run the Digger:

digger.sh -p PID -FSmdeoz -f qergsFetch -t qer*,qesa* > hj_stats_dis_qer_qesa.txt

run the query, the output: hj_stats_dis_qer_qesa.txt

We can see here that “build input” is building inside qertbFetchByRowID (the table TBL), and walking through “probe input” inside qertbFetch (the table DUAL).

-> qergsFetch(0xBF321E00, 0xFFFFFD7FFDC3B230, 0x4B6D830)
  -> qerhjFetch(0xBF321FA0, 0xFFFFFD7FFDC3AFD8, 0x39983B8)
    ....
    -> qertbFetchByRowID(0xBF3266D0, 0xFFFFFD7FFDC37258, 0x6C7FED0)
      -> qerixtFetch(0xBF326958, 0xFFFFFD7FFDC36B50, 0x0)
      <- qerixtFetch = 0x0
      -> qertbGetPartitionNumber(0xFFFFFD7FFFDFB0B4, 0xBF3267E0, 0xFFFFFD7FFDC37320)
      <- qertbGetPartitionNumber = 0x1
      -> qerhjSplitBuild(0xFFFFFD7FFFDFB2E0, 0x7FFF, 0x6C7FED0)
      <- qerhjSplitBuild = 0x7FFE
      -> qerixtFetch(0xBF326958, 0xFFFFFD7FFDC36B50, 0x0)
      <- qerixtFetch = 0x0
      -> qertbGetPartitionNumber(0xFFFFFD7FFFDFB0B4, 0xBF3267E0, 0xFFFFFD7FFDC37320)
      <- qertbGetPartitionNumber = 0x1
      -> qerhjSplitBuild(0xFFFFFD7FFFDFB2E0, 0x7FFE, 0x6C7FED0)
      <- qerhjSplitBuild = 0x7FFE
      ...
    <- qertbFetchByRowID = 0x7FFE
     ...
    -> qertbFetch(0xBF322208, 0xFFFFFD7FFDC375D0, 0x6C85860)
      -> qerhjInnerProbeHashTable(0xFFFFFD7FFFDFB2E0, 0x7FFF, 0x0)
        -> qerhjBuildHashTable(0xBF321FA0, 0xFFFFFD7FFDC3AFD8, 0xFFFFFD7FFFDFB2E0
          -> qerhjAllocHashTable(0xBF321FA0, 0xFFFFFD7FFDC3AFD8, 0x32)
          <- qerhjAllocHashTable = 0x80
        <- qerhjBuildHashTable = 0x51
        -> qerhjSplitProbe(0xFFFFFD7FFFDFB2E0, 0xFFFFFD7FFDC3AFD8, 0x0)
        <- qerhjSplitProbe = 0x0
        -> qerhjWalkHashBucket(0xFFFFFD7FFFDFB2E0, 0xFFFFFD7FFDC3AFD8, 0x7FFF)
          -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0xFFFFFD7FFDA7F366)
          <- qesaFastAggNonDistSS = 0x7FFF
          -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0xFFFFFD7FFDA7F2F7)
          <- qesaFastAggNonDistSS = 0x7FFF
          -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0xFFFFFD7FFDA7F288)
          <- qesaFastAggNonDistSS = 0x7FFF
          -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0xFFFFFD7FFDA7F219)
          <- qesaFastAggNonDistSS = 0x7FFF
          -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0xFFFFFD7FFDA7F1AA)
          <- qesaFastAggNonDistSS = 0x7FFF
          -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0xFFFFFD7FFDA7F13B)
          <- qesaFastAggNonDistSS = 0x7FFF
          -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0xFFFFFD7FFDA7F0CC)
          <- qesaFastAggNonDistSS = 0x7FFF
          -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB540, 0x7FFF, 0xFFFFFD7FFDA7F05D)
          <- qesaFastAggNonDistSS = 0x7FFF
          ...
        <- qerhjWalkHashBucket = 0x7FFF
      <- qerhjInnerProbeHashTable = 0x7FFF
    <- qertbFetch = 0x7FFF
    ...
  <- qerhjFetch = 0x7FFF
<- qergsFetch = 0x0

Why it is important in context of our topic – because it means that Oracle cannot measure time just before enter in rowsource and after exit, something like this:

-> qerstSnapStats() -- get time here
  -> qerhjFetch()
    -> qerstSnapStats() -- get time here
      -> qertbFetchByRowID()
      <- qertbFetchByRowID
    <- qerstUpdateStats -- get time here, calculate time spent in qertbFetchByRowID
  <- qerhjFetch
<- qerstUpdateStats -- get time here, calculate time spent in qerhjFetch

In order to to separate time spent in one rowsource from another, in mentioned cases Oracle has to call qerstSnapStats/qerstUpdateStats for every row using function qerstRowP:
(the output for the latest query and the same with enabled rowsource statistics: hj_freq1.txt)

-> qerstRowP(0xFFFFFD7FFFDFAEC0, 0x7FFE, 0x3570158)
  -> qerstUpdateStats(0xFFFFFD7FFDC37608, 0xFFFFFD7FFFDFAF00, 0x1)
    -> gethrtime(0x59, 0xCA6EA940, 0x5A)
    <- gethrtime = 0x68E025
  <- qerstUpdateStats = 0x236
  -> qerstRowP(0xFFFFFD7FFFDFAFE0, 0x7FFE, 0x368F5DD68)
    -> qerstUpdateStats(0xFFFFFD7FFDC3A780, 0xFFFFFD7FFFDFB020, 0x1)
      -> gethrtime(0x59, 0xCA6EA940, 0x5A)
      <- gethrtime = 0x6B3FC8
    <- qerstUpdateStats = 0x339
    -> qerhjSplitBuild(0xFFFFFD7FFFDFB100, 0x7FFE, 0x368F5DE04)
    <- qerhjSplitBuild = 0x7FFE
    -> gethrtime(0xA545FF26, 0xFFFFFD7FFDB80FD0, 0x1)
    <- gethrtime = 0x6EDEB8
  <- qerstRowP = 0x7FFE
  -> gethrtime(0xA545FF26, 0x4E, 0x59)
  <- gethrtime = 0x706660
<- qerstRowP = 0x7FFE

or

-> qerstRowP(0xFFFFFD7FFFDFB250, 0x7FFF, 0xFFFFFD7FFDA7DFE3)
  -> qerstUpdateStats(0xFFFFFD7FFDC3B520, 0xFFFFFD7FFFDFB290, 0x1)
    -> gethrtime(0x59, 0xCA6EA940, 0x5A)
    <- gethrtime = 0x323F47B
  <- qerstUpdateStats = 0x27
  -> qesaFastAggNonDistSS(0xFFFFFD7FFFDFB450, 0x7FFF, 0xFFFFFD7FFDC16CD0)
  <- qesaFastAggNonDistSS = 0x7FFF
  -> gethrtime(0xFFFFFD7FFFDFB450, 0x7FFF, 0xFFFFFD7FFDC32938)
  <- gethrtime = 0x324AA83
<- qerstRowP = 0x7FFF

qerstUpdateStats() here updates statistics for qertbFetchByRowID, it allows the time spent in hash join or “sort aggregate” be not included in TABLE ACCESS BY INDEX ROWID despite the fact that physically hashing and counting happens inside TABLE ACCESS BY INDEX ROWID.

The result of this flow: these rowsources calculate time for every processed row. Thus, number of timer calls is approximately equal number of processed rows in every rowsource (except top) multiplied by two (two timestamps on row – SnapStats/UpdateStats), instead of just number of rowsources.

Now is the time to look in more detail at timing: Part 2: Overhead and inconsistent time

1. DTrace output can be shufled in multi-CPU environment.

This means that output can be in not chronological order. It is not something Digger specific, it is how DTrace works.

When DTrace script is being executed there are two parts to DTrace: the in kernel part and the DTrace script process in userland. When a dtrace probe fires, the data traced is placed in a per CPU buffer. Then, periodically, the DTrace script reads the buffers (in round-robin style) and continues processing the data for final output. While the data from any single CPU was entered into its buffer in order the probes are firing asynchronously with respect to all the CPUs.

If probe1 fired on CPU3 and after that probe2 fired on CPU2, and after that probe3 fired on CPU1 it is possible to get output as, for example, as following:
probe3
probe2
probe1

The most common case where we can see this are the cases when a process(thread) migrated from one CPU to another and DTrace consumer reads and prints current CPU-buffer before previous. It does not happen always but sometimes you can see it.

So, do not panic if some part of the output is not chronological in multi-CPU environment. It is expected.

If the output order is important, then it is strongly recommended to print such columns as timestamp or relative timestamp in order to be sure that the output is chronological, otherwise to fix it manually or using tools like sort (1). Column CPU# also is helpful because allows to see places where a process(thread) migrated from one CPU to another.

2. How much is performance impact of the Digger?

It depends on the usage. When you trace everything – all application functions or all library calls – performance impact can be significant and amount of printed info can be huge.

The Digger allows to restrict
a) traced area – where tracing is enabled
b) traced contents – which functions are traced

For example, in order to trace kcb* functions and all syscalls *read* inside function qertbFetch
command line should be:

digger.sh -p PID -Fcdeo -f qertbFetch -t kcb* -s *read*

Then generated dtrace script will be containing only required probes:

pid$target:a.out:qertbFetch:entry
pid$target:a.out:qertbFetch:return
+
pid$target:a.out:kcb*:entry
pid$target:a.out:kcb*:return
+
syscall::*read*:entry
syscall::*read*:entry

Note: “a.out” is synonym of executing binary.

It helps to trace only relevant information, only required functions. It decreases performance impact and decreases probability of drops.
So, if you need to decrease impact of the tracing just try to restrict traced area and contents.

3. Important bug is fixed.
There was an issue related with replacement tab symbols on spaces during generating html wordpress page with source code (thanks Zhenx Li for let me know). The issue led to possible error “syntax error at line n: `end of file’ unexpected” when the source of the tool was copied/pasted. Now links on html pages with source are replaced on direct download links.

There are many cases where Oracle revisit some buffer in the buffer cache many times inside one database call. It such cases it can pin the buffer and hold the buffer pinned and just read pinned buffer in consequences visits. It allows to avoid redundant logical reads.

There are statistics “buffer is pinned count” and “buffer is not pinned count”.
The concept is simple: there is the function kcbispnd (“Kernel Cache Buffer Is Pinned” as I suggest) where Oracle checks if a buffer is pinned or not. If a buffer is pinned then statistic “buffer is pinned count” is incremented, otherwise Oracle increments “buffer is not pinned count” and usually initiates logical read after that.
There is analog of kcbispnd, the function kcbipnns. It does the same checks but does not change any statistics. The function kcbispnd is usually called before logical read, the function kcbispnd can be called in another places, for example, before to release a buffer.

Inside a procedure performing logical read Oracle pins the buffer (except examination), attaches buffer handle to x$kcbbf by the function kcbzgs. When it exits from the function performing logical read it holds the buffer pinned and continues to work visiting the buffer or doing another work until it will be released. The function kcbrls (Kernel Cache Buffer Release) is used to release a buffer, also it is possible from another places, for example from kcb_post_apply can call kcbzfs (detach handle from x$kcbbf) directly.

• Note: read more about buffer_handles

Very simplified it looks like something like this

if kcbispnd(buffer) != 1 then
    <logical read>
end if;
<works with buffer>
<release the buffer> -- usually by kcbrls;

In such cases Oracle:
– acquires latch “cache buffer chains” during logical read, inside kcbgtcr
– acquires can acquire latch “cache buffer handles” during attaching of buffer handle, inside kcbzgs
– acquires latch “cache buffer chains” during releasing of a buffer
– acquires can acquire latch “cache buffer handles” during detaching of buffer handle, inside kcbzfs

• Upd: In order to do not grab latch “cache buffer handles” during every buffer pin, a session has _db_handles_cached (5 by default) buffer handles. The latch is captured if a session needs more than this number.

There are cases when Oracle just read and does not pin a buffer inside logical read. In such cases it increments statistic “consistent gets – examination”. In such cases only one latch “cache buffer chains” is acquired.

• Note: read more about latches acquiring inside Oracle functions

It is necessary to note, that Oracle does not execute the function kcbispnd before every logical read, but only in some cases, when in some point of source code it does not know if a buffer will be pinned or not, when if buffer is pinned or not depends on some conditions, on some data. In some cases Oracle uses kcbipnns to check if a buffer is pinned. And when it is known that a buffer will be pinned in some point of time Oracle just re-visit it without any check.

Example
Note: Oracle 11.2.0.2 on Solaris 10 is used

Let’s try to look at a simple example: reading table via an index.
The query:

select --+ index(tbl)
       count(pad) 
  from tbl;

The execution plan:

--------------------------------------
| Id  | Operation                    |
--------------------------------------
|   0 | SELECT STATEMENT             |
|   1 |  SORT AGGREGATE              |
|   2 |   TABLE ACCESS BY INDEX ROWID|
|   3 |    INDEX FULL SCAN           |
--------------------------------------

In this example we create simple table with index

SQL> create table tbl(id not null, pad) as
  2      select rownum id,
  3             rpad('*', 100, '*') pad
  4        from all_objects
  5       where rownum <= 100;
 
Table created

SQL> create index idx on tbl(id) pctfree 95;
 
Index created

SQL> exec dbms_stats.gather_table_stats(user, 'TBL', estimate_percent => 100, cascade => true);
 
PL/SQL procedure successfully completed

SQL> select blevel,
  2         leaf_blocks,
  3         clustering_factor
  4    from dba_ind_statistics
  5   where index_name = 'IDX';
 
    BLEVEL LEAF_BLOCKS CLUSTERING_FACTOR
---------- ----------- -----------------
         1           7                 2

Notice that I use pctfree on the index to force Oracle create more than one index block for our small table. It is because mechanics of pinning of root index block slightly differs from pinning of leaf index block.

I execute the following query to get rowsource statistics

select --+ index(tbl) gather_plan_statistics
       count(pad) 
  from tbl;

SQL> select * from table(dbms_xplan.display_cursor(null, null, 'allstats last'));

--------------------------------------------------------------------------------------
| Id  | Operation                    | Name | Starts | A-Rows |   A-Time   | Buffers |
--------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |      |      1 |      1 |00:00:00.01 |      10 |
|   1 |  SORT AGGREGATE              |      |      1 |      1 |00:00:00.01 |      10 |
|   2 |   TABLE ACCESS BY INDEX ROWID| TBL  |      1 |    100 |00:00:00.01 |      10 |
|   3 |    INDEX FULL SCAN           | IDX  |      1 |    100 |00:00:00.01 |       8 |
--------------------------------------------------------------------------------------

8 buffer gets (1 root index block + 7 leaf blocks) were done during INDEX FULL SCAN and two buffer gets (clustering factor) were done during TABLE ACCESS BY INDEX ROWID.

Below is an excerpt from DTraceLIO aggregation output (full DTraceLIO output: dtracelio_opifch.trc):

================================= Logical I/O Summary (grouped by object) ================================
 object_id  data_object_id       lio        cr    cr (d)        cu    cu (d) ispnd (Y) ispnd (N)   pin rls
---------- --------------- --------- --------- --------- --------- --------- --------- --------- ---------
         0               0         0         0         0         0         0         0         1         0
     79083           79083         2         2         0         0         0       197         2         2
     79084           79084         8         8         0         0         0         0         1         7
---------- --------------- --------- --------- --------- --------- --------- --------- --------- ---------
     total                        10        10         0         0         0       197         4         9
==========================================================================================================

Here we can see again that this query requires 8 logical reads of the object 79084 (index IDX) and two logical reads of object 79083 (the table TBL).
Pay attention on
isnpd (Y) = 197 and ispnd (N) = 2 for the table TBL,
isnpd (N) = 1 for the index IDX,
isnpd (N) = 1 for object with object_id = 0.

pin rls (pin release, kcbrls function) has been called twice for the table and 7 times for the index.

So, now the time to use the Digger!

Before I execute our query I execute the Digger and wait for prompt in the file.

digger.sh -p 16829 -Fgdeoz -f opifch* > opifch.trc

the output: opifch.trc (200Kb)

these keys means:
-p PID – to trace process ID 16829
-F – to use flow indents
-gdeo – print columns: timestamp, relative timestamp (cpu time of the process minus DTrace overhead), elapsed time of a call (ela), cpu time of a call (cpu)
-z – to print call stack before enter to traced function
-f – to trace calls inside function opifch*
A filter (option t) is not used, it means that all application calls will be traced inside traced function function opifch.

Note: Pay attention, that we trace the function calls only from the application and do not trace the calls from libraries and syscalls. Don’t worry about it, we will see syscalls in another Digger’s examples.

Why opifch? It is possible to trace a function qergsFetch (qer Group by Sort Fetch) the step 2 in the execution plan, it is where our execution plan starts to execute. Or even qertbFetchByRowID, step 3, where buffer pinning will be caused. But there is some important piece of code related with buffer pinning outside of qergsFetch. It is why I trace opifch.

Why I use the symbol star (opifch*) instead of opifch?
There are functions opifch and opifch2. Different clients uses different API. As a result with different clients you can get different entry points of a fetch call. An exmple, an output from PL/SQL Developer.
Notice that in the output SQL contains two fetch calls. It is because SQL+ calls the second fetch to get “no data found”.
And output from PL/SQL Developer also contains two fetch calls, it is because PL/SQL Developer run “select x from dual” before each query.

After that I execute the query and look at the output

select --+ index(tbl)
       count(pad) 
  from tbl;

So, lets take look at the output. I will start from the function qergsFetch, this is the step 2 in the execution plan, it is where our execution plan starts to execute. We will investigate calls before qergsFetch in following examples.

   -> qergsFetch(0x89FAF9B0, 0xFFFFFD7FFD99EB60, 0x4D98000)
     -> qeaeCnt(0xFFFFFD7FFD99FCA8, 0xFFFFFD7FFD99EB60, 0x1)
     <- qeaeCnt = 0x8
     -> qertbFetchByRowID(0x89FAFB80, 0xFFFFFD7FFD99E858, 0x3ABDE00)
       -> qerixtFetch(0x89FAFE08, 0xFFFFFD7FFD99E170, 0x0)

Here we see that Oracle calls qergsFetch in order to perform SORT AGGREGATE, step 2.
SORT AGGREGATE needs to get data from the step 3, so it calls qertbFetchByRowID in order to perform TABLE ACCESS BY INDEX ROWID.
TABLE ACCESS BY INDEX ROWID needs to get rowids from the index, so it calls qerixtFetch in order to perform INDEX FULL SCAN.

         -> qeilsr(0xFFFFFD7FFD99E3A8, 0xFFFFFD7FFD99E568, 0x0)
           -> qeilbk1(0xFFFFFD7FFD99E568, 0x0, 0xFFFFFD7FFD99E3A8)
             -> kcbispnd(0xFFFFFD7FFD99E610, 0x0, 0x0)
             <- kcbispnd = 0x0
             -> ktrgtc2(0xFFFFFD7FFD99E600, 0xFFFFFD7FFD99E1A0, 0x46491D0)
               -> ktsmg_max_query(0x0, 0x0, 0xFFFFFD7FFDC12038)
               <- ktsmg_max_query = 0x3F
               -> kcbgtcr(0xFFFFFD7FFD99E610, 0x1, 0x47C)
                 -> ktrexc(0xFFFFFD7FFFDFA960, 0xE771F80, 0x0)
                   -> ktrEvalBlockForCR(0xFFFFFD7FFD99DE9C, 0xE771F80, 0xFFFFFD7FFD99DEC0)
                   <- ktrEvalBlockForCR = 0x1
                   -> ktcckv(0xFFFFFD7FFFDFA980, 0xFFFFFD7FFD99DE9C, 0x129FC2)
                   <- ktcckv = 0x0
                   -> kdifkc(0x74ADA014, 0xFFFFFD7FFD99E568, 0x129FC2)
                     -> kdxbrs1(0x74ADA04C, 0xFFFFFD7FFD9A2CD0, 0xFFFFFD7FFFDF9F5E)
                       -> lmebucp(0xFFFFFD7FFD9A2D11, 0x0, 0x74ADBFB1)
                       <- lmebucp = 0xFFFFFFFF
                       -> lmebucp(0xFFFFFD7FFD9A2D11, 0x0, 0x74ADBFC1)
                       <- lmebucp = 0xFFFFFFFF
                     <- kdxbrs1 = 0x1000BCC
                   <- kdifkc = 0x1
                 <- ktrexc = 0x2
               <- kcbgtcr = 0x0
             <- ktrgtc2 = 0x0

Here Oracle gets index root block. It executes function kcbispnd and check if the buffer is pinned yet. Buffer is not pinned, so function kcbispnd increments statistic “buffer is not pinned count” and returns 0. After that Oracle initiates logical read (ktrgtc2->kcbgtcr).
Notice that function ktrexc is called in kcbgtcr. It is examination – the case when Oracle just read the buffer and does not pin it. The statistic “consistent gets – examination” is incremented inside this function.

             -> kcbipnns(0xFFFFFD7FFD99E610, 0x1, 0xFFFFFD7FFD99E600)
             <- kcbipnns = 0x0
             -> ktrget2(0xFFFFFD7FFD99E600, 0xFFFFFD7FFD99E1A0, 0x47D)
               -> ktsmg_max_query(0x0, 0x0, 0xFFFFFD7FFDC12038)
               <- ktsmg_max_query = 0x3F
               -> kcbgtcr(0xFFFFFD7FFD99E610, 0x0, 0x47D)
                 -> ktrexf(0xFFFFFD7FFFDFA350, 0xE771F80, 0x0)
                 <- ktrexf = 0x9
                 -> kcbzgs(0x1, 0xE771F80, 0x1)
                   -> kssadf_numa_intl(0x26, 0x91F2BB00, 0x924905D8)
                   <- kssadf_numa_intl = 0x91A74018
                 <- kcbzgs = 0x91A74018
                 -> kcbz_fp_buf(0x74BDF348, 0x91A74098, 0x1)
                 <- kcbz_fp_buf = 0x1
               <- kcbgtcr = 0x748C4014
               -> kcbcge(0xFFFFFD7FFD99E610, 0xFFFF8000, 0x0)
               <- kcbcge = 0x129D0C
               -> ktcckv(0xFFFFFD7FFD99E610, 0xFFFFFD7FFD99DE9C, 0x0)
               <- ktcckv = 0x129D0C
             <- ktrget2 = 0x748C4064

Here Oracle reads index leaf block.
Pay attention that it uses function kcbipnns instead of kcbispnd to determine if buffer pinned or not, found that buffer is not pinned and initiates logical read of leaf block. Inside kcbgtcr Oracle pins the buffer, attaches the buffer handle to x$kcbbf (kcbzgs).
After Oracle exits from kcbgtcr it holds a buffer pinned.

           <- qeilbk1 = 0x0
         <- qeilsr = 0x0
         -> kdifxs(0xFFFFFD7FFD99E3A8, 0x1, 0x0)
           -> kdifxs1(0xFFFFFD7FFD99E3A8, 0x1, 0x0)
           <- kdifxs1 = 0x748C5FB8
         <- kdifxs = 0x748C5FB8
         -> kafgex1(0x0, 0x748C5FB8, 0x0)
         <- kafgex1 = 0x0
       <- qerixtFetch = 0x0

After that Oracle fetches row from index block (kdifxs) and exits from qerixtFetch.
Now Oracle has the first rowid and needs to read table block to get value of the column PAD from the table.

       -> qetlbr(0xFFFFFD7FFD99E970, 0xFFFFFD7FFD9A2C14, 0x0)
         -> kcbispnd(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD9A2C14, 0x0)
         <- kcbispnd = 0x0
         -> kdsgrp(0xFFFFFD7FFD99E970, 0x0, 0xFFFFFD7FFD99E970)
           -> kcbispnd(0xFFFFFD7FFD99E988, 0x0, 0xFFFFFD7FFD99E970)
           <- kcbispnd = 0x0
           -> ktrget2(0xFFFFFD7FFD99E978, 0xFFFFFD7FFD99E8A0, 0x360)
             -> ktsmg_max_query(0x0, 0x0, 0xFFFFFD7FFDC12038)
             <- ktsmg_max_query = 0x3F
             -> kcbgtcr(0xFFFFFD7FFD99E988, 0x0, 0x360)
               -> ktrexf(0xFFFFFD7FFFDFA720, 0xE771F80, 0x0)
               <- ktrexf = 0x9
               -> kcbzgs(0x1, 0xE771F80, 0x1)
                 -> kssadf_numa_intl(0x26, 0x91F2BB00, 0x924905D8)
                 <- kssadf_numa_intl = 0x91A74398
               <- kcbzgs = 0x91A74398
               -> kcbz_fp_buf(0x74BF2CC8, 0x91A74418, 0x1)
               <- kcbz_fp_buf = 0x1
             <- kcbgtcr = 0x74AD4014
             -> kcbcge(0xFFFFFD7FFD99E988, 0x0, 0x0)
             <- kcbcge = 0x0
             -> ktcckv(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD99DE9C, 0x0)
             <- ktcckv = 0x0
           <- ktrget2 = 0x74AD407C
         <- kdsgrp = 0x74AD5F91
       <- qetlbr = 0x74AD5F91

Here we can see similar strategy – Oracle executes kcbispnd and checks if a buffer containing required table block is pinned yet or not. The function kcbispnd increments statistic “buffer is not pinned count”, returns 0, and Oracle initiates logical read of a buffer containing table block.
Pay attention that it executes function kcbispnd twice: the first time before function kdsgrp (Kernel Data Scan Get Row Piece) and the second time inside this procedure.
Be aware that the first call of kcbispnd here is the call for object_id=0 which we saw in DTraceLIO above.

Why kcbispnd is executed twice here?
As we will see below, for each rowid from the index oracle performs two checks:
1) the first call of kcbispnd is checking if buffer from previous rowid still be pinned. If yes and if it differs then block from latest read rowid from the index it releases pinned buffer.
In our case at this point of time there was no previous rowid and there was no pinned buffer containing table block. It is why object_id=0.
2) the second call of kcbispnd is checking if buffer from current(latest) rowid is pinned. If not then Oracle initiates logical read.

Now we have two buffers are pinned, these buffers contain index block and table block.

After that Oracle repeats following pattern (except two cases when Oracle either needs to read next leaf block or rowid from the index points to another table block):

       -> qerixtFetch(0x89FAFE08, 0xFFFFFD7FFD99E170, 0x0)
         -> kdifxs(0xFFFFFD7FFD99E3A8, 0x1, 0x0)
           -> kdifxs1(0xFFFFFD7FFD99E3A8, 0x1, 0x0)
           <- kdifxs1 = 0x748C5FAC
         <- kdifxs = 0x748C5FAC
         -> kafgex1(0x0, 0x748C5FAC, 0x0)
         <- kafgex1 = 0x0
       <- qerixtFetch = 0x0

Here Oracle again calls fetch from the index (qerixtFetch) and gets row from the index without execution of kcbispnd and logical I/O. It just reads a pinned buffer.

       -> qetlbr(0xFFFFFD7FFD99E970, 0xFFFFFD7FFD9A2C14, 0x0)
         -> kcbispnd(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD9A2C14, 0x0)
         <- kcbispnd = 0x1
         -> kdsgrp(0xFFFFFD7FFD99E970, 0x0, 0xFFFFFD7FFD99E970)
           -> kcbispnd(0xFFFFFD7FFD99E988, 0x0, 0xFFFFFD7FFD99E970)
           <- kcbispnd = 0x1
         <- kdsgrp = 0x74AD5F26
       <- qetlbr = 0x74AD5F26

After that Oracle reads row from the table buffer with twice execution of kcbispnd: before kdsgrp and inside kdsgrp.

Pay attention that Oracle executes kcbispnd before each logical read (or visit to pinned) of a buffer containing table block, but does not execute it before each read of buffer containing index block. It is what I said that Oracle uses this concept of using kcbispnd not in each case when it is going to perform logical read, but in cases when in some point of source code it does not know if a buffer will be pinned or not, when it depends on some conditions, on data in our case. In this case in some point of source code it is known that index buffer will be pinned, it does not depend on anything, in contrast the table buffer which depends of previous rowid from the index.

In our case with the current execution plan Oracle would gets index blocks one-by-one, gets each next index buffer once, reads rows from the buffer and releases it. In contrast a table buffer can be got many times. It depends on clustering factor. It is exactly where clustering factor is important.

Two exception from previous pattern:

       -> qerixtFetch(0x89FAFE08, 0xFFFFFD7FFD99E170, 0x0)
         -> kdifxs(0xFFFFFD7FFD99E3A8, 0x1, 0x0)
           -> kdifxs1(0xFFFFFD7FFD99E3A8, 0x1, 0x0)
             -> kcbipnns(0xFFFFFD7FFD99E3C0, 0x1, 0x0)
             <- kcbipnns = 0x1
             -> kcbrls(0xFFFFFD7FFD99E3C0, 0x1, 0x0)
               -> kcbzar(0x91A74958, 0x91F5D738, 0x100000)
               <- kcbzar = 0x8
               -> kcbzfs(0x91A748D8, 0x2000000000000019, 0xFFFFFFFD)
                 -> kjbilms(0x91A748D8, 0x2000000000000019, 0xFFFFFFFD)
                 <- kjbilms = 0x0
                 -> kssrmf_numa_intl(0x91A748D8, 0x91F2BB00, 0x0)
                 <- kssrmf_numa_intl = 0x0
               <- kcbzfs = 0x0
             <- kcbrls = 0x0
             -> ksuttctest(0xE768C18, 0x1, 0x0)
               -> nioqts(0xE768D28, 0x0, 0x0)
               <- nioqts = 0x0
             <- ksuttctest = 0x0
             -> ktrget2(0xFFFFFD7FFD99E3B0, 0xFFFFFD7FFD99E1A0, 0x391)
               -> ktsmg_max_query(0x0, 0x0, 0xFFFFFD7FFDC12038)
               <- ktsmg_max_query = 0x3F
               -> kcbgtcr(0xFFFFFD7FFD99E3C0, 0x0, 0x391)
                 -> ktrexf(0xFFFFFD7FFFDF94D0, 0xE771F80, 0x0)
                 <- ktrexf = 0x9
                 -> kcbzgs(0x1, 0xE771F80, 0x1)
                   -> kssadf_numa_intl(0x26, 0x91F2BB00, 0x924905D8)
                   <- kssadf_numa_intl = 0x91A748D8
                 <- kcbzgs = 0x91A748D8
                 -> kcbz_fp_buf(0x74BDF218, 0x91A74958, 0x1)
                 <- kcbz_fp_buf = 0x1
               <- kcbgtcr = 0x748C2014
               -> kcbcge(0xFFFFFD7FFD99E3C0, 0xFFFF8000, 0x0)
               <- kcbcge = 0x129D0C
               -> ktcckv(0xFFFFFD7FFD99E3C0, 0xFFFFFD7FFD99DE9C, 0x0)
               <- ktcckv = 0x129D0C
             <- ktrget2 = 0x748C2064
           <- kdifxs1 = 0x748C3FB8
         <- kdifxs = 0x748C3FB8
         -> kafgex1(0x0, 0x748C3FB8, 0x0)
         <- kafgex1 = 0x0
       <- qerixtFetch = 0x0

Here Oracle found that it needs to read the next leaf block. It releases the pinned buffer and initiates logical read in order to get the next leaf block.

       -> qetlbr(0xFFFFFD7FFD99E970, 0xFFFFFD7FFD9A2C14, 0x0)
         -> kcbispnd(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD9A2C14, 0x0)
         <- kcbispnd = 0x1
         -> kcbipnns(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD9A2C14, 0x0)
         <- kcbipnns = 0x1
         -> kcbrls(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD9A2C14, 0x0)
           -> kcbzar(0x91A74418, 0x919D7E10, 0x100000)
           <- kcbzar = 0x8
           -> kcbzfs(0x91A74398, 0x2000000000000019, 0xFFFFFFFD)
             -> kjbilms(0x91A74398, 0x2000000000000019, 0xFFFFFFFD)
             <- kjbilms = 0x0
             -> kssrmf_numa_intl(0x91A74398, 0x91F2BB00, 0x0)
             <- kssrmf_numa_intl = 0x0
           <- kcbzfs = 0x0
         <- kcbrls = 0x0
         -> kdsgrp(0xFFFFFD7FFD99E970, 0x0, 0xFFFFFD7FFD99E970)
           -> kcbispnd(0xFFFFFD7FFD99E988, 0x0, 0xFFFFFD7FFD99E970)
           <- kcbispnd = 0x0
           -> ktrget2(0xFFFFFD7FFD99E978, 0xFFFFFD7FFD99E8A0, 0x360)
             -> ktsmg_max_query(0x0, 0x0, 0xFFFFFD7FFDC12038)
             <- ktsmg_max_query = 0x3F
             -> kcbgtcr(0xFFFFFD7FFD99E988, 0x0, 0x360)
               -> ktrexf(0xFFFFFD7FFFDFA720, 0xE771F80, 0x0)
               <- ktrexf = 0x9
               -> kcbzgs(0x1, 0xE771F80, 0x1)
                 -> kssadf_numa_intl(0x26, 0x91F2BB00, 0x924905D8)
                 <- kssadf_numa_intl = 0x91A74398
               <- kcbzgs = 0x91A74398
               -> kcbz_fp_buf(0x74BDF6D8, 0x91A74418, 0x1)
               <- kcbz_fp_buf = 0x1
             <- kcbgtcr = 0x748CA014
             -> kcbcge(0xFFFFFD7FFD99E988, 0x0, 0x0)
             <- kcbcge = 0x0
             -> ktcckv(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD99DE9C, 0x0)
             <- ktcckv = 0x0
           <- ktrget2 = 0x748CA07C
         <- kdsgrp = 0x748CBF91
       <- qetlbr = 0x748CBF91

Here Oracle got rowid pointing to another table block. It checks if previous buffer is pinned, kcbispnd returns 1, this means that buffer is pinned, Oracle finds pinned buffer contains another table block then block from current rowid and releases pinned buffer. It is why Oracle executes kcbispnd before kdsgrp. It checks if rowid from the index contains the same table buffer as it keeps pinned or not. Inside kdsgrp it executes kcbispnd again to check that block from current rowid is pinned, finds that this buffer is not pinned yet, increments “buffer is not pinned count”, returns 0, and initiates logical read of the table block.

It is why clustering factor is important.
After Oracle reads next index entry from an index it checks if this index entry points to the same table buffer as previous index entry or not. If yes then Oracle just revisit a pinned buffer. Otherwise Oracle releases previous pinned table buffer and get and pin the buffer from the current index entry.

Have you noticed that a few buffers are still pinned in this point? These buffers are released at the end of the fetch call.

   -> qecrlssub(0x89FB0D58, 0xFFFFFD7FFDC12038, 0xA)
     -> qergsRelease(0x89FAF9B0, 0xFFFFFD7FFD99EB60, 0xA)
       -> qertbRelease(0x89FAFB80, 0xFFFFFD7FFD99E858, 0xA)
         -> kcbipnns(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD99E858, 0xA)
         <- kcbipnns = 0x1
         -> kcbrls(0xFFFFFD7FFD99E988, 0xFFFFFD7FFD99E858, 0xA)
           -> kcbzar(0x91A74418, 0x918C9830, 0x100000)
           <- kcbzar = 0x8
           -> kcbzfs(0x91A74398, 0x2000000000000019, 0xFFFFFFFD)
             -> kjbilms(0x91A74398, 0x2000000000000019, 0xFFFFFFFD)
             <- kjbilms = 0x0
             -> kssrmf_numa_intl(0x91A74398, 0x91F2BB00, 0x0)
             <- kssrmf_numa_intl = 0x0
           <- kcbzfs = 0x0
         <- kcbrls = 0x0
         -> qerixRelease(0x89FAFE08, 0xFFFFFD7FFD99E170, 0x0)
           -> qerixReleaseSelf(0x89FAFE08, 0x1, 0x0)
             -> kcbipnns(0xFFFFFD7FFD99E610, 0x1, 0x0)
             <- kcbipnns = 0x1
             -> kcbipnns(0xFFFFFD7FFD99E610, 0x1, 0x0)
             <- kcbipnns = 0x1
             -> kcbrls(0xFFFFFD7FFD99E610, 0x1, 0x0)
               -> kcbzar(0x91A74098, 0x918A0548, 0x100000)
               <- kcbzar = 0x8
               -> kcbzfs(0x91A74018, 0x2000000000000019, 0xFFFFFFFD)
                 -> kjbilms(0x91A74018, 0x2000000000000019, 0xFFFFFFFD)
                 <- kjbilms = 0x0
                 -> kssrmf_numa_intl(0x91A74018, 0x91F2BB00, 0x0)
                 <- kssrmf_numa_intl = 0x0
               <- kcbzfs = 0x0
             <- kcbrls = 0x0
             -> kcbipnns(0xFFFFFD7FFD99E3C0, 0x91F2BB00, 0x0)
             <- kcbipnns = 0x1
             -> kcbrls(0xFFFFFD7FFD99E3C0, 0x91F2BB00, 0x0)
               -> kcbzar(0x91A74958, 0x91F9D780, 0x100000)
               <- kcbzar = 0x8
               -> kcbzfs(0x91A748D8, 0x2000000000000019, 0xFFFFFFFD)
                 -> kjbilms(0x91A748D8, 0x2000000000000019, 0xFFFFFFFD)
                 <- kjbilms = 0x0
                 -> kssrmf_numa_intl(0x91A748D8, 0x91F2BB00, 0x0)
                 <- kssrmf_numa_intl = 0x0
               <- kcbzfs = 0x0
             <- kcbrls = 0x0
           <- qerixReleaseSelf = 0x1
         <- qerixRelease = 0xE6
       <- qertbRelease = 0xE6
     <- qergsRelease = 0x26
   <- qecrlssub = 0x26

Conclusion
We can conclude that “buffer is pinned count”
– is not a number of times when Oracle re-visit pinned buffer. It can read from pinned buffer without any checks, it is just reading from memory, as fetched rows from the index in our example.
– is not a check before every logical read.
– is not a number of times when a buffer has been pinned. Number times when a buffer has been pinned is number of logical reads, except examinations and direct-path reads.
– is called when Oracle in some point of source code does not know if a buffer is pinned or not. When it depends on some conditions and on data.

Notice that there are no calls related exactly with latches such as kslget or ksl_get_shared_latch. It is what I was talking about scripts KSLBEGIN/KSLEND inside procedures kcbgtcr and kcbrls (look at Important note #1).

Appendix
There is an option to trace buffer pinning by set the parameter _trace_pin_time:

alter system set "_trace_pin_time"=1 scope=spfile;

The database should be restarted.

In this case buffer pins (where, data block address, time) in all sessions will be traced. Example:

pin kdiwh15: kdifxs dba 0x100071d:1 time 3630971669
pin kdswh05: kdsgrp dba 0x1000714:1 time 3630973016
pin kdswh05: kdsgrp dba 0x1000713:1 time 3630973465
pin kdswh05: kdsgrp dba 0x1000714:1 time 3630974088
pin kdswh05: kdsgrp dba 0x1000713:1 time 3630974532

Every row contains

pin <where> dba <DBA> time <timestamp>

“where” means function name inside which buffer has been pinned.

Following query can be used in order to see used (active) buffer handles

select bf.*, 
       w.kcbwhdes
  from (select b.*, 
               decode(kcbbfcr, 1, 'CR', 'CUR') cr
          from x$kcbbf b
         where bitand(b.kcbbfso_flg, 1) = 1) bf,
       x$kcbwh w
 where bf.kcbbfwhr = w.indx

Join with x$kcbwh here is helpful just to see kcbwhdes which means “where” that is a function name where buffer has been pinned.

Post Scriptum
I hope it helped you to understand better what is “buffer is pinned count” and how to use the Digger.

By the way look at the annoucements at the right side of the page. It is what I am going to publish next time. Next blog entries will be about timing, rowsource statistics and wait events.
Coming soon.

Continue…

Let’s consider merge of two simple tables.

 merge into t1
 using t2 on (t1.id = t2.id)
  when matched 
  then update set n = 1;

The tables:

SQL> select * from v$version where rownum = 1;
 
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

SQL > create table t1 compress
  2             as
  3         select rownum as id,
  4                0 as n,
  5                lpad('*', 4000, '*') as pad
  6           from dual
  7        connect by level <= 1000000;

Table created.

SQL > create table t2 compress
  2             as
  3         select 1000000 + rownum as id,
  4                1 as n,
  5                lpad('*', 4000, '*') as pad
  6           from dual
  7        connect by level <= 1100000;

Table created.

SQL > exec dbms_stats.gather_table_stats(user, 'T1', estimate_percent => 100, degree => 4);

PL/SQL procedure successfully completed.

SQL > exec dbms_stats.gather_table_stats(user, 'T2', estimate_percent => 100, degree => 4);

PL/SQL procedure successfully completed.

SQL > alter session set statistics_level=all;

Session altered.

And a simple statement:

SQL > merge into t1
  2   using t2 on (t1.id = t2.id)
  3    when matched 
  4    then update set n = 1;

0 rows merged.

Elapsed: 00:08:41.93

Let’s look at the execution plan with rowsource statistics:

SQL > select * from table(dbms_xplan.display_cursor(null, null, 'allstats last'));

-------------------------------------
SQL_ID  b5cp092vum9nw, child number 0
-------------------------------------
merge into t1 using t2 on (t1.id = t2.id)  when matched then update set n = 1
Plan hash value: 3423882595
----------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  | Writes |  OMem |  1Mem | Used-Mem | Used-Tmp|
----------------------------------------------------------------------------------------------------------------------------------------------
|   0 | MERGE STATEMENT      |      |      1 |        |      0 |00:08:41.66 |   52742 |    608K|    556K|       |       |          |         |
|   1 |  MERGE               | T1   |      1 |        |      0 |00:08:41.66 |   52742 |    608K|    556K|       |       |          |         |
|   2 |   VIEW               |      |      1 |        |      0 |00:08:41.66 |   52742 |    608K|    556K|       |       |          |         |
|*  3 |    HASH JOIN         |      |      1 |      1 |      0 |00:08:41.66 |   52742 |    608K|    556K|  2047M|    56M|   30M (1)|    4346K|
|   4 |     TABLE ACCESS FULL| T1   |      1 |   1000K|   1000K|00:00:05.96 |   22174 |  22163 |      0 |       |       |          |         |
|   5 |     TABLE ACCESS FULL| T2   |      1 |   1100K|   1100K|00:00:05.58 |   30568 |  30556 |      0 |       |       |          |         |
----------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("T1"."ID"="T2"."ID")

So, our statement is executing more than 8 minutes. All this time has been consumed on the step 3 – HASH JOIN. Notice amount of used temp space 4346 Mb (this is old oddity that value in Used-Temp should be multiplied by 1000) as a result high amount of I/O operations on temp space (columns Reads, Writes – 556K of physical reads and 556K physical writes via direct path read temp and direct path write temp) due to size of workarea (hash_area_size) was not enough (Used-Mem = 30Mb).
Note: obviously in your case it can be absolutely another time and amount of I/O, in particular it depends on hash area size, in my case it was about 30Mb

How can we improve the performance of the statement and especially hash join?
Should we increase hash area size?

May be, but before let’s take look at the same execution plan with column projection:

SQL> select * from table(dbms_xplan.display_cursor('b5cp092vum9nw', null, 'allstats last +projection'));
 
-------------------------------------
SQL_ID  b5cp092vum9nw, child number 0
-------------------------------------
merge into t1 using t2 on (t1.id = t2.id)  when matched then update set n = 1
Plan hash value: 3423882595
----------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  | Writes |  OMem |  1Mem | Used-Mem | Used-Tmp|
----------------------------------------------------------------------------------------------------------------------------------------------
|   0 | MERGE STATEMENT      |      |      1 |        |      0 |00:08:41.66 |   52742 |    608K|    556K|       |       |          |         |
|   1 |  MERGE               | T1   |      1 |        |      0 |00:08:41.66 |   52742 |    608K|    556K|       |       |          |         |
|   2 |   VIEW               |      |      1 |        |      0 |00:08:41.66 |   52742 |    608K|    556K|       |       |          |         |
|*  3 |    HASH JOIN         |      |      1 |      1 |      0 |00:08:41.66 |   52742 |    608K|    556K|  2047M|    56M|   30M (1)|    4346K|
|   4 |     TABLE ACCESS FULL| T1   |      1 |   1000K|   1000K|00:00:05.96 |   22174 |  22163 |      0 |       |       |          |         |
|   5 |     TABLE ACCESS FULL| T2   |      1 |   1100K|   1100K|00:00:05.58 |   30568 |  30556 |      0 |       |       |          |         |
----------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("T1"."ID"="T2"."ID")
Column Projection Information (identified by operation id):
-----------------------------------------------------------
   1 - SYSDEF[4], SYSDEF[32720], SYSDEF[1], SYSDEF[96], SYSDEF[32720]
   3 - (#keys=1) "T1"."ID"[NUMBER,22], "T2"."ID"[NUMBER,22], "T1".ROWID[ROWID,10], "T1"."PAD"[VARCHAR2,4000], "T1"."N"[NUMBER,22], 
       "T2"."PAD"[VARCHAR2,4000], "T2"."N"[NUMBER,22]
   4 - "T1".ROWID[ROWID,10], "T1"."ID"[NUMBER,22], "T1"."N"[NUMBER,22], "T1"."PAD"[VARCHAR2,4000]
   5 - "T2"."ID"[NUMBER,22], "T2"."N"[NUMBER,22], "T2"."PAD"[VARCHAR2,4000]

Pay attention on column projection of the steps 3, 4, 5. It contains ALL columns from the tables T1 and T2 although the query uses only some of them. The part of the execution plan below of the step 2 is equivalent to

select t1.rowid,
       t1.id,
       t1.n,
       t1.pad,
       t2.id,
       t2.n,
       t2.pad
  from t1, t2
 where t1.id = t2.id

It looks like a bug or “not implemented yet”. In our case it means that all columns from the table T1 (especially the fat column PAD) will be retrieved and stored in the workarea.

This can dramatically affect the performance:

• For operations using workareas it means increased amount of dataset. If required amount of available memory is not enough then it leads to additional I/O in temp space.
  In our particular case all columns from the table T1 are retrieved and put into the workarea. If the condition in the statement would be something like

  t1.id between t2.id and t2.id + 100
  

  then Hash Join would be impossible, because it can be based on “equality” condition. In that case we can expect Merge Join and then both datasests must be sorted.

  SQL> explain plan for
    2  merge into t1
    3  using t2
    4  on (t1.id between t2.id and t2.id + 100)
    5  when matched then
    6      update set n = 1;
   
  Explained
   
  SQL> @plan
   
  ---------------------------------------------------------------------------------------
  | Id  | Operation              | Name | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     |
  ---------------------------------------------------------------------------------------
  |   0 | MERGE STATEMENT        |      |     1 |     2 |       |  1766K  (1)| 05:53:21 |
  |   1 |  MERGE                 | T1   |       |       |       |            |          |
  |   2 |   VIEW                 |      |       |       |       |            |          |
  |   3 |    MERGE JOIN          |      |     1 |  8018 |       |  1766K  (1)| 05:53:21 |
  |   4 |     SORT JOIN          |      |  1000K|  3822M|  7812M|   840K  (1)| 02:48:06 |
  |   5 |      TABLE ACCESS FULL | T1   |  1000K|  3822M|       |  4928   (1)| 00:01:00 |
  |*  6 |     FILTER             |      |       |       |       |            |          |
  |*  7 |      SORT JOIN         |      |  1100K|  4206M|  8593M|   926K  (1)| 03:05:16 |
  |   8 |       TABLE ACCESS FULL| T2   |  1100K|  4206M|       |  6779   (1)| 00:01:22 |
  ---------------------------------------------------------------------------------------
  Predicate Information (identified by operation id):
  ---------------------------------------------------
     6 - filter("T1"."ID"<="T2"."ID"+100)
     7 - access(INTERNAL_FUNCTION("T1"."ID")>=INTERNAL_FUNCTION("T2"."ID"))
         filter(INTERNAL_FUNCTION("T1"."ID")>=INTERNAL_FUNCTION("T2"."ID"))
  

• It affects data access optimization. If there would be indexes containing all columns using in a query as t1(id, n) or t2(id,n), then Index Fast Full Scan will be impossible here because the indexes do not contain required redundant columns (as PAD).
  If there would be a condition doing Index Range Scan appropriate access path then TABLE ACCESS BY INDEX ROWID would exist to get rest of unnecessary columns.
• If a row is chained then it leads to additional logical (and maybe physical) I/O to read redundant columns
• If the query would be run on Exadata and one or both Full Table Scan would be offloading, then there would no Column Projection optimization, returned amount of data from Storage Cells to PGA would be redundant and contains unnecessary columns from a table.
• etc

It is what can happen if we ignore Column Projection.

Our particular case can be fixed easy

SQL >  merge into (select id, n from t1) t1
  2    using (select id, n from t2) t2 on (t1.id = t2.id)
  3     when matched
  4     then update set n = 1;

0 rows merged.

Elapsed: 00:00:20.09

upd note: (select id, n from t2) is redundant here. It could be (select id from t2).

The execution plan with rowsurce statistics and column projection:

----------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  | Writes |  OMem |  1Mem | Used-Mem | Used-Tmp|
----------------------------------------------------------------------------------------------------------------------------------------------
|   0 | MERGE STATEMENT      |      |      1 |        |      0 |00:00:19.83 |   52742 |  35651 |   2032 |       |       |          |         |
|   1 |  MERGE               | T1   |      1 |        |      0 |00:00:19.83 |   52742 |  35651 |   2032 |       |       |          |         |
|   2 |   VIEW               |      |      1 |        |      0 |00:00:19.83 |   52742 |  35651 |   2032 |       |       |          |         |
|*  3 |    HASH JOIN         |      |      1 |      1 |      0 |00:00:19.83 |   52742 |  35651 |   2032 |    36M|  6589K|   34M (1)|   16384 |
|   4 |     TABLE ACCESS FULL| T2   |      1 |   1100K|   1100K|00:00:11.77 |   30568 |  28841 |      0 |       |       |          |         |
|   5 |     TABLE ACCESS FULL| T1   |      1 |   1000K|   1000K|00:00:04.44 |   22174 |   4778 |      0 |       |       |          |         |
----------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("ID"="ID")
Column Projection Information (identified by operation id):
-----------------------------------------------------------
   1 - SYSDEF[4], SYSDEF[32720], SYSDEF[1], SYSDEF[96], SYSDEF[32720]
   3 - (#keys=1) "ID"[NUMBER,22], "ID"[NUMBER,22], "N"[NUMBER,22], "T1".ROWID[ROWID,10], "N"[NUMBER,22]
   4 - "ID"[NUMBER,22], "N"[NUMBER,22]
   5 - "T1".ROWID[ROWID,10], "ID"[NUMBER,22], "N"[NUMBER,22]

Now column projection does not contain unnecessary columns, amount of required memory and as a result amount of temp space (Used-Tmp ~ 16Mb) and I/O (2032 physical reads and 2032 physical writes via direct path read temp and direct path write temp) was significantly reduced, and the statement was executed about 20 seconds instead of 8 minutes.

Notice, that in this case Oracle takes the table T2 as build input. It is because RowID from the table T1 is required to perform MERGE.
The part of the execution plan below of the step 2 now looks like

select t1.id, 
       t1.n, 
       t1.rowid, 
       t2.id, 
       t2.n
  from t1, t2
 where t1.id = t2.id

It estimates that amount of resultset

1100K rows * ("ID"[NUMBER,22], "N"[NUMBER,22])

less than

1000K rows * ("T1".ROWID[ROWID,10], "ID"[NUMBER,22], "N"[NUMBER,22])

and that resultset requires less amount of workarea.
Thus it is reasonable to use T2 as build input here.


]]> http://alexanderanokhin.wordpress.com/2012/07/18/dont-forget-about-column-projection/feed/ 4 alexanderanokhin http://alexanderanokhin.wordpress.com/2012/06/22/change-in-parallel-insert-between-11-1-and-11-2-with-serial-data-access/ http://alexanderanokhin.wordpress.com/2012/06/22/change-in-parallel-insert-between-11-1-and-11-2-with-serial-data-access/#comments Fri, 22 Jun 2012 09:34:52 +0000 Alexander Anokhin http://alexanderanokhin.wordpress.com/?p=756 ]]> There is an interesting small change in parallel insert between 11.1 and 11.2 in queries with parallelized insert, but serialized data access. On a practice the simplest case where we can see this it is CTAS or parallel insert as select from remote table.
As an example I will use CTAS from remote table.

create table tbllocal parallel 2
    as select *
  from tblremote@dblink;

Note: even if remote part of this query (select from tblremote@dblink) will be parallelized, it will be run in parallel on remote side. Communication between remote and local servers will be performed in one thread: local query coordinator – remote query coordinator. So, in our case it does not matter if query part parallelized or not.

11.1.0.7

SQL> select * from v$version where rownum = 1;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production

SQL> show parameter optimizer_features_enable

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
optimizer_features_enable            string      11.1.0.7

SQL> explain plan for
  2  create table tbllocal parallel 2
  3      as select * from tblremote@dblink;

SQL> @plan

-------------------------------------------------------------------------------------------------------------------
| Id  | Operation                | Name      | Rows  | Bytes | Cost (%CPU)| Time     | TQ/Ins |IN-OUT| PQ Distrib |
-------------------------------------------------------------------------------------------------------------------
|   0 | CREATE TABLE STATEMENT   |           |    41 |    82 |     3   (0)| 00:00:01 |        |      |            |
|   1 |  PX COORDINATOR          |           |       |       |            |          |        |      |            |
|   2 |   PX SEND QC (RANDOM)    | :TQ10001  |    41 |    82 |     2   (0)| 00:00:01 |  Q1,01 | P->S | QC (RAND)  |
|   3 |    LOAD AS SELECT        | TBLLOCAL  |       |       |            |          |  Q1,01 | PCWP |            |
|   4 |     BUFFER SORT          |           |       |       |            |          |  Q1,01 | PCWC |            |
|   5 |      PX RECEIVE          |           |    41 |    82 |     2   (0)| 00:00:01 |  Q1,01 | PCWP |            |
|   6 |       PX SEND ROUND-ROBIN| :TQ10000  |    41 |    82 |     2   (0)| 00:00:01 |        | S->P | RND-ROBIN  |
|   7 |        REMOTE            | TBLREMOTE |    41 |    82 |     2   (0)| 00:00:01 | DBLINK | R->S |            |
-------------------------------------------------------------------------------------------------------------------
Remote SQL Information (identified by operation id):
----------------------------------------------------
   7 - SELECT "DUMMY" FROM "TBLREMOTE" "TBLREMOTE" (accessing 'DBLINK.WORLD' )

This execution plan means (a little bit simlified):
Query Coordinator:
[step 7] get result from remote side,
[step 6] put data into special structures in SGA

Slave process:
[step 5] gets data from SGA
[step 4] put the data in workarea (PGA)
[step 3] performes direct-path insert

Pay attention on the step 4 – BUFFER SORT.
BUFFER SORT it is buffering technic using sort area size, without actual sorting.
Note: additional details: Jonathan Lewis – buffer-sorts

The problem here is that BUFFER SORT should get ALL data from child rowsource before parent operation will be performed. Before inserts will be started all data from remote side will be read and put into workareas of slave processes. As the result if you are going to carry out CTAS from huge remote table you need the same huge amount of memory (sort area size) and if you do not have enough memory the data will be spilled in temp.
As a result there is probability to get many “direct-path read/write temp” wait events and even “ORA-01652: unable to extend temp segment .. in tablespace TEMP” before the first row will be inserted.

11.2.0.2
Fortunately Oracle 11.2 does not have this step in execution plan.

Note: Oracle 11.2 does not have this step in execution plan even optimizer_features_enable = 11.1.0.7, 10.2.0.5, and adds this step when optimizer_features_enable = 10.2.0.4

SQL> select * from v$version where rownum = 1;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production

SQL> show parameter optimizer_features_enable

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
optimizer_features_enable            string      11.2.0.2

SQL> explain plan for
  2  create table tbllocal parallel 2
  3      as select * from tblremote@dblink;

Explained

SQL> @plan
------------------------------------------------------------------------------------------------------------------
| Id  | Operation               | Name      | Rows  | Bytes | Cost (%CPU)| Time     | TQ/Ins |IN-OUT| PQ Distrib |
------------------------------------------------------------------------------------------------------------------
|   0 | CREATE TABLE STATEMENT  |           |    41 |    82 |     3   (0)| 00:00:01 |        |      |            |
|   1 |  PX COORDINATOR         |           |       |       |            |          |        |      |            |
|   2 |   PX SEND QC (RANDOM)   | :TQ10001  |    41 |    82 |     2   (0)| 00:00:01 |  Q1,01 | P->S | QC (RAND)  |
|   3 |    LOAD AS SELECT       | TBLLOCAL  |       |       |            |          |  Q1,01 | PCWP |            |
|   4 |     PX RECEIVE          |           |    41 |    82 |     2   (0)| 00:00:01 |  Q1,01 | PCWP |            |
|   5 |      PX SEND ROUND-ROBIN| :TQ10000  |    41 |    82 |     2   (0)| 00:00:01 |        | S->P | RND-ROBIN  |
|   6 |       REMOTE            | TBLREMOTE |    41 |    82 |     2   (0)| 00:00:01 | DBLINK | R->S |            |
------------------------------------------------------------------------------------------------------------------
Remote SQL Information (identified by operation id):
----------------------------------------------------
   6 - SELECT "DUMMY" FROM "TBLREMOTE" "TBLREMOTE" (accessing 'DBLINK.WORLD' )

Here slave process will inserts data immediately after get a portion from query coordinator.
Thus, here you do not need huge amount of memory and temp space to perform CTAS from huge remote table.


]]> http://alexanderanokhin.wordpress.com/2012/06/22/change-in-parallel-insert-between-11-1-and-11-2-with-serial-data-access/feed/ 5 alexanderanokhin http://alexanderanokhin.wordpress.com/2012/05/21/serial-direct-read-for-small-tables-in-11-2-0-2/ http://alexanderanokhin.wordpress.com/2012/05/21/serial-direct-read-for-small-tables-in-11-2-0-2/#comments Mon, 21 May 2012 17:18:07 +0000 Alexander Anokhin http://alexanderanokhin.wordpress.com/?p=638 ]]> Today I have fixed an issue related with serial direct path reads.

There is 11.2.0.2 database

SQL> select * from v$version where rownum = 1;
 
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
 
SQL> SELECT a.ksppinm  "Parameter",
  2         b.ksppstvl "Session Value",
  3         c.ksppstvl "Instance Value"
  4    FROM x$ksppi a, x$ksppcv b, x$ksppsv c
  5   WHERE a.indx = b.indx
  6     AND a.indx = c.indx
  7     AND a.ksppinm = '_serial_direct_read';
 
Parameter               Session Value           Instance Value
----------------------- ----------------------- -----------------------
_serial_direct_read     auto                    auto

There is a very small table, let’s call it dualcopy.

SQL> exec dbms_stats.gather_table_stats(user, 'dualcopy', estimate_percent => 100);
 
PL/SQL procedure successfully completed
 
SQL> select object_type,
  2         num_rows,
  3         blocks,
  4         empty_blocks
  5    from all_tab_statistics
  6   where table_name = 'DUALCOPY';
 
OBJECT_TYPE    NUM_ROWS     BLOCKS EMPTY_BLOCKS
------------ ---------- ---------- ------------
TABLE                 1          1            0

Let’s try to do select from the table with enabled 10046 event:

SQL> alter session set events '10046 trace name context forever, level 8';
 
Session altered

SQL> select * from dualcopy;
 
DUMMY
-----
X

Below is an excerpt from the trace file

PARSING IN CURSOR #18446744071497556472 len=24 dep=0 uid=2500 oct=3 lid=2500 tim=3988723224325 hv=250290826 ad='3e911e388' sqlid='g32a9w87fq8na'
select * from dualcopy
END OF STMT
PARSE #18446744071497556472:c=20000,e=26430,p=0,cr=27,cu=0,mis=1,r=0,dep=0,og=2,plh=769194902,tim=3988723224324
EXEC #18446744071497556472:c=0,e=21,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=2,plh=769194902,tim=3988723224417
WAIT #18446744071497556472: nam='SQL*Net message to client' ela= 4 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=3988723224541
WAIT #18446744071497556472: nam='SQL*Net message from client' ela= 66047 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=3988723290684
WAIT #18446744071497556472: nam='direct path read' ela= 7852 file number=705 first dba=23954 block cnt=1 obj#=2229858 tim=3988723301159
WAIT #18446744071497556472: nam='SQL*Net message to client' ela= 5 driver id=1413697536 #bytes=1 p3=0 obj#=2229858 tim=3988723301360
FETCH #18446744071497556472:c=10000,e=10521,p=1,cr=2,cu=0,mis=0,r=1,dep=0,og=2,plh=769194902,tim=3988723301437
STAT #18446744071497556472 id=1 cnt=1 pid=0 pos=1 obj=2229858 op='TABLE ACCESS FULL DUALCOPY (cr=2 pr=1 pw=0 time=10419 us cost=2 size=2 card=1)'

What?! serial direct path read for the table in one block?! What’s going on?

The table can be small, but if your application accesses it many times it can lead to significant amount of physical I/O.

Serial direct read mode (_serial_direct_read) is a property of a child cursor. Similar behavior is possible if the same query was parsed earlier in a session with ALWAYS or TRUE mode (_serial_direct_read=always or _serial_direct_read=true). If we pay attention on the parse call, it was a hard parsing (mis=1) and new child cursor. Thus, this is not the reason in our case.

The reason is that the table has been configured for using of KEEP pool, but database has not.

SQL> select table_name, buffer_pool
  2    from all_tables
  3   where table_name = 'DUALCOPY';
 
TABLE_NAME                     BUFFER_POOL
------------------------------ -----------
DUALCOPY                       KEEP

SQL> show parameter db_keep_cache_size
 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_keep_cache_size                   big integer 0

This is the bug 12530276. Oracle 11.2.0.2 treats all ‘keep objects’ as large objects when ‘keep pool’ is not configured.

Allocation of KEEP pool fixes the problem.


]]> http://alexanderanokhin.wordpress.com/2012/05/21/serial-direct-read-for-small-tables-in-11-2-0-2/feed/ 7 alexanderanokhin http://alexanderanokhin.wordpress.com/2012/03/19/dtrace-lio-new-features/ http://alexanderanokhin.wordpress.com/2012/03/19/dtrace-lio-new-features/#comments Mon, 19 Mar 2012 15:44:30 +0000 Alexander Anokhin http://alexanderanokhin.wordpress.com/?p=489 ]]> What’s new in DTrace LIO?
Jump to introduction: Dynamic tracing of Oracle logical I/O

1. The list of supported functions performing logical I/O is extended:
Note: some function names below are my suggestions.

Consistent gets:

• kcbgtcr – Kernel Cache Buffer Get Consistent Read. This is general entry point for consistent read.
• kcbldrget – Kernel Cache Buffer Load Direct-Read Get. The function performing direct-path read. Interesting detail: 10.2 the function kcbldrget is called just after kcbgtcr, in 11.2 by (from) kcbgtcr.

Current gets (db block gets):

• kcbgcur – Kernel Cache Buffer Get Current Read
• kcbget – Kernel Cache Buffer Get Buffer. This is analogue of kcbgcur function, as I observed this function is called for index branch and leaf blocks
• kcbnew – Kernel Cache Buffer New Buffer
• kcblnb (kcblnb_dscn in 11.2) – Kernel Cache Buffer Load New Buffer. The function performing direct-path load. Decoding of block coordinates is not supported for this function in the current version of DTraceLIO. The parameters such as object_id, data_object_id will be shown as 0 (zero)

Note: KCB and KCBL are module names
KCB – Kernel Cache Buffer (Get, change, and release buffers)
KCBL – Kernel Cache Buffer Load (Direct I/O routines)


2. Functions related with buffer pinning are added.
Following functions calls are traced:

• kcbispnd – Kernel Cache Buffer Is Pinned – functions where Oracle checks is some buffer pinned yet or not. In this function Oracle either increments statistic “buffer is pinned count” if buffer is pinned yet or “buffer is not pinned count” if buffer is not pinned.
• kcbrls – Kernel Cache Buffer Release Pin

Important note: buffer pinning itself is performed inside logical I/O functions. So, unfortunately it is impossible to see it as function calls.

3. Changed parameters
Usage:
dtracelio.d PID [show_each_call] [interval]

• PID – unix process ID
• show_each_call – This is the bitmask determining how functions calls will be shown.
  1st bit – to show logical I/O functions
  2nd bit – to show buffer pinning
  Examples:
  0: output of each call will be disabled
  1: logical I/O functions will be shown
  3: logical I/O and buffer pinning functions will be shown
  Default value: 1 (only logical I/O functions will be shown)
• interval – Specifies the number of seconds over which Summary form with cumulative figures will be shown. Works only when show_each_call is disabled.
  Default value: 0 (disabled)

4. Existed summary table is changed, new columns are added
The existent summary table “Logical I/O Summary (grouped by function/object)” is changed.

============================= Logical I/O Summary (grouped by function/object) ===========================
 function    stat   object_id   data_object_id   mode_held   where     bufs     calls
--------- ------- ----------- ---------------- ----------- ------- -------- ---------
  kcbgcur      cu           0               -1           2     381        1         1
  kcbgcur      cu       56756            56756           2     594        1         1
   kcbnew      cu           0               -1                 383        1         1
  kcbgtcr      cr       56756            56756                 446        2         2
  kcbgtcr      cr       56756            56756                 447        2         2
  kcbgtcr      cr       56756            56756                 577        2         2
==========================================================================================================

New columns:

• stat: “cr” – consistent get, “cr (d)” – consistent get direct, “cu” – current get, ”cu (d)” – current get direct
• mode_held: mode in which buffer is pinned. The column is supported only for functions kcbgcur and kcbget
• bufs: number of processed bufers. This number is equal number of calls for almost all functions except kcbnew. In function kcbnew N (many) buffers can be processed by one function call.

5. New summary table is added

In addition to old summary output grouped by function/object new output form “grouped by object” is added.

An example:

================================= Logical I/O Summary (grouped by object) ================================
 object_id  data_object_id       lio        cr    cr (d)        cu    cu (d) ispnd (Y) ispnd (N)   pin rls
---------- --------------- --------- --------- --------- --------- --------- --------- --------- ---------
         1              -1        21         0         0        21         0         0         0        21
        17              17       108       108         0         0         0         0         0       108
         0              -1     64042        18         0     64024         0         0         0        36
    415893         2142703    463565     49351         0    414214         0         0    387842     88604
   2142704         2142704   1977665         0         0   1977665         0         0         0      9467
---------- --------------- --------- --------- --------- --------- --------- --------- --------- ---------
     total                   2505401     49477         0   2455924         0    387842         0     98236
==========================================================================================================

Where following figures are showed:

• lio – All logical reads done by an object. Equal of “session logical reads” statistic. lio = cr + cu
• cr – consistent gets. This statistic includes cr (d)
• cr (d) – consistent gets direct
• cu – current gets. This statistic includes cu (d)
• cu (d) – current gets direct
• ispnd (Y) – buffer is pinned count
• ispnd (N) – buffer is not pinned count
• pin rls – buffer pin is released

The list of objects is sorted by lio column.

Enjoy!

p.s.
Announcement: One of the next blog entry will be about statistics “buffer is [not] pinned count”. I will show what these statistics do mean, how it works and how DTraceLIO with new functionality might be used.
Coming soon.


]]> http://alexanderanokhin.wordpress.com/2012/03/19/dtrace-lio-new-features/feed/ 6 alexanderanokhin http://alexanderanokhin.wordpress.com/2011/11/13/dynamic-tracing-of-oracle-logical-io/ http://alexanderanokhin.wordpress.com/2011/11/13/dynamic-tracing-of-oracle-logical-io/#comments Sun, 13 Nov 2011 03:01:47 +0000 Alexander Anokhin http://alexanderanokhin.wordpress.com/?p=224 ]]> I would like to provide one more option to investigate Oracle logical I/O with DTrace.
This method allows to see each consistent/current gets with details about block, object and location from which function is called.

This is the DTrace script: dtracelio.d
Note: Dtrace LIO with new features is released

Short description:
This tool allows to see:
– details of each call (consistent/current get)
– details of blocks been read
– function from which call been performed without investigating of call stack (“where”)
– aggregation of performed calls

The tool can be especially helpful in 11g because x$kcbuwhy (x$kcbsw in 10g) is not populated properly there.
Note: Examples of using views x$kcbsw/x$kcbuwhy:
- Investigating Logical I/O by Jonathan Lewis
- Finding the reasons for excessive logical IOs by Tanel Poder


Current version of the script monitors execution of two Kernel Cache Layer functions:
kcbgtcr – Kernel Cache Buffer Get Consistent Read. Number of calls of this function we can see as statistic “consistent gets” and in 10046 trace as “cr=”.
kcbgcur – Kernel Cache Buffer Get Current Read. Number of calls of this function we can see as statistic “db block gets” in 10046 trace as “cu=”.
upd:
statistic “db block gets” is incremented not only by kcbgcur, but also by kcbget (included in current version of dtracelio), kcbnew (coming soon), kcblnb* (direct-path load) and some others. But kcbgcur is most often used.


In the both functions:
- the first argument is a pointer on the structure which describes a block;
- the second argument is unknown, but I could suggest that this is lock_mode (from MOS bug 7109078);
- the third argument (the least significant bits) is “where” (x$kcbwh), that is a module where the function is called.

Usage:
dtracelio.d PID [show_each_call]

PID – unix process ID
show_each_call – if 0 then output of each call will be disabled, else details of each call will be shown

A little example of usage. Let’s try to see what is going on during select on very small table:

SQL> @spid
 
       SID ORACLE_DEDICATED_PROCESS CLIENTPID
---------- ------------------------ ------------------------
        41 18949                    900:408
 
SQL> create table dualcopy as select * from dual;
 
Table created
 
SQL> select * from dualcopy;
 
DUMMY
-----
X

Notice, that I have executed “select * from dualcopy” to avoid hard parsing in consequences executions.

Let’s execute dtracelio

dtracelio.d 18949

I have executed the script with default parameter show_each_call, it means that details of each calls will be shown.

Now I am executing our query:

SQL> select * from dualcopy;
 
DUMMY
-----
X

And this is an output of dtracelio

kcbgtcr(0xFFFFFD7FFFDFB0F0,0,742,0) [tsn: 4 rdba: 0x100060a (4/1546) obj: 79218 dobj: 79218] where: 742
kcbgtcr(0xFFFFFD7FFFDFAED0,0,743,0) [tsn: 4 rdba: 0x100060a (4/1546) obj: 79218 dobj: 79218] where: 743
kcbgtcr(0xFFFFFD7FFDC3B2E0,0,860,0) [tsn: 4 rdba: 0x100060b (4/1547) obj: 79218 dobj: 79218] where: 860

You can see here:
kcbgtcr(0xFFFFFD7FFFDFB0F0,0,742,0) – it is function call with 3 arguments,
tsn: 4 – a tablespace number, ts# from v$tablespace
rdba: 0x100060a – a relative dba (data block address)
(4/1546) – file 4 block 1546
obj: 79218 – dictionary object number, object_id from dba_objects
dobj: 79218 – data object number, data_object_id from dba_objects
where: 742 – location from function (kcbgtcr in this case) was executed. This is INDX from x$kcbwf.

SQL> select indx, kcbwhdes from x$kcbwh where indx in (742, 743, 860);
 
      INDX KCBWHDES
---------- ----------------------------------------------------------------
       742 ktewh25: kteinicnt
       743 ktewh26: kteinpscan
       860 kdswh01: kdstgr

It is functions from which our function kcbgtcr was executed. It is parent function which you will see if will print call stack during the call of kcbgtcr.
If you would print call stack you will see (I’ve just added ustack() in the script):

kcbgtcr(0xFFFFFD7FFFDFB0F0,0,742,0) [tsn: 4 rdba: 0x100060a (4/1546) obj: 79218 dobj: 79218] where: 742

              oracle`kcbgtcr
              oracle`ktecgshx+0x3f6
              oracle`kteinicnt1+0x1cf
              oracle`qertbFetch+0xc2a
              oracle`opifch2+0xaa2
              oracle`opifch+0x3a
              oracle`opiodr+0x433
              oracle`ttcpip+0x599
              oracle`opitsk+0x600
              oracle`opiino+0x675
              oracle`opiodr+0x433
              oracle`opidrv+0x32e
              oracle`sou2o+0x57
              oracle`opimai_real+0x219
              oracle`ssthrdmain+0x14e
              oracle`main+0xcb
              oracle`0x159e67c

kcbgtcr(0xFFFFFD7FFFDFAED0,0,743,0) [tsn: 4 rdba: 0x100060a (4/1546) obj: 79218 dobj: 79218] where: 743

              oracle`kcbgtcr
              oracle`ktecgshx+0x3f6
              oracle`kteinpscan+0x1b6
              oracle`kteiniscan+0x2c
              oracle`kdselini+0x2f
              oracle`kdsirs1+0x792
              oracle`kdsirs+0x2b
              oracle`qertbFetch+0xbb0
              oracle`opifch2+0xaa2
              oracle`opifch+0x3a
              oracle`opiodr+0x433
              oracle`ttcpip+0x599
              oracle`opitsk+0x600
              oracle`opiino+0x675
              oracle`opiodr+0x433
              oracle`opidrv+0x32e
              oracle`sou2o+0x57
              oracle`opimai_real+0x219
              oracle`ssthrdmain+0x14e
              oracle`main+0xcb

kcbgtcr(0xFFFFFD7FFDC3B2E0,0,860,0) [tsn: 4 rdba: 0x100060b (4/1547) obj: 79218 dobj: 79218] where: 860

              oracle`kcbgtcr
              oracle`ktrget2+0x27d
              oracle`kdstgr+0x46b
              oracle`qertbFetch+0x466
              oracle`opifch2+0xaa2
              oracle`opifch+0x3a
              oracle`opiodr+0x433
              oracle`ttcpip+0x599
              oracle`opitsk+0x600
              oracle`opiino+0x675
              oracle`opiodr+0x433
              oracle`opidrv+0x32e
              oracle`sou2o+0x57
              oracle`opimai_real+0x219
              oracle`ssthrdmain+0x14e
              oracle`main+0xcb
              oracle`0x159e67c

So, 3 consistent gets, 3 logical reads were performed.
Two times block (4/1546) – this is segment header – has been read. The first time from function kteinicnt and the second time from function kteinpscan.
And the third get is block (4/1547) – this is data block – was read from function kdstgr.

After that, if you push Ctrl+C, dtracelio script will be finished, and you will see Summary section:

^C

========================= Summary ==========================
object_id    data_object_id  function     where        count
79218        79218           kcbgtcr      742          1
79218        79218           kcbgtcr      743          1
79218        79218           kcbgtcr      860          1

Summary section is aggregation of all calls grouped by object_id, data_object_id, function, where.

example 2
There was question on sql.ru. The author asked “why number of current gets (db block gets) during update in distributed transaction is more then in usual, non-distributed transaction?”.
In his case it were:
3070178 current gets in non-distributed transaction
6070211 current gets in distributed transaction

So, let’s find out what exactly is read during distributed transaction and is not read in usual, non-distributed transaction.

SQL> select * from v$version;
 
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE	11.2.0.2.0	Production
TNS for Solaris: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

SQL> @spid
 
       SID ORACLE_DEDICATED_PROCESS CLIENTPID
---------- ------------------------ ------------------------
         1 16727                    900:408

SQL> create table test as select * from all_objects;
 
Table created

SQL> alter session set optimizer_dynamic_sampling=0;
 
Session altered

SQL> @10046
 
Session altered

Here we are executing dtracelio.d:

dtracelio.d 16727 0

Here I have asked to show me only summary section of executions of functions performing logical I/O by process 16727.

1st update, without distributed transaction

SQL> begin
  2      update test set owner = owner;
  3      commit;
  4  end;
  5  /
 
PL/SQL procedure successfully completed

After the update has been completed we can push Ctrl+C to finish the DTrace and see the Summary section:

^C
========================= Summary ==========================
object_id    data_object_id  function     where        count
...
0            -1              kcbgcur      48           16
0            -1              kcbgcur      156          16
17           17              kcbgtcr      869          18
0            -1              kcbgcur      51           253
0            -1              kcbgcur      79           1365
79204        79204           kcbgcur      878          1656
0            -1              kcbgcur      50           2357
-1           79204           kcbgcur      859          2357
79204        79204           kcbgtcr      869          2726
79204        79204           kcbgcur      1053         75105

An excerpt from raw 10046 trace:

=====================
PARSING IN CURSOR #18446741324891442008 len=60 dep=0 uid=0 oct=47 lid=0 tim=229288260276 hv=2516922591 ad='88f557c0' sqlid='6sj578yb0ac6z'
begin
    update test set owner = owner;
    commit;
end;
END OF STMT
....
EXEC #18446741324891442008:c=2780000,e=4884618,p=0,cr=2776,cu=84545,mis=0,r=1,dep=0,og=1,plh=0,tim=229293144944

Pay attention, cu=84545.

Now let’s execute the same update within distributed transaction

SQL> declare
  2    i integer;
  3  begin
  4  
  5    select count(*) into i from dual@dblink;
  6    update test set owner = owner;
  7    commit;
  8  
  9  end;
 10  /

PARSING IN CURSOR #18446741324891442008 len=122 dep=0 uid=0 oct=47 lid=0 tim=229346672450 hv=1954107067 ad='88f04bc0' sqlid='arss8t9u7kmpv'
declare
  i integer;
begin
  select count(*) into i from dual@dblink;
  update test set owner = owner;
  commit;
end;
END OF STMT
...
EXEC #18446741324891442008:c=2950000,e=3432722,p=0,cr=2298,cu=159479,mis=0,r=1,dep=0,og=1,plh=0,tim=229350105247

As we can see cu=159479, about in two times more then in previous execution.

dtracelio.d output:

========================= Summary ==========================
object_id    data_object_id  function     where        count
...
0            -1              kcbgcur      48           16
0            -1              kcbgcur      156          16
17           17              kcbgtcr      869          18
0            -1              kcbgcur      51           49
0            -1              kcbgcur      50           442
-1           79204           kcbgcur      859          442
0            -1              kcbgcur      79           1161
79204        79204           kcbgcur      878          1177
79204        79204           kcbgtcr      869          2248
79204        79204           kcbgcur      1053         75105
0            -1              kcbgcur      109          79853

Pay attention on the bottom row

0            -1              kcbgcur      109          79853

79853 additional db block gets of some undo block which does not exist in previous output.

If we take a look at the output of dtracelio.d with enabled output of each call we will see something like this:

...
kcbgcur(0xFFFFFD7FFDC42D68,2,1053,0) [tsn: 0 rdba: 0x4186e6 (1/100070) obj: 79204 dobj: 79204] where: 1053
kcbgcur(0xFFFFFD7FFFDF46A0,1,109,0) [tsn: 2 rdba: 0xc000f0 (3/240) obj: 0 dobj: -1] where: 109
kcbgcur(0xFFFFFD7FFDC42D68,2,1053,0) [tsn: 0 rdba: 0x4186e6 (1/100070) obj: 79204 dobj: 79204] where: 1053
kcbgcur(0xFFFFFD7FFFDF46A0,1,109,0) [tsn: 2 rdba: 0xc000f0 (3/240) obj: 0 dobj: -1] where: 109
kcbgcur(0xFFFFFD7FFDC42D68,2,1053,0) [tsn: 0 rdba: 0x4186e6 (1/100070) obj: 79204 dobj: 79204] where: 1053
kcbgcur(0xFFFFFD7FFFDF46A0,1,109,0) [tsn: 2 rdba: 0xc000f0 (3/240) obj: 0 dobj: -1] where: 109
kcbgcur(0xFFFFFD7FFDC42D68,2,1053,0) [tsn: 0 rdba: 0x4186e6 (1/100070) obj: 79204 dobj: 79204] where: 1053
kcbgcur(0xFFFFFD7FFFDF46A0,1,109,0) [tsn: 2 rdba: 0xc000f0 (3/240) obj: 0 dobj: -1] where: 109
kcbgcur(0xFFFFFD7FFDC42D68,2,1053,0) [tsn: 0 rdba: 0x4186e6 (1/100070) obj: 79204 dobj: 79204] where: 1053
kcbgcur(0xFFFFFD7FFFDF46A0,1,109,0) [tsn: 2 rdba: 0xc000f0 (3/240) obj: 0 dobj: -1] where: 109
kcbgcur(0xFFFFFD7FFDC42D68,2,1053,0) [tsn: 0 rdba: 0x4186e6 (1/100070) obj: 79204 dobj: 79204] where: 1053
...

Here we can see these additional calls “where: 109″ and that the same block (3/240) is read during these calls.
Here is only excerpt, but I checked, all 79853 calls for the same block from the same “where” (109 in this case).
Update: updated version of DTraceLIO allows to see that these current gets were performed in shared mode, so content of the block is not changed.

Let’s try to look at inside the block.

alter system dump datafile 3 block 240;

An excerpt from block dump:

...
Block dump from disk:
buffer tsn: 2 rdba: 0x00c000f0 (3/240)
scn: 0x0000.0015240d seq: 0x02 flg: 0x04 tail: 0x240d2602
frmt: 0x02 chkval: 0x54dc type: 0x26=KTU SMU HEADER BLOCK
...

So, this is the undo header.

What is “where: 109″? “Where” means location in code, let’s try to find function name with indx 109

SQL> select indx, kcbwhdes from x$kcbwh where indx = 109;
 
      INDX KCBWHDES
---------- ----------------------------------------------------------------
       109 ktuwh87: ktugus:ktuGetExtTxnInfo

It is exactly what you will see if print call stack during call with “where: 109″ (the second row, function from kcbgcur was called is ktuGetExtTxnInfo):

 
 0000000002c63d01 kcbgcur () + 1
 00000000029272d4 ktuGetExtTxnInfo () + 174
 00000000028f5193 ktugti () + 13
 00000000028e2561 ktuchg2 () + 1971
 0000000002944d2f ktbchg2 () + 12f
 0000000001b04809 kdu_array_flush_retry () + c69
 0000000001b01f42 kdu_array_buf () + 522
 0000000001af5390 kduurp () + 460
 0000000001adea00 kdusru () + 13e0
 0000000001ac6c12 kauupd () + 1b2
 0000000005a3370c updrow () + a5c
 0000000007b8bd27 qerupFetch () + 397
 000000000367cfdc qerstFetch () + 58c
 0000000005a3f2fe updaul () + 4ce
 0000000005a4469d updThreePhaseExe () + 1cd
 0000000005a43abd updexe () + 1ed
 000000000447b2c5 opiexe () + 2455
...

It were for 11.2.0.2. In my 10.2.0.5 all these additional calls are with “where: 383″, “ktuwh02: ktugus” which Jonathan Lewis called “Get Undo Segment header for commit” (Seems it is not correct descriptoin).
And following call stack is during these calls

              oracle`kcbgcur
              oracle`ktugusc+0x321
              oracle`ktugti+0xf2
              oracle`ktuchg+0x139a
              oracle`ktbchg2+0x115
              oracle`kddchg+0x2c2
              oracle`kddlok+0x7da
              oracle`kddlkr+0x16a
              oracle`updrow+0x2417
              oracle`qerupRowProcedure+0x4f
              oracle`qerupFetch+0x339
              oracle`updaul+0x481
              oracle`updThreePhaseExe+0xc72
              oracle`updexe+0x171
              oracle`opiexe+0xf1b
              oracle`opipls+0x98e
              oracle`opiodr+0x433
              oracle`rpidrus+0xde
              oracle`skgmstack+0x80
              oracle`rpidru+0x86

Conclusion
We have determined what exactly are extra current gets within distributed transaction.
Why Oracle 79853 times reads undo header is a topic for further research. I would prefer to write another post about it.

Little oftopic, but interesting point is that Oracle opens distributed transaction (binds undo header, transaction appears in v$transaction, v$global_transaction) for PLSQL block with dblinks during hard parsing. If hard parsing is not performed then during parsing/execution of statement with dblink.
It follows that if you will execute a statement like this

declare
    i integer;
begin
    DML;
    DML;
    DML;
    if 1=0 then
        select count(*) into i from dual@dblink;
    end if;
end;

then either it will be executed within dustributed transaction (with extra current gets) or not depends on will hard parsing be performed or not.

p.s.
Attention!
I am still looking for additional information about
* structure kcbds,
* argments of functions kcbgtcr
* argments of functions kcbgcur
If you have this information please let me know.


]]> http://alexanderanokhin.wordpress.com/2011/11/13/dynamic-tracing-of-oracle-logical-io/feed/ 11 alexanderanokhin