At first we will find one row in an unrelated noise table, which is hashed to the same CBC latch as index root block.
In the first session, we update that unrelated row, and stop it by suspending CBC latch request.
In the second session, insert one row into our test table to trigger a leaf block split, which results in root block modification (insert of new split leaf block). Since they share the same CBC latch, the second session is also blocked.
Consequently any other session (e.q. SMON, DB Writer) which requires that CBC latch is also blocked.
Note: Tested in Oracle 12c, 19c
1. Test Setup
Same as previous blog, we create a table with one unique key reverse index (usually recommended to reduce index contention):
drop table test_tab purge;
create table test_tab (id number, val number);
create unique index test_tab#r on test_tab(id) reverse;
alter table test_tab add constraint test_tab#r primary key (id);
SQL > exec leaf_split_finder(5000);
--- Leaf Splitted at row count: 4906, (id between 1 and 11) in 2 Blocks
block: 2661263, cnt: 5, min_id: 1, max_id: 5
block: 3665621, cnt: 6, min_id: 6, max_id: 11
drop table cbc_noise purge;
create table cbc_noise
as select level x, rpad('CBC_NOISE', 100, 'X') y, level seq from dual connect by level < 1e6;
--- insert more rows by:
-- insert into cbc_noise select * from cbc_noise;
commit;
create index cbc_noise#i#1 on cbc_noise(x, y);
exec dbms_stats.gather_table_stats(null, 'CBC_NOISE', cascade=> true);
select segment_name, header_file, header_block, (header_block+1) root_block
from dba_segments where segment_name = 'TEST_TAB#R';
SEGMENT_NAME HEADER_FILE HEADER_BLOCK ROOT_BLOCK
------------ ----------- ------------ ----------
TEST_TAB#R 1548 2661258 2661259
select object_name, object_id, to_char(object_id, 'XXXXXXXX') id_hex, data_object_id, to_char(data_object_id, 'XXXXXXXX') did_hex
from dba_objects where object_name in ('TEST_TAB', 'TEST_TAB#R');
OBJECT_NAME OBJECT_ID ID_HEX DATA_OBJECT_ID DID_HEX
----------- --------- ------ -------------- -------
TEST_TAB 2459814 2588A6 2464078 25994E
TEST_TAB#R 2464076 25994C 2464077 25994D
select object_name, object_id, to_char(object_id, 'XXXXXXXX') id_hex, data_object_id, to_char(data_object_id, 'XXXXXXXX') did_hex
from dba_objects where object_name in ('CBC_NOISE', 'CBC_NOISE#I#1');
OBJECT_NAME OBJECT_ID ID_HEX DATA_OBJECT_ID DID_HEX
------------- --------- ------ -------------- -------
CBC_NOISE 2464039 259927 2464067 259943
CBC_NOISE#I#1 2464043 25992B 2464066 259942
2. Prepare Test
With above index root block number (2661259), we can find noise table index blocks which share the same CBC latch.
select * from (
with sq as (select object_name, data_object_id
from dba_objects
where object_name in ('TEST_TAB#R', 'CBC_NOISE#I#1')
)
,bh as (select hladdr, obj, file#, dbablk, sum(tch) tch
from sys.x_bh group by hladdr, obj, file#, dbablk)
select hladdr cbc_latch_addr
,sum(tch) tch
,listagg(tch || '-' || obj || '(' || object_name || ')/' || file# || '/' ||dbablk, ';')
within group (order by tch desc) tch_list -- "tch-obj(name)/file/blk_list"
,count(*) blk_cnt
from bh, sq
where bh.obj = sq.data_object_id
and tch > 0
-- and (hladdr like '%18B6072A8%')
group by hladdr
order by tch desc)
where 1=1
and (tch_list like '%TEST_TAB#R%2661259%')
;
CBC_LATCH_ADDR TCH TCH_LIST BLK_CNT
---------------- --- -------------------------------------- -------
000000018B6072A8 7 3-2464048(TEST_TAB#R) /1548/2661259 3
2-2464066(CBC_NOISE#I#1)/1548/1482751
2-2464066(CBC_NOISE#I#1)/1548/1511408
Take one CBC_NOISE#I#1 block, for example, 1482751, we can list all its indexed table rows.
select blk, x from (
select x
,dbms_rowid.rowid_block_number(sys_op_lbid (2464043, 'L', rowid)) blk
from cbc_noise)
where blk in (1482751);
BLK X
---------- ----------
1482751 200656
1482751 200657
...
60 rows selected.
truncate table test_tab;
insert into test_tab select level, -level from dual connect by level <= 4905;
commit;
-- using above index object_id to substitute for 2458815
select blk, count(*), min(id) min_id, max(id) max_id from (
select id
,dbms_rowid.rowid_block_number(sys_op_lbid (2464076, 'L', rowid)) blk
from test_tab)
where id between 1 and 11
group by blk order by min_id, blk;
BLK COUNT(*) MIN_ID MAX_ID
------- -------- ------ ------
2661263 11 1 11
3. Blocking Test
Open 3 Sqlplus sessions.
3.1 Session-1: Update cbc_noise
At first, start appended dtrace_stop.d to trace Session-1 on above shared CBC latch (same Dtrace script as Blog: Cache Buffer Chains Latch Contention Case Study-1: Reverse Key Index ).
Then in Session-1 (SPID 8438, SID 184), update one found row of cbc_noise, for example, x = 200656:
SQL (SPID 8438, SID 184) > update cbc_noise set y = 'cbc_test' where x = 200656;
oracle@testdb > ksh ./dtrace_stop.d 8438 0x18B6072A8
dtrace: allowing destructive actions
CPU ID FUNCTION:NAME
2 81245 ksl_get_shared_latch:entry
---------Step 1 at 2020 Apr 5 06:33:03 (Elapsed-ns=1580535694)---------
***State Entry>>>(nproc_pid=>0, flag=>0, gets=>86515, latch=>228, level=>1, where=>2195)
>>>Transition>>>ksl_get_shared_latch:entry
(Addr=>0x18B6072A8, Immed_Get=>1, Why=>0, Where=>2141(0x85D), Mode=>16, PID=>0x200000000000001A)
oracle`ksl_get_shared_latch
a.out`kcbget+0x3411
a.out`ktbxchg+0x10c
a.out`kdifind+0x504
a.out`kdimod0+0x2360
a.out`kauXuFinish+0x1a8
a.out`kxibFinish+0x2bc
oracle`qerupFetch+0x993
a.out`qerstFetch+0x543
oracle`updaul+0x92c
oracle`updThreePhaseExe+158
oracle`updexe+307
oracle`opiexe+2c55
oracle`kpoal8+a45
2 81246 ksl_get_shared_latch:return
>>>State Return***(nproc_pid=>26, flag=>536870912, gets=>86516, latch=>228, level=>1, where=>2141)
5 2 :END
---------Step 2 at 2020 Apr 5 06:33:03 (Elapsed-ns=420179764)---------
Latch Gets = 0 (86515 - 86515)
(where=>2195: kcbgtcr: fast path pin
where=>2141: kcbget: new pin)
3.2 Session-2: Insert to Trigger Leaf Block Split and Root Block Modification
At first, start appended dtrace_kslgess.d to trace Session-2 on shared latch (same Dtrace script as Blog: Cache Buffer Chains Latch Contention Case Study-2: Reverse Primary Key Index).
Then in Session-2 (SPID 8668, SID 367), insert one row into test_tab. Immediately it is blocked by Session-1 in Wait Event: 'latch: cache buffers chains'.
SQL (SPID 8668, SID 367) >> insert into test_tab values(4906, -4906);
oracle@testdb > ksh ./dtrace_kslgess.d 8668 0x18B6072A8
dtrace: allowing destructive actions
CPU ID FUNCTION:NAME
0 81263 kslgess:entry
---------Step 1 at 2020 Apr 5 06:33:43 (Elapsed-ns=19370786619)---------
***State Entry>>>(nproc_pid=>26, flag=>536870912, gets=>86516, latch=>228, level=>1, where=>2141)
>>>Transition>>>kslgess:entry
(Addr=>0x18B6072A8, Where=>2196)
oracle`kslgess
oracle`kcbgcur+0xd45
oracle`ktbgcur+0xb4
a.out`kdifind+0x39a
a.out`kdiins1+0xa3c
a.out`kdiinsp0+0x7b
a.out`kauxsin+0x799
a.out`qesltcLoadIndexList+0x36e
a.out`qerltcSimpleSingleInsRowCBK+0x5a
a.out`qerltcSingleRowLoad+0x150
3.3 Session-3: Select Statement
In Session-3, we start one select on test_tab, and it is also blocked by Session-1 in Wait Event: 'latch: cache buffers chains'.
SQL (SPID 8670, SID 548) > select * from test_tab where id = 234;
4. System-Wide Blocking
Let system run with the above blocking for a while. One hour later, run a query on v$wait_chains, order them by blocking sequence (IN_WAIT_SECS). It shows SMON, DB Writer and other background processes also blocked in wait event: 'latch: cache buffers chains' or 'buffer busy waits' (file#=3 is UNDO file).
select (select s.program from v$session s where sid=w.sid) program
,chain_signature, osid, pid, sid
,blocker_is_valid bvalid, blocker_sid, p1, p1_text, in_wait_secs, num_waiters
from v$wait_chains w
order by in_wait_secs desc nulls first;
PROGRAM CHAIN_SIGNATURE OSID PID SID BVALID BLOCKER_SID P1 P1_TEXT IN_WAIT_SECS NUM_WAITERS
------- -------------------------------------------------- ---- --- --- ------ ----------- ---------- ------- ------------ -----------
sqlplus 'latch: cache buffers chains'<='buffer busy waits' 8438 26 184 FALSE 7
sqlplus 'latch: cache buffers chains' 8668 25 367 TRUE 367 6633321128 address 6,286 0
sqlplus 'latch: cache buffers chains' 8670 27 548 TRUE 367 6633321128 address 6,257 0
(DBW0) 'latch: cache buffers chains' 15528 11 897 TRUE 367 6633321128 address 6,106 0
(M001) 'latch: cache buffers chains'<='buffer busy waits' 12184 32 368 TRUE 367 6633321128 address 4,675 3
(SMON) 'latch: cache buffers chains'<='buffer busy waits' 15540 15 541 TRUE 368 3 file# 4,375 0
(W002) 'latch: cache buffers chains'<='buffer busy waits' 12186 23 903 TRUE 368 3 file# 4,374 0
(M000) 'latch: cache buffers chains'<='buffer busy waits' 12474 36 5 TRUE 368 3 file# 245 0
IN_WAIT_SECS indicates the blocking sequence.
NUM_WAITERS shows number of sessions blocked by this session. In above example, 4 are directly blocked, 3 indirectly blocked (blocking chain).
Appendex-3 lists their Call Stacks.
5. Blocking Objects and Blocks
To terminate the blocking, run command "prun 8438" to terminate the blocking.
With above shared latch address 0x18B6072A8, we can run a query to list its protected objects and blocks currently in use:
select * from (
with sq as (select object_name, data_object_id
from dba_objects
-- where object_name in ('TEST_TAB#R', 'CBC_NOISE#I#1')
)
,bh as (select hladdr, obj, file#, dbablk, sum(tch) tch
from sys.x_bh group by hladdr, obj, file#, dbablk)
select hladdr cbc_latch_addr
,sum(tch) tch
,listagg(tch || '-' || obj || '(' || object_name || ')/' || file# || '/' ||dbablk, ';')
within group (order by tch desc) tch_list -- "tch-obj(name)/file/blk_list"
,count(*) blk_cnt
from bh, sq
where bh.obj = sq.data_object_id
and tch > 0
and (hladdr like '%18B6072A8%')
group by hladdr
order by tch desc)
;
CBC_LATCH_ADDR TCH TCH_LIST BLK_CNT
---------------- --- -------------------------------------- -------
000000018B6072A8 9 2-2464089(TEST_TAB#R) /1548/2661259 5
4-2464066(CBC_NOISE#I#1) /1548/1482751
1-8 (C_FILE#_BLOCK#)/1/111481
1-8 (SEG$) /1/111481
1-8 (UET$) /1/111481
In summary, this Blog is trying to manually hold a CBC latch to simulate CBC latch contention. It demonstrated that one unrelated DML can block other index root (branch) operations by CBC latch, and even has system-wide impacts.
6. Index Root Block CBC Latch Contention Test
Similar to two previous Blog: Cache Buffer Chains Latch Contention Case Study-1: Reverse Key Index and Cache Buffer Chains Latch Contention Case Study-2: Reverse Primary Key Index, we start 12 Job sessions to access test_tab:
exec test_tab_select_job(6, 1e9, 1, 0);
exec test_tab_select_job(6, 1e9, 6, 0);
-- stop all jobs by:
-- exec clean_jobs;
begin
for i in 1..1e9 loop
update cbc_noise set y = 'cbc_test' where x = 200656;
end loop;
end;
/
begin
for i in 1..1e9 loop
for c in (select * from test_tab where id = 234) loop
null;
end loop;
end loop;
end;
/
select s.program, s.client_identifier, s.sid, s.serial#, p.pid, p.spid, s.event, p1, p2, p3, latchwait, latchspin
from v$session s, v$process p
where s.paddr=p.addr and (lower(s.program) like '%j0%' or lower(s.program) like '%sql%')
order by s.client_identifier;
PROGRAM CLIENT_IDENTIFIER SID SERIAL# PID SPID EVENT P1 P2 P3
-------------------- -------------------- --- ------- --- ---- --------------------------- ---------- --- --
oracle@testdb (J000) test_tab_select_id_1 369 58913 26 21665 latch: cache buffers chains 6633321128 228 0
...
oracle@testdb (J006) test_tab_select_id_6 546 11249 39 21675 latch: cache buffers chains 6633321128 228 0
...
sqlplus.exe 727 61279 46 21740 latch: cache buffers chains 6633321128 228 0
sqlplus.exe 729 40349 44 21917 latch: cache buffers chains 6633321128 228 0
select * from v$latch_misses
where parent_name = 'cache buffers chains' and sleep_count > 0
order by sleep_count desc;
WHERE NWFAIL_COUNT SLEEP_COUNT WTR_SLP_COUNT LONGHOLD_COUNT LOCATION
------------------------ ------------ ----------- ------------- -------------- -----------------------
kcbgtcr: fast path exam 0 5283 4592 840 kcbgtcr: fast path exam
kcbso1: set no access 0 477 1236 18 kcbso1: set no access
kcbgtcr: fast path pin 0 66 34 22 kcbgtcr: fast path pin
kcbgcur: fast path excl 0 38 11 24 kcbgcur: fast path excl
kcbget: new pin 0 33 0 21 kcbget: new pin
kcbchg: cr pin change 0 10 2 5 kcbchg: cr pin change
kcbchg: change complete 0 5 8 4 kcbchg: change complete
select * from v$latchholder;
PID SID LADDR NAME GETS
--- --- ---------------- -------------------- ----------
24 10 000000018B6072A8 cache buffers chains 142727994
select latch#, child#, addr, gets, misses, sleeps, spin_gets, wait_time, round(wait_time/sleeps) us_per_sleep
from v$latch_children
where addr = hextoraw('000000018B6072A8');
LATCH# CHILD# ADDR GETS MISSES SLEEPS SPIN_GETS WAIT_TIME US_PER_SLEEP
------ ------ ---------------- --------- -------- ------- --------- ---------- ------------
228 15672 000000018B6072A8 191753377 1460415 6280 1454291 27969528879 4453747
7. Appendix
7.1 dtrace_stop.d
sudo dtrace -w -n \
'
typedef unsigned int ub4;
typedef struct latchstate /* latch state */ {
ub4 nproc_pid; /* Nproc or pid */
ub4 flag; /* blocking flag */
ub4 gets; /* number of gets */
ub4 latch; /* latch number */
ub4 level; /* latch level */
ub4 where; } latchstate;
BEGIN {laddr = $1; step = 1; begin_gets = 0, end_gets = 0; prev_ts = timestamp;
self->evt = "begin"; self->loc = 0x0;}
pid$target::ksl_get_shared_latch:entry /arg0 == laddr && (self->evt != probefunc || self->loc != arg3)/ {
printf("\n---------Step %d at %Y (Elapsed-ns=%d)---------", step, walltimestamp, (timestamp-prev_ts));
ls = ((latchstate *) copyin(arg0, sizeof(latchstate)));
begin_gets = (begin_gets == 0 ? ls->gets: begin_gets); end_gets = ls->gets;
printf("\n***State Entry>>>(nproc_pid=>%d, flag=>%d, gets=>%d, latch=>%d, level=>%d, where=>%d)"
,ls->nproc_pid, ls->flag, ls->gets, ls->latch, ls->level, ls->where);
printf("\n>>>Transition>>>%s:%s\n (Addr=>0x%X, Immed_Get=>%d, Why=>%d, Where=>%d(0x%X), Mode=>%d, PID=>0x%X)",
probefunc, probename, arg0, arg1, arg2, arg3, arg3, arg4, arg5);
self->evt = probefunc; self->loc = arg3;
ustack(14, 0);
/* stop(); exit(0); */ /* no latch blocking, failed get, try again */
step = step + 1; self->entry = 1; prev_ts = timestamp;}
pid$target::ksl_get_shared_latch:return /self->entry == 1/ {
ls = ((latchstate *) copyin(laddr, sizeof(latchstate)));
printf("\n>>>State Return***(nproc_pid=>%d, flag=>%d, gets=>%d, latch=>%d, level=>%d, where=>%d)"
,ls->nproc_pid, ls->flag, ls->gets, ls->latch, ls->level, ls->where);
stop(); exit(0); /* simulate long latch holder, run by: dtrace -w -n */
self->entry = 2;}
pid$target::kslfre:entry /self->entry == 2 && arg0 == laddr && self->evt != probefunc/ {
printf("\n---------Step %d at %Y (Elapsed-ns=%d)---------", step, walltimestamp, (timestamp-prev_ts));
ls = ((latchstate *) copyin(arg0, sizeof(latchstate)));
end_gets = ls->gets;
printf("\n>>>Transition>>>%s:%s(Addr=>0x%-X)", probefunc, probename, arg0);
self->evt = probefunc; self->loc = 0x0;
/* stop(); exit(0); */ /* simulate long latch holder, run by: dtrace -w -n */
printf("\n===Latch freed===");
ustack(10, 0); step = step + 1; prev_ts = timestamp;}
END {
printf("\n---------Step %d at %Y (Elapsed-ns=%d)---------", step, walltimestamp, (timestamp-prev_ts));
printf("\nLatch Gets = %d (%d - %d)", (end_gets - begin_gets), (unsigned long)begin_gets, (unsigned long)end_gets); }
' -p $1 $2
7.2 dtrace_kslgess.d
sudo dtrace -w -n \
'
typedef unsigned int ub4;
typedef struct latchstate /* latch state */ {
ub4 nproc_pid; /* Nproc or pid */
ub4 flag; /* blocking flag */
ub4 gets; /* number of gets */
ub4 latch; /* latch number */
ub4 level; /* latch level */
ub4 where; } latchstate;
BEGIN {laddr = $1; step = 1; begin_gets = 0, end_gets = 0; prev_ts = timestamp;
self->evt = "begin"; self->loc = 0x0;}
pid$target::kslgess:entry /arg0 == laddr && (self->evt != probefunc || self->loc != arg4)/ {
printf("\n---------Step %d at %Y (Elapsed-ns=%d)---------", step, walltimestamp, (timestamp-prev_ts));
ls = ((latchstate *) copyin(arg0, sizeof(latchstate)));
begin_gets = (begin_gets == 0 ? ls->gets: begin_gets); end_gets = ls->gets;
printf("\n***State Entry>>>(nproc_pid=>%d, flag=>%d, gets=>%d, latch=>%d, level=>%d, where=>%d)"
,ls->nproc_pid, ls->flag, ls->gets, ls->latch, ls->level, ls->where);
printf("\n>>>Transition>>>%s:%s\n (Addr=>0x%X, Where=>%d)",
probefunc, probename, arg0, arg4);
self->evt = probefunc; self->loc = arg4;
ustack(10, 0);
/*stop(); exit(0);*/ /* no latch blocking, failed get, try again */
step = step + 1; self->entry = 1; prev_ts = timestamp;}
pid$target::kslgess:return /self->entry == 1/ {
ls = ((latchstate *) copyin(laddr, sizeof(latchstate)));
printf("\n>>>State Return***(nproc_pid=>%d, flag=>%d, gets=>%d, latch=>%d, level=>%d, where=>%d)"
,ls->nproc_pid, ls->flag, ls->gets, ls->latch, ls->level, ls->where);
/*stop(); exit(0);*/ /* simulate long latch holder, run by: dtrace -w -n */
self->entry = 2;}
pid$target::kslfre:entry /self->entry == 2 && arg0 == laddr && self->evt != probefunc/ {
printf("\n---------Step %d at %Y (Elapsed-ns=%d)---------", step, walltimestamp, (timestamp-prev_ts));
ls = ((latchstate *) copyin(arg0, sizeof(latchstate)));
end_gets = ls->gets;
printf("\n>>>Transition>>>%s:%s(Addr=>0x%-X)", probefunc, probename, arg0);
self->evt = probefunc; self->loc = 0x0;
/*stop(); exit(0);*/ /* simulate long latch holder, run by: dtrace -w -n */
printf("\n===Latch freed===");
ustack(10, 0); step = step + 1; prev_ts = timestamp;}
END {
printf("\n---------Step %d at %Y (Elapsed-ns=%d)---------", step, walltimestamp, (timestamp-prev_ts));
printf("\nLatch Gets = %d (%d - %d)", (end_gets - begin_gets), (unsigned long)begin_gets, (unsigned long)end_gets); }
' -p $1 $2
7.3 Call Stacks
7.3.1 Session-1 (SID 184) 'latch: cache buffers chains'<='buffer busy waits'
oracle@testdb > pstack 8438
8438: testdb (LOCAL=NO)
------------ lwp# 1 / thread# 1 ---------------
00000000059e3711 kcbget () + 3411
00000000059e01cc ktbxchg () + 10c
00000000059de834 kdifind () + 504
00000000059d70e0 kdimod0 () + 2360
000000000d986c88 kauXuFinish () + 1a8
000000000f5485ac kxibFinish () + 2bc
0000000005729233 qerupFetch () + 993
00000000092cc193 qerstFetch () + 543
00000000057234ec updaul () + 92c
0000000005720428 updThreePhaseExe () + 158
000000000571f6d7 updexe () + 307
0000000005707995 opiexe () + 2c55
0000000005c73275 kpoal8 () + a45
00000000056f87e3 opiodr () + 433
0000000005701323 ttcpip () + 593
00000000056fd59c opitsk () + 6cc
00000000056fc405 opiino () + 3c5
00000000056f87e3 opiodr () + 433
00000000056f7dcb opidrv () + 33b
00000000056f79fa sou2o () + 7a
00000000056f7775 opimai_real () + 265
00000000056f70be ssthrdmain () + 22e
00000000056f6e64 main () + a4
7.3.2 Session-2 (SID 367) 'latch: cache buffers chains'
----- without using dtrace_kslgess.d, only use pstack
oracle@testdb > pstack 8668
8668: testdb (LOCAL=NO)
------------ lwp# 1 / thread# 1 ---------------
ffff80ffbbf93d4b semsys (2, 27, ffff80ffbffe21e8, 1, a45)
000000000578adc5 sskgpwwait () + 1e5
000000000578a965 skgpwwait () + c5
000000000578972d kslgess () + 92d
00000000057a9be5 kcbgcur () + d45
00000000057eecb4 ktbgcur () + b4
00000000059de6ca kdifind () + 39a
0000000005a224fc kdiins1 () + a3c
0000000005a21a5b kdiinsp0 () + 7b
0000000005a1e269 kauxsin () + 799
0000000005a1c42e qesltcLoadIndexList () + 36e
0000000005ae06fa qerltcSimpleSingleInsRowCBK () + 5a
0000000005a1b210 qerltcSingleRowLoad () + 150
0000000005a19f36 qerltcFetch () + 1d6
00000000092cc11a qerstFetch () + 4ca
0000000005a19297 insexe () + 407
00000000057064df opiexe () + 179f
0000000005c73275 kpoal8 () + a45
00000000056f87e3 opiodr () + 433
0000000005701323 ttcpip () + 593
00000000056fd59c opitsk () + 6cc
00000000056fc405 opiino () + 3c5
00000000056f87e3 opiodr () + 433
00000000056f7dcb opidrv () + 33b
00000000056f79fa sou2o () + 7a
00000000056f7775 opimai_real () + 265
00000000056f70be ssthrdmain () + 22e
00000000056f6e64 main () + a4
7.3.3 oracle@testdb (DBW0 SID 897) 'latch: cache buffers chains'
oracle@testdb > pstack 15528
15528: ora_dbw0_testdb
------------ lwp# 1 / thread# 1 ---------------
ffff80ffbbf93d4b semsys (2, 27, ffff80ffbfff88a8, 1, 623f7)
000000000578adc5 sskgpwwait () + 1e5
000000000578a965 skgpwwait () + c5
000000000578972d kslgess () + 92d
00000000057889bf ksl_get_shared_latch () + 39f
00000000066154ec kcbbxsv () + 2fc
00000000066119f9 kcbb_coalesce () + 149
0000000008c389d5 kcbbwdl () + 655
000000000660b716 kcbbdrv () + df6
00000000065d9443 ksbcti () + 583
00000000065cfadd ksbabs () + 91d
00000000065cde36 ksbrdp () + 6c6
000000000627c9fc opirip () + 34c
00000000056f7d1b opidrv () + 28b
00000000056f79fa sou2o () + 7a
00000000056f764d opimai_real () + 13d
00000000056f70be ssthrdmain () + 22e
00000000056f6e64 main () + a4
7.3.4 oracle@testdb (SMON SID 541)'latch: cache buffers chains'<='buffer busy waits'
oracle@testdb > pstack 15540
15540: ora_smon_testdb
ffff80ffbbf93d4b semsys (4, 27, ffff80ffbfff70c8, 1, ffff80ffbfff70d0)
000000000578acd8 sskgpwwait () + f8
000000000578a965 skgpwwait () + c5
0000000005944ffc ksliwat () + 8dc
0000000005944350 kslwaitctx () + 90
0000000005955d2d kcbzwb () + 86d
00000000057aeadc kcbgcur () + 5c3c
00000000057a551a ktuGetUsegHdr () + 3da
00000000067ba665 kturRecoverUndoSegment () + 1f5
00000000067b87e3 kturRecoverActiveTxns1 () + 4d3
00000000067b8252 kturRecoverActiveTxns () + 22
00000000067b6697 ktprbeg1 () + 177
00000000067b645f ktprbeg () + 1f
00000000067b16fc ktmmon () + fbc
00000000067afe94 ktmSmonMain () + f4
00000000065cde36 ksbrdp () + 6c6
000000000627c9fc opirip () + 34c
00000000056f7d1b opidrv () + 28b
00000000056f79fa sou2o () + 7a
00000000056f764d opimai_real () + 13d
00000000056f70be ssthrdmain () + 22e
00000000056f6e64 main () + a4
7.3.5 oracle@testdb (W002 SID 903) 'latch: cache buffers chains'<='buffer busy waits'
oracle@testdb > pstack 12186
12186: ora_w002_testdb
ffff80ffbbf93d4b semsys (4, 27, ffff80ffbfff5e88, 1, ffff80ffbfff5e90)
000000000578acd8 sskgpwwait () + f8
000000000578a965 skgpwwait () + c5
0000000005944ffc ksliwat () + 8dc
0000000005944350 kslwaitctx () + 90
0000000005955d2d kcbzwb () + 86d
00000000057aeadc kcbgcur () + 5c3c
00000000057a551a ktuGetUsegHdr () + 3da
0000000005ae819a ktuGetExtTxnInfo () + 2ca
00000000062131dd ktuxcm () + 9d
00000000057a6f9d ktecgetsh () + 67d
00000000057a6903 ktecgshx () + 23
0000000005c39c26 kteinicnt3 () + 2e6
0000000005c3992c kteinicnt1 () + 2c
0000000008dea761 ktssupd_segment_extblks_bkg () + 511
0000000006287799 ktsj_task_switch () + 149
0000000006286100 ktsj_execute_task () + 2b0
0000000006284d6e ktsj_slave_main () + 42e
000000000627ddcd ksvrdp () + aed
000000000627c9ab opirip () + 2fb
00000000056f7d1b opidrv () + 28b
00000000056f79fa sou2o () + 7a
00000000056f764d opimai_real () + 13d
00000000056f70be ssthrdmain () + 22e
00000000056f6e64 main () + a4
7.3.6 oracle@testdb (M001 SID 368) 'latch: cache buffers chains'<='buffer busy waits'
oracle@testdb > pstack 12184
12184: ora_m001_testdb
------------ lwp# 1 / thread# 1 ---------------
ffff80ffbbf93d4b semsys (2, 27, ffff80ffbffb28a8, 1, f6)
000000000578adc5 sskgpwwait () + 1e5
000000000578a965 skgpwwait () + c5
000000000578972d kslgess () + 92d
00000000057889bf ksl_get_shared_latch () + 39f
000000000579b3bc kcbnew () + 3c9c
00000000057957c7 ktugnb () + 327
0000000005790ad8 ktugur () + a08
000000000574359e ktuchg2 () + 49e
0000000005743007 ktbchg2 () + 1d7
0000000005a81c6b kdisle () + 3ed3b
0000000005a25ae3 kdiins1 () + 4023
0000000005a1e8fc kauxsin () + e2c
0000000005a1c659 qesltcLoadIndexList () + 599
0000000005afff45 qerltcFlushKdtBuffer () + 225
0000000005b2e1c4 qerltcBufRow () + 74
0000000005b2dbd2 qerltcKdtBufferedInsRowCBK () + 152
0000000005afea47 qerltcLoadStateMachine () + 237
0000000005fd71b1 qerltcInsertSelectRop () + 101
00000000092cb832 qerstRowP () + 412
00000000092cb832 qerstRowP () + 412
0000000005d1c2f3 qerfxFetch () + c73
00000000092cc11a qerstFetch () + 4ca
0000000005c78d3f rwsfcd () + 6f
00000000092cc11a qerstFetch () + 4ca
0000000005a1a211 qerltcFetch () + 4b1
00000000092cc11a qerstFetch () + 4ca
0000000005a19297 insexe () + 407
00000000057064df opiexe () + 179f
0000000005c73275 kpoal8 () + a45
00000000056f87e3 opiodr () + 433
0000000005de3542 kpoodrc () + 22
000000000571a07d rpiswu2 () + 2fd
0000000005de3187 kpoodr () + 287
0000000005de272b upirtrc () + e0b
0000000005de18e0 kpurcsc () + 70
0000000005ddc2da kpuexec () + 1b0a
0000000005dda7bb OCIStmtExecute () + 2b
0000000005f1914f kewrgwxf1_gwrsql_exft_1 () + 3df
0000000005f180bb kewrgwxf_gwrsql_exft () + 23b
0000000009aabb65 kewrft_flush_table () + 4e5
0000000009aaaeab kewrftec_flush_table_ehdlcx () + c7b
0000000009ab9b3a kewrfat1_flush_all_tables_1 () + 42a
0000000009aba3ec kewrfat_flush_all_tables () + 3ec
0000000009abb5a6 kewrfop_flush_one_phase () + 226
0000000009abbedb kewrfap_flush_all_phases () + 7b
0000000009aa5eb2 kewrfos_flush_onesnap () + bf2
0000000009aa6b14 kewrfsc_flush_snapshot_c () + 564
0000000009aa4ad0 kewrcmdbs_flush_mdb_snap () + 460
0000000009af4283 kewrafs_auto_flush_slave () + 6e3
000000000633455c kebm_slave_main () + 29c
000000000627ddcd ksvrdp () + aed
000000000627c9ab opirip () + 2fb
00000000056f7d1b opidrv () + 28b
00000000056f79fa sou2o () + 7a
00000000056f764d opimai_real () + 13d
00000000056f70be ssthrdmain () + 22e
00000000056f6e64 main () + a4
