Friday, December 17, 2021

How to count the inserts and updates from merge ?

This blog will make two attempts to count the inserts and updates from merge without changing merge statement. One is by programming, another by Oracle stats. There are some discussions on this topic in AskTom: How to count the inserts and updates from merge.

Note: Tested in Oracle 19c


1. Test Setup



drop table source_tab;
drop table target_tab;
 
-- 7000 rows, x from 1 to 7000
create table target_tab as select level x, rpad('TTT', 2000, 'X') y from dual connect by level <= 7000;
 
-- 5000 rows, x from 3001 to 8000
create table source_tab as select 3000+level x, rpad('SSS', 2000, 'X') y from dual connect by level <= 5000;
 
alter table target_tab add (constraint target_tab#p primary key (x));
alter table source_tab add (constraint source_tab#p primary key (x));
 
exec dbms_stats.gather_table_stats(user, 'TARGET_TAB', cascade=> true);
exec dbms_stats.gather_table_stats(user, 'SOURCE_TAB', cascade=> true);


2. Test Run



declare
  l_time          date;
  l_merge_cnt     number;
  l_before_cnt    number;
  l_after_cnt     number;
  l_before_change number;
  l_before_insert number;
  l_after_change  number;
  l_after_insert  number;
  l_upd_oracle    number;
begin
  l_time := sysdate;
  select count(*) into l_before_cnt from target_tab;  -- assume no other DML on target_tab
 
  select b.value into l_before_change from v$statname a, v$mystat b where a.statistic# = b.statistic# and name = 'HSC Heap Segment Block Changes';
  select b.value into l_before_insert from v$statname a, v$mystat b where a.statistic# = b.statistic# and name =  'Heap Segment Array Inserts';
 
  merge /*+ index(a target_tab#p) */ into target_tab a
        --using source_tab b
        using (select /*+ index(t source_tab#p) */ * from source_tab t where x > 4000)  b
    on (a.x = b.x)
    when matched then
      update set a.y       = b.y
    when not matched then
      insert (x, y) values (b.x, b.y);
     
  l_merge_cnt := sql%rowcount;
 
  select b.value into l_after_change from v$statname a, v$mystat b where a.statistic# = b.statistic# and name = 'HSC Heap Segment Block Changes';
  select b.value into l_after_insert from v$statname a, v$mystat b where a.statistic# = b.statistic# and name =  'Heap Segment Array Inserts';
 
  --select count(*) into l_before_cnt from target_tab as of timestamp l_time;  -- sys.smon_scn_time.TIME_DP is DATE type
  select count(*) into l_after_cnt from target_tab;
 
  dbms_output.put_line('Merge sql%rowcount = '||l_merge_cnt);
  dbms_output.put_line('Table Rows before  = '||l_before_cnt);
  dbms_output.put_line('Table Rows after   = '||l_after_cnt);
 
  dbms_output.put_line('============= inserts and updates by Programmer =============');
  dbms_output.put_line('Insert Rows = '|| (l_after_cnt - l_before_cnt));
  dbms_output.put_line('Update Rows = '|| (l_merge_cnt - (l_after_cnt - l_before_cnt)));
 
  dbms_output.put_line('============= inserts and updates by Oracle =============');
  l_upd_oracle := (l_after_change  - l_before_change) - (l_after_insert - l_before_insert);
  dbms_output.put_line('Insert Rows = '|| (l_merge_cnt - l_upd_oracle));
  dbms_output.put_line('Update Rows = '|| l_upd_oracle);
 
  rollback;
end;
/


3. Test Output



Merge sql%rowcount = 4000
Table Rows before  = 7000
Table Rows after   = 8000
 
============= inserts and updates by Programmer =============
Insert Rows = 1000
Update Rows = 3000
 
============= inserts and updates by Oracle =============
Insert Rows = 1000
Update Rows = 3000

Sunday, November 28, 2021

Oracle dbms_aq.dequeue_array Shared Pool "KTC Latch Subh" Memory Leak

This Blog will try to demonstrate shared pool "KTC Latch Subh" memory leak when using AQ array dequeue: dbms_aq.dequeue_array (enqueue_array and dequeue_array introduced in 10g).

Note 1: Tested in Oracle 19c with following settings:
 
  -- disable autosga to generate ORA-04031
  alter system set "_memory_imm_mode_without_autosga"=false;
 
  -- set small shared_pool to quickly hit ORA-04031 due to "KTC latch subh"
  alter system set shared_pool_size=1008M scope=spfile;  
  
  -- set 3 subpools in shared_pool
  alter system set "_kghdsidx_count"=3 scope=spfile; 
 
Note 2: The behaviour was first observed by other people in Oracle applications.


1. "KTC latch subh" Memory Leak Test


At first, we run test_aq_loop (see section 4 "Test Code") to make 3 array_enqueue / array_dequeue calls with array_size: 1000, 2000, and 3000 one after another. Each time the same number of messages are first enqueued and then dequeued.

The output shows that 'KTC latch subh' memory is increased progressively from 0.5 MB to 6 MB. The increase only occurs in Dequeue_Array, but not Enqueue_Array.

SQL > select round(bytes/1024/1024, 2) KTC_MB from v$sgastat where name = 'KTC latch subh';
    KTC_MB
    ------
       .05
 
SQL > exec test_aq_loop('ArrayEnq',  'ArrayDeq',  3, 0, 1e3);
 
  RUN = 1 at 06:27:46
    Enqueue_Array.Size = 1000, KTC latch subh MB = .05
    Dequeue_Array.Size = 1000, KTC latch subh MB = 1.04
    Enqueue_Array.Size = 2000, KTC latch subh MB = 1.05
    Dequeue_Array.Size = 2000, KTC latch subh MB = 3.03
    Enqueue_Array.Size = 3000, KTC latch subh MB = 3.04
    Dequeue_Array.Size = 3000, KTC latch subh MB = 6.01
   
    Elapsed: 00:00:03.34
 
SQL > select round(bytes/1024/1024, 2) KTC_MB from v$sgastat where name = 'KTC latch subh';
    KTC_MB
    ------
      6.01
Then make 3 array_enqueue / array_dequeue calls with 10 times bigger array_size: 10000, 20000, and 30000 one after another.

The output shows that 'KTC latch subh' memory is increased progressively from 6 MB to 55 MB. Again the increase only occurs in Dequeue_Array, but not Enqueue_Array.

SQL > exec test_aq_loop('ArrayEnq',  'ArrayDeq',  3, 0, 1e4);
 
  RUN = 2 at 06:28:41
    Enqueue_Array.Size = 10000, KTC latch subh MB = 6.01
    Dequeue_Array.Size = 10000, KTC latch subh MB = 15.93
    Enqueue_Array.Size = 20000, KTC latch subh MB = 15.94
    Dequeue_Array.Size = 20000, KTC latch subh MB = 25.85
    Enqueue_Array.Size = 30000, KTC latch subh MB = 25.86
    Dequeue_Array.Size = 30000, KTC latch subh MB = 55.61
   
    Elapsed: 00:00:23.01
 
SQL > select round(bytes/1024/1024, 2) KTC_MB from v$sgastat where name = 'KTC latch subh';
 
    KTC_MB
    ------
     55.61
As a third test, make 3 array_enqueue / array_dequeue calls with 100 times bigger array_size: 100000, 200000, and 300000 one after another.

The output shows that 'KTC latch subh' memory reaches 445 MB from 55 MB and process terminated with "ORA-04031: unable to allocate 872 for KTC latch subh". Same as above two tests, the increase only occurs in Dequeue_Array, but not Enqueue_Array.

Only first two Dequeue_Array calls succeeded, the third Dequeue_Array call requires more than 445 MB "KTC latch subh" and no more free memory is available for 'KTC latch subh'.

SQL > exec test_aq_loop('ArrayEnq',  'ArrayDeq',  3, 0, 1e5);
 
  RUN = 3 at 06:30:14
  Enqueue_Array.Size = 100000, KTC latch subh MB = 55.61
  Dequeue_Array.Size = 100000, KTC latch subh MB = 154.79
  Enqueue_Array.Size = 200000, KTC latch subh MB = 154.8
  Dequeue_Array.Size = 200000, KTC latch subh MB = 350.19
  Enqueue_Array.Size = 300000, KTC latch subh MB = 350.19
  BEGIN test_aq_loop('ArrayEnq',  'ArrayDeq',  3, 0, 1e5); END;
 
  *
  ERROR at line 1:
  ORA-04031: unable to allocate 456 bytes of shared memory ("shared pool","select cols,audit$,textlengt...","SQLA^ca34c3df","opixpop:kctdef")
  ORA-06512: at "K.TEST_AQ_LOOP", line 15
  ORA-06512: at "K.TEST_AQ_LOOP", line 58
  ORA-04031: unable to allocate 872 bytes of shared memory ("shared pool","unknown object","KTC latch subh","KTCCC OBJECT")
  ORA-06512: at "SYS.DBMS_AQ", line 1107
  ORA-06512: at "K.TEST_DEQ_ARRAY", line 16
  ORA-06512: at "K.TEST_AQ_LOOP", line 49
  ORA-06512: at line 1
 
  Elapsed: 00:08:11.47
 
SQL > select round(bytes/1024/1024, 2) KTC_MB from v$sgastat where name = 'KTC latch subh';
    KTC_MB
    ------
     445.9
We can also list 'KTC latch subh' KSMCHPAR areas ("DS" or "desc"), each of which contains a set of Chunks. There are 11 such areas (more areas in bigger shared_pool). Average Chunk size is about 4 KB. (above ORA-04031 occurs when unable to allocate 872 bytes).

select ksmchpar, round(sum(ksmchsiz)/1024/1024, 2) mb, count(*), round(avg(ksmchsiz)) avg, min(ksmchsiz) min, max(ksmchsiz) max
  from  x$ksmsp
 where ksmchcom = 'KTC latch subh'
 group by ksmchpar order by mb desc;
 
  KSMCHPAR                 MB   COUNT(*)        AVG        MIN        MAX
  ---------------- ---------- ---------- ---------- ---------- ----------
  00000000934D5B50     199.52      50031       4182        920       4320
  00000000934D58A8     126.21      32822       4032        920       4320
  00000000934D5DF8      99.76      25016       4182       1000       4320
  00000000934D5BD8      19.96       5002       4184       2616       4320
  00000000934D5C60          2        502       4184       4184       4320
  00000000934D5A40          1        254       4148        920       4320
  00000000934D59B8        .01          2       4252       4184       4320
  00000000934D5930        .01          2       4252       4184       4320
  00000000934D5CE8        .01          2       4252       4184       4320
  00000000934D5D70        .01          3       1899        504       4320
  00000000934D5AC8        .01          2       4252       4184       4320
 
  11 rows selected.
To further identify memory usage, we can pick one KSMCHPAR and make a heapdump, for example:

  oradebug dump heapdump_addr 2 0x00000000934D5B50        
The dump file list all "KTC latch subh" Chunks, each of which is noted with "sz= 4144". The most noticeable text is "TEST_Q", which is exactly the queue_name used in our Test Code.

  ******************************************************
  HEAP DUMP heap name="KTC latch subh"  desc=0x934d5b50
   extent sz=0x1040 alt=32767 het=32767 rec=9 flg=0x3 opc=0
   parent=0x601476e0 owner=(nil) nex=(nil) xsz=0x1040 heap=(nil)
   fl2=0x24, fl3=0x0, nex=(nil), idx=1, dur=1, dsxvers=1, dsxflg=0x0
  dsx first ext=0x849428e0
   dsx empty ext bytes=207207648  subheap rc link=0x84942960,0x84942960
   dsx heap size=207211904, dsx heap depth=1
   pdb id=0, src pdb id=0
  EXTENT 0 addr=0x667a9180
    Chunk        0667a9190 sz=     4144    free      "               "
  Dump of memory from 0x00000000667A9190 to 0x00000000667AA1C0
  0667A9190 00001031 D0B38F00 00000000 00000000  [1...............]
  0667A91A0 667AA1F8 00000000 84942930 00000000  [..zf....0)......]
  0667A91B0 01010101 00000000 00000000 00000000  [................]
  0667A91C0 00000000 00000000 00000000 00000000  [................]
  ...
  0667A9EB0 00000000 00000000 00000000 00000000  [................]
          Repeat 6 times
  0667A9F20 00000001 54534554 0000515F 00000000  [....TEST_Q......]
  0667A9F30 00000000 00000000 00000000 00000000  [................]
By the way, above "SQLA^ca34c3df" can be found by:

select hash_value, sql_id, last_active_time, executions, rows_processed, sql_text  --, v.*
  from v$sqlarea v where hash_value = to_number('ca34c3df', 'xxxxxxxxx');
 
  HASH_VALUE SQL_ID        LAST_ACTIVE_TIME     EXECUTIONS ROWS_PROCESSED SQL_TEXT
  ---------- ------------- ---------------- ---------- -------------- ------------------------------------------------------------------------------------------
  3392455647 c1aqra3539hyz         07:25:03          2              2 select cols,audit$,textlength,intcols,nvl(property,0),flags,rowid from view$ where obj#=:1
During the test, we also record the "KTC latch subh" and PGA allocation in table test_aq_array_runs. Here the details about each STEP of every RUN.

We can see KTC_SUBH_MB increase in each ArrayDeq (KTC_DELTA > 0), but not ArrayEnq (KTC_DELTA = 0). Therefore it seems memory leak in dbms_aq.dequeue_array. The memory increased size is almost linear to dequeue array_size.

select d.*, ktc_subh_mb - lag(ktc_subh_mb) over (partition by run order by step) ktc_delta
  from test_aq_array_runs d order by 1, 2, 3;
 
  RUN STEP TIME      ACTION                    ARRAY_SIZE KTC_SUBH_MB PGA_ALLOC_MB SUBH_DETAILS                                              KTC_DELTA
  --- ---- --------  ------------------------- ---------- ----------- ------------ -------------------------------------------------------- ----------
    1    0 06:27:47  ArrayEnq-ArrayDeq Start                        0           11 ksmchcls: (freeabl=0MB, CNT=3); (recr=0MB, CNT=9)
    1    1 06:27:48  ArrayEnq                        1000           0           23 ksmchcls: (freeabl=0MB, CNT=4); (recr=0MB, CNT=9)                 0
    1    1 06:27:48  ArrayDeq                        1000           1           40 ksmchcls: (freeabl=1MB, CNT=256); (recr=0MB, CNT=9)               1
    1    2 06:27:48  ArrayEnq                        2000           1           44 ksmchcls: (freeabl=1MB, CNT=257); (recr=0MB, CNT=9)               0
    1    2 06:27:49  ArrayDeq                        2000           3           58 ksmchcls: (freeabl=3MB, CNT=757); (recr=0MB, CNT=9)               2
    1    3 06:27:49  ArrayEnq                        3000           3           58 ksmchcls: (freeabl=3MB, CNT=757); (recr=0MB, CNT=10)              0
    1    3 06:27:50  ArrayDeq                        3000           6           76 ksmchcls: (freeabl=6MB, CNT=1510); (recr=0MB, CNT=10)             3
 
    2    0 06:28:42  ArrayEnq-ArrayDeq Start                        6           10 ksmchcls: (freeabl=6MB, CNT=1510); (recr=0MB, CNT=10)
    2    1 06:28:44  ArrayEnq                       10000           6          120 ksmchcls: (freeabl=6MB, CNT=1511); (recr=0MB, CNT=10)             0
    2    1 06:28:45  ArrayDeq                       10000          16          202 ksmchcls: (freeabl=16MB, CNT=4011); (recr=0MB, CNT=10)           10
    2    2 06:28:50  ArrayEnq                       20000          16          218 ksmchcls: (freeabl=16MB, CNT=4011); (recr=0MB, CNT=11)            0
    2    2 06:28:53  ArrayDeq                       20000          26          380 ksmchcls: (freeabl=26MB, CNT=6511); (recr=0MB, CNT=11)           10
    2    3 06:29:00  ArrayEnq                       30000          26          380 ksmchcls: (freeabl=26MB, CNT=6512); (recr=0MB, CNT=11)            0
    2    3 06:29:04  ArrayDeq                       30000          56          562 ksmchcls: (freeabl=56MB, CNT=14013); (recr=0MB, CNT=11)          30
 
    3    0 06:30:14  ArrayEnq-ArrayDeq Start                       56           10 ksmchcls: (freeabl=56MB, CNT=14013); (recr=0MB, CNT=11)
    3    1 06:30:43  ArrayEnq                      100000          56          981 ksmchcls: (freeabl=56MB, CNT=14013); (recr=0MB, CNT=11)           0
    3    1 06:31:09  ArrayDeq                      100000         155         1817 ksmchcls: (freeabl=156MB, CNT=39027); (recr=0MB, CNT=11)         99
    3    2 06:32:19  ArrayEnq                      200000         155         1961 ksmchcls: (freeabl=156MB, CNT=39028); (recr=0MB, CNT=11)          0
    3    2 06:33:52  ArrayDeq                      200000         350         3595 ksmchcls: (freeabl=352MB, CNT=88304); (recr=0MB, CNT=11)        195
    3    3 06:36:00  ArrayEnq                      300000         350         2893 ksmchcls: (freeabl=352MB, CNT=88305); (recr=0MB, CNT=11)          0
Above output shows that all allocated memory are with Chunk type: "freeabl" (Freeable), no Chunk type "recr" (Recreatable). But they cannot be evicted even with flushing shared_pool ("alter system flush shared_pool"). The last resort is DB restart.

Chunk types are documented in Oracle MOS: "Troubleshooting and Diagnosing ORA-4031 Error [Video] (Doc ID 396940.1)"

  Chunk types:
 
    Normal (freeable) chunks - These chunks are allocated in such a way that the user can explicitly free
    the chunk once they have finished with the memory.
   
    Free chunks - These chunks are free and available for reuse should a request come into the pool for
    this chunk size or smaller.
   
    Recreatable chunks - This is a special form of "freeable" memory.  These chunks are placed on an
    LRU list when they are unpinned.   If memory is needed, we go to the LRU list and free "recreatable"
    memory that hasn't been used for a while.
    
    Permanent chunks - These chunks can be allocated in different ways.   Some chunks are allocated
    and will remain in use for the "life" of the instance.   Some "permanent" chunks are allocated but can
    be used over and over again internally as they are available.


2. ORA-04031 incident file


In ORA-04031 incident file, Call Stack shows that ORA-04031 is raised in "ktccAddCbkObj" calling to "kghalo", which hit no space "kghnospc".

----- Call Stack -----                                                     
 FRAME [1]  (ksedst1()+95 -> kgdsdst())                                                 
 FRAME [2]  (ksedst()+58 -> ksedst1())                                                  
 FRAME [3]  (dbkedDefDump()+23448 -> ksedst())                                          
 FRAME [4]  (ksedmp()+577 -> dbkedDefDump())                                            
 FRAME [5]  (dbgexPhaseII()+2092 -> ksedmp())                                           
 FRAME [6]  (dbgexExplicitEndInc()+285 -> dbgexPhaseII())                               
 FRAME [7]  (dbgeEndDDEInvocationImpl()+314 -> dbgexExplicitEndInc())  
                  
 FRAME [8]  (kghnospc()+4787 -> dbgeEndDDEInvocationImpl())                              
 FRAME [9]  (kghalo()+4255 -> kghnospc())                                               
 FRAME [10] (ktccAddCbkObj()+487 -> kghalo())                                           
 FRAME [11] (kwqidracbk()+255 -> ktccAddCbkObj())                                      
 FRAME [12] (kwqidcpmc()+2716 -> kwqidracbk())                                         
 FRAME [13] (kwqidafm0()+6345 -> kwqidcpmc())                                           
 FRAME [14] (kwqididqx()+2368 -> kwqidafm0())                                          
 FRAME [15] (kwqideqarr()+2295 -> kwqididqx())                                         
 FRAME [16] (kwqideqarr0()+49 -> kwqideqarr())                                         
 FRAME [17] (spefcifa()+1286 -> kwqideqarr0())                                         
 FRAME [18] (spefmccallstd()+436 -> spefcifa())                                        
 FRAME [19] (peftrusted()+139 -> spefmccallstd())                                      
 FRAME [20] (psdexsp()+280 -> peftrusted())                                            
 FRAME [21] (rpiswu2()+2077 -> psdexsp())                                              
 FRAME [22] (kxe_push_env_internal_pp_()+362 -> rpiswu2())                             
 FRAME [23] (kkx_push_env_for_ICD_for_new_session()+149 -> kxe_push_env_internal_pp_())
 FRAME [24] (psdextp()+387 -> kkx_push_env_for_ICD_for_new_session())                  
 FRAME [25] (pefccal()+663 -> psdextp())                                               
 FRAME [26] (pefcal()+223 -> pefccal())                                                
 FRAME [27] (pevm_FCAL()+90 -> pefcal())                                               
 FRAME [28] (pfrinstr_FCAL()+62 -> pevm_FCAL())                                        
 FRAME [29] (pfrrun_no_tool()+52 -> pfrinstr_FCAL())                                    
 FRAME [30] (pfrrun()+902 -> pfrrun_no_tool())                                         
 FRAME [31] (plsql_run()+1498 -> pfrrun())                                             
In fact, kghalo first calls kghfnd_in_free_lists, but no space found, thus jump to kghnospc (no space found).

  #0  0x0000000004e19e5d in kghfnd_in_free_lists ()
  #1  0x0000000004e177ad in kghalo ()
  #2  0x0000000001d1ccf7 in ktccAddCbkObj ()
ORA-04031 incident file also shows the detail memory allocation in each subpools (we set "_kghdsidx_count"=3 in our test), where "KTC latch subh" are ranked as the highest.

By the way, three occurrences of "SO private sga" in each of three subpools showed that the bug revealed in Blog: Oracle 19c new shared pool "SO private sga" and "SO private so latch" Performance Impacts is fixed in this test DB.

  ==============================================
  TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 1
  ----------------------------------------------
  "KTC latch subh                 "   198 MB 48%
  "free memory                    "    69 MB 17%
  "FileIdentificatonBlock         "    13 MB  3%
  "ksunfy_meta 1                  "  9324 KB  2%
  "SQLA                           "  8702 KB  2%
  "SO private sga                 "  8223 KB  2%
  "KGLH0                          "  6878 KB  2%
  "db_block_hash_buckets          "  5440 KB  1%
  "private strands                "  5187 KB  1%
  "KGLS                           "  4221 KB  1%
      
  ==============================================
  TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 2
  ----------------------------------------------
  "KTC latch subh                 "   146 MB 46%
  "free memory                    "    51 MB 16%
  "ksunfy_meta 1                  "  9324 KB  3%
  "SO private sga                 "  8777 KB  3%
  "ksipc state object             "  7602 KB  2%
  "KSRMA State Object             "  5705 KB  2%
  "db_block_hash_buckets          "  5504 KB  2%
  "private strands                "  5320 KB  2%
  "ASH buffers                    "  4096 KB  1%
  "KSFD SGA I/O b                 "  4092 KB  1%
      
  ==============================================
  TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 3
  ----------------------------------------------
  "KTC latch subh                 "   101 MB 37%
  "free memory                    "    45 MB 17%
  "ksunfy_meta 1                  "  9324 KB  3%
  "SO private sga                 "  8925 KB  3%
  "SQLA                           "  7807 KB  3%
  "KGLH0                          "  6938 KB  2%
  "db_block_hash_buckets          "  5444 KB  2%
  "private strands                "  5187 KB  2%
  "PLMCD                          "  4619 KB  2%
  "ASH buffers                    "  4096 KB  1%
In incident file, we can also find above heapdump KSMCHPAR: desc=0x934d5b50, which is marked as "ds=0x934d5b50".

For "KSMCHPAR = 00000000934D5B50" in above x$ksmsp query, size (199.52 MB) and chunk count (50031) is noted as "sz=209213168 ct= 50031" in incident file.

  Chunk        0667a9168 sz=     4184    freeable  "KTC latch subh "  ds=0x934d5b50
  Chunk        0667aa1c0 sz=     4184    freeable  "KTC latch subh "  ds=0x934d5b50
  ...
  Chunk        084677d58 sz=     4184    freeable  "KTC latch subh "  ds=0x934d5b50
       ds        0934d5b50 sz=209213168 ct=    50031
Pick one Chunk addr, for example, 0667a9168, with following query, it can be found by "lower(ksmchptr) like '%0667a9168'" in x$ksmsp.

select * from x$ksmsp
 where ksmchcom = 'KTC latch subh' and ksmchpar = '00000000934D5B50' and lower(ksmchptr) like '%0667a9168';   
 
  ADDR               INDX INST_ID CON_ID   KSMCHIDX KSMCHDUR KSMCHCOM       KSMCHPTR         KSMCHSIZ KSMCHCLS KSMCHTYP KSMCHPAR
  ---------------- ------ ------- ------ ---------- -------- -------------- ---------------- -------- -------- -------- ----------------
  00007F05EBE80DF8 235272       1      0          1        1 KTC latch subh 00000000667A9168     4184 freeabl         0 00000000934D5B50


3. Other enqueue/dequeue Cases


If we make message "single enqueue/single dequeue" or "array enqueue/ single dequeue", there are no 'KTC latch subh' memory leak observed.

  exec test_aq_loop('SingleEnq', 'SingleDeq', 3, 0, 1e3);
  exec test_aq_loop('ArrayEnq',  'SingleDeq', 3, 0, 1e3);
But with "single enqueue/array dequeue", we can also observe the similar 'KTC latch subh' memory leak as "array enqueue/array dequeue".

  exec test_aq_loop('SingleEnq', 'ArrayDeq',  3, 0, 1e3);
In conclusion, this Blog demonstrated "KTC Latch Subh" memory leak in dbms_aq.dequeue_array.


4. Test Code


----==================== 1. AQ Setup====================----
 
begin sys.dbms_aqadm.drop_queue_table(queue_table => 'TEST_QTAB', force=> TRUE); end;
/
 
drop type payload_rec force;
 
create or replace noneditionable type payload_rec force as object(r1 number, r2 varchar2(10))
/
 
drop type test_payload force;
 
create or replace noneditionable type test_payload is object (id number, txt varchar2(1000), lob clob, rec payload_rec)
/                                                                             
 
drop type test_payload_array force;
 
create  or replace noneditionable type test_payload_array as table of test_payload;
/
 
begin
  sys.dbms_aqadm.create_queue_table
   (queue_table            => 'TEST_QTAB'
    ,queue_payload_type    => 'TEST_PAYLOAD'
    ,compatible            => '10.0.0'  --'8.1'
    ,sort_list             => 'PRIORITY,ENQ_TIME'
    ,multiple_consumers    =>  false
    ,message_grouping      =>  0
    ,comment               =>  'KS Test Queue Table'
    ,secure                =>  false);
end;
/
 
begin
  sys.dbms_aqadm.stop_queue ( queue_name => 'TEST_Q');
  sys.dbms_aqadm.drop_queue ( queue_name => 'TEST_Q');
end;
/
 
begin
  sys.dbms_aqadm.create_queue
   (queue_name          => 'TEST_Q'
   ,queue_table         => 'TEST_QTAB'
   ,queue_type          =>  sys.dbms_aqadm.normal_queue
   ,max_retries         =>  100
   ,retry_delay         =>  2
   ,retention_time      =>  604800
   ,comment             => 'KS Test Queue');
end;
/
 
begin sys.dbms_aqadm.start_queue(queue_name => 'TEST_Q', enqueue => true, dequeue => true); end;
/

----==================== 2. Single Enqueue / Dequeue ====================----
 
create or replace procedure test_enq_single(p_id number := 1) as
  l_enqueue_options          dbms_aq.enqueue_options_t;
  l_message_properties       dbms_aq.message_properties_t;
  l_payload_rec              payload_rec;
  l_payload                  test_payload;
  l_message_handle           raw(16);
 
  l_array_msg_properties     dbms_aq.message_properties_array_t := dbms_aq.message_properties_array_t();
  l_array_payloads           test_payload_array := new test_payload_array();
  l_array_msg_ids            dbms_aq.msgid_array_t;
  l_array_errors             dbms_aq.error_array_t;
  l_enq_cnt                  number;
begin
  l_payload_rec := payload_rec(p_id, 'XXXYYY');
  l_payload     := test_payload(p_id, rpad('ABC', 1000, 'X'), rpad('CBA', 7000, 'X'), l_payload_rec);
 
  dbms_aq.enqueue(queue_name         => 'TEST_Q',
                  enqueue_options    => l_enqueue_options,
                  message_properties => l_message_properties,
                  payload            => l_payload,
                  msgid              => l_message_handle);
  commit;
end;
/             
 
-- exec test_enq_single(1);
 
create or replace procedure test_deq_single(p_dur number := 1) as
  l_dequeue_options       dbms_aq.dequeue_options_t;
  l_message_properties    dbms_aq.message_properties_t;
  l_payload               test_payload;
                l_message_handle        raw(16);
begin
                l_dequeue_options.wait := p_dur;
               
  dbms_aq.dequeue(queue_name         => 'TEST_Q',
                  dequeue_options    => l_dequeue_options,
                  message_properties => l_message_properties,
                  payload            => l_payload,
                  msgid              => l_message_handle);
       
  --dbms_output.put_line ('MSG id : ' || l_payload.id);
  commit;
 
  exception when others then dbms_output.put_line ('Error: ' || SQLERRM);
end;
/
                                                                                                               
-- exec test_deq_single();
 
create or replace procedure test_enq_single_loop (p_cnt number, p_id number := 1) as
  l_subh_mb                  number;
begin
  for i in 1..p_cnt loop
    test_enq_single(i);
  end loop;
  select round(sum(bytes)/1024/1024, 2) into l_subh_mb from v$sgastat s where name = 'KTC latch subh';
  dbms_output.put_line ('SingleEnqueCNT = '||p_cnt||', KTC latch subh MB = '||l_subh_mb);
end;
/
 
-- exec test_enq_single_loop(10);
 
create or replace procedure test_deq_single_loop (p_cnt number, p_dur number := 1) as
  l_subh_mb                  number;
begin
  for i in 1..p_cnt loop
    test_deq_single(p_dur);
  end loop;
  select round(sum(bytes)/1024/1024, 2) into l_subh_mb from v$sgastat s where name = 'KTC latch subh';
  dbms_output.put_line ('SingleDequeCNT = '||p_cnt||', KTC latch subh MB = '||l_subh_mb);
end;
/
 
-- exec test_deq_single_loop(10);

----==================== 3. Array Enqueue / Dequeue ====================----
 
create or replace procedure test_enq_array(p_array_size number := 1) as
  l_enqueue_options          dbms_aq.enqueue_options_t;
  l_message_properties       dbms_aq.message_properties_t;
  l_array_msg_properties     dbms_aq.message_properties_array_t := dbms_aq.message_properties_array_t();
  l_payload_rec              payload_rec;
  l_array_payloads           test_payload_array := new test_payload_array();
  l_array_msg_ids            dbms_aq.msgid_array_t;
  l_array_errors             dbms_aq.error_array_t;
  l_enq_cnt                  number;
  l_subh_mb                  number;
begin
  for i in 1..p_array_size loop
    l_payload_rec :=  payload_rec(i, 'XXXYYY');
    l_array_payloads.extend;
    l_array_payloads(l_array_payloads.last) := test_payload(i, rpad('ABC', 1000, 'X'), rpad('CBA', 7000, 'X'), l_payload_rec);
    l_array_msg_properties.extend;
    l_array_msg_properties(l_array_msg_properties.last) := l_message_properties;
  end loop;
 
  l_enq_cnt := dbms_aq.enqueue_array(
                       queue_name               => 'TEST_Q',
                       enqueue_options          => l_enqueue_options,
                       array_size               => l_array_payloads.count,
                       message_properties_array => l_array_msg_properties,
                      payload_array            => l_array_payloads,
                       msgid_array              => l_array_msg_ids,
                       error_array              => l_array_errors);
  commit;
 
  select round(sum(bytes)/1024/1024, 2) into l_subh_mb from v$sgastat s where name = 'KTC latch subh';
  dbms_output.put_line('Enqueue_Array.Size = '||l_enq_cnt||', KTC latch subh MB = '||l_subh_mb);
end;
/                             
 
-- exec test_enq_array(3);
 
create or replace procedure test_deq_array(p_array_size number := 1, p_sleep number := 5) as
   l_dequeue_options       dbms_aq.dequeue_options_t;
   l_array_msg_properties  dbms_aq.message_properties_array_t;
   l_array_payloads        test_payload_array;
   l_array_msg_ids         dbms_aq.msgid_array_t;
   l_deq_cnt               number;
   l_subh_mb                  number;
begin
   l_array_payloads := test_payload_array();
   l_array_payloads.extend(p_array_size);
   l_array_msg_properties := dbms_aq.message_properties_array_t();
   l_array_msg_properties.extend(p_array_size);
   l_array_msg_ids := dbms_aq.msgid_array_t();
   l_dequeue_options.wait := p_sleep;
 
   l_deq_cnt := dbms_aq.dequeue_array(
                        queue_name               => 'TEST_Q',
                        dequeue_options          => l_dequeue_options,
                        array_size               => p_array_size,
                        message_properties_array => l_array_msg_properties,
                        payload_array            => l_array_payloads,
                        msgid_array              => l_array_msg_ids);
  
   select round(sum(bytes)/1024/1024, 2) into l_subh_mb from v$sgastat s where name = 'KTC latch subh';
   dbms_output.put_line('Dequeue_Array.Size = '||l_deq_cnt||', KTC latch subh MB = '||l_subh_mb);
   commit;
  
    --for i in 1..l_deq_cnt loop
    --  dbms_output.put_line ('Payload id: ' || l_array_payloads(i).id);
    --end loop;
end;
/
 
-- exec test_deq_array(3);

----==================== 4. Loop Test Setup and Test Outcome Recording ====================----
drop table test_aq_array_runs;
 
create table test_aq_array_runs(run number, step number, time date, action varchar2(30), array_size number,
                                ktc_subh_mb number, pga_alloc_mb number, subh_details varchar2(300));
 
 

----==================== 5. Array-Single Enqueue / Dequeue Loop ====================----
 
create or replace procedure test_aq_loop(p_enq_mode varchar2, p_deq_mode varchar2, p_steps number := 1, p_base_size number := 0, p_delta_size number := 100000) as
  l_array_size  number;
  l_run         number := 0;
  l_step        number := 0;
  l_purge_opt   dbms_aqadm.aq$_purge_options_t;
 
  procedure save_stats (p_name varchar2) as
    l_subh_mb         number;
    l_pga_mb          number;
    l_subh_details    varchar2(300);
  begin
    select round(sum(bytes)/1024/1024) into l_subh_mb
      from v$sgastat s where name = 'KTC latch subh';
     
    select round(pga_alloc_mem/1024/1024) into l_pga_mb
      from v$session s, v$process p where s.paddr=p.addr and s.sid in (select sid from v$mystat where rownum=1);
     
   select 'ksmchcls: '|| listagg(subh_areas, '; ') within group (order by subh_areas) into l_subh_details
     from (select '('||ksmchcls||'='||round(sum(ksmchsiz)/1024/1024)||'MB, CNT='||count(*)||')' subh_areas
             from sys.x_ksmsp where lower(ksmchcom) like 'ktc%subh' group by ksmchcls);
  
   insert into test_aq_array_runs values (l_run, l_step, sysdate, p_name, l_array_size, l_subh_mb, l_pga_mb, l_subh_details);
   commit;
  end;
begin
  --l_purge_opt.block := true;
  --dbms_aqadm.purge_queue_table(queue_table => 'TEST_QTAB', purge_condition => null, purge_options => l_purge_opt);
  --select state, count(*) from test_qtab where q_name = 'TEST_Q' group by state;  -- 0 READY, 1 WAITING, 2 RETAINED or PROCESSED, 3 EXPIRED
 
  select nvl(max(run), 0) + 1 into l_run from test_aq_array_runs;
  dbms_output.put_line ('RUN = '||l_run||' at '||sysdate);
  save_stats(p_enq_mode||'-'||p_deq_mode||' Start');
 
  for i in 1..p_steps loop
    l_step       := i;
    l_array_size := p_base_size + (l_step*p_delta_size);
   
    -- Enqueue Array or Single
    if p_enq_mode = 'ArrayEnq' then
      test_enq_array(l_array_size);
      save_stats('ArrayEnq');
    else
      test_enq_single_loop(l_array_size);
      save_stats('SingleEnq');
    end if;
   
    -- Dequeue Array or Single
    if p_deq_mode =  'ArrayDeq' then
      test_deq_array(l_array_size);
      save_stats('ArrayDeq');
    else
      test_deq_single_loop(l_array_size);
      save_stats('SingleDeq');     
    end if;
  end loop;
 
  exception when others then
    save_stats('Error');
    raise;
end;
/
 
 
-- exec test_aq_loop('SingleEnq', 'SingleDeq', 3, 0, 1e1);
-- exec test_aq_loop('ArrayEnq',  'SingleDeq', 3, 0, 1e1);
-- exec test_aq_loop('SingleEnq', 'ArrayDeq',  3, 0, 1e1);
-- exec test_aq_loop('ArrayEnq',  'ArrayDeq',  3, 0, 1e1);

Friday, October 22, 2021

Oracle Temp Space Usage Monitoring and View v$tempseg_usage

v$tempseg_usage reports temp space usage per session. This Blog will first examine its coverage in two aspects: temp object creations and sql executions, then discuss Temp Space Usage Monitoring in queries, trace, and dump.

Note: Tested in Oracle 19c with following settings:

  locally managed uniform temporary tablespace (extent=256 8K blocks)
  db_block_size        = 8192
  pga_aggregate_target = 400M
  db_cache_size        = 2000M
  temp_undo_enabled    = TRUE
  nls_length_semantics = CHAR
  db_files             = 3072
  temp file maxsize    = 8000M

1. v$tempseg_usage and v$sort_usage


v$tempseg_usage (9i onwards) is v$sort_usage, which is retained for backward compatibility and removed from Oracle Docu.

gv$tempseg_usage is a synonym for sys.gv_$sort_usage and defined as follows:

CREATE OR REPLACE FORCE VIEW SYS.GV_$SORT_USAGE as
SELECT so.inst_id                                       INST_ID,       
       username                                         USERNAME,      
       username                                         "USER",        
       ktssoses                                         SESSION_ADDR,  
       ktssosno                                         SESSION_NUM,   
       prev_sql_addr                                    SQLADDR,       
       prev_hash_value                                  SQLHASH,       
       prev_sql_id                                      SQL_ID,        
       ts.name                                          TABLESPACE,    
       DECODE (ktssotbst, 1, 'TEMPORARY', 'PERMANENT')  CONTENTS,      
       DECODE (ktssosegt, 1, 'SORT',
                          2, 'HASH',
                          3, 'DATA',
                          4, 'INDEX',
                          5, 'LOB_DATA',
                          6, 'LOB_INDEX',
                          7, 'TEMP_UNDO',
                             'UNDEFINED')               SEGTYPE,        
       ktssofno                                         SEGFILE#,       
       ktssobno                                         SEGBLK#,        
       ktssoexts                                        EXTENTS,        
       ktssoblks                                        BLOCKS,         
       ktssorfno                                        SEGRFNO#,       
       ktssotsnum                                       TS#,            
       so.con_id                                        CON_ID,         
       ktssosqlid                                       SQL_ID_TEMPSEG  
  FROM x$ktsso so, v$session, v$tablespace ts
WHERE     so.ktssotsnum = ts.ts#
       AND so.con_id = ts.con_id
       AND ktssoses = v$session.saddr
       AND ktssosno = v$session.serial#
SEGTYPE indicates 7 different types of temp segments. In Oracle 19c Docu, it is noted as "Type of sort segment", and only first 6 types are listed. 'TEMP_UNDO' is not included, due to 12c new introduced "temp_undo_enabled"
(Note: Temp LOB_INDEX seems special, see later discussion).


2. Temp Object Creations


First we create a temp table including 5 different temp segment types: DATA, INDEX, LOB_INDEX, LOB_DATA, TEMP_UNDO. And fill it with 1000 rows.

truncate table temp_tab;
drop table temp_tab;
create global temporary table temp_tab(a number, b clob) on commit preserve rows;
create index temp_tab#i1 on temp_tab(a);
 
insert /*+ append */ into temp_tab select level, rpad('ABC', 7000, 'X') from dual connect by level <= 1e3;
commit;
Then we list all created temp segments (test session id = 123):

select s.sid, t.tablespace, t.segtype, t.blocks, t.sql_id_tempseg, substr(a.sql_text, 1, 110) sql_text --, t.*
  from v$tempseg_usage t, v$session s, v$sqlarea a
 where t.session_addr=s.saddr and t.sql_id_tempseg = a.sql_id and sid = 123
 order by t.segtype, t.blocks;
 
  SID TABL SEGTYPE    BLOCKS SQL_ID_TEMPSEG SQL_TEXT
  --- ---- ---------- ------ -------------- ---------------------------------------------------------------------------------------------------------
  123 TEMP DATA          256 d61k5majnv4mb  insert /*+ append */ into temp_tab select level, rpad('ABC', 7000, 'X') from dual connect by level <= 1e3
  123 TEMP INDEX         256 d61k5majnv4mb  insert /*+ append */ into temp_tab select level, rpad('ABC', 7000, 'X') from dual connect by level <= 1e3
  123 TEMP INDEX         256 d61k5majnv4mb  insert /*+ append */ into temp_tab select level, rpad('ABC', 7000, 'X') from dual connect by level <= 1e3
  123 TEMP LOB_DATA     1024 d61k5majnv4mb  insert /*+ append */ into temp_tab select level, rpad('ABC', 7000, 'X') from dual connect by level <= 1e3
  123 TEMP TEMP_UNDO     256 d61k5majnv4mb  insert /*+ append */ into temp_tab select level, rpad('ABC', 7000, 'X') from dual connect by level <= 1e3
We can aslo query underlined x$ktsso to reveal TEMP_OBJECT_ID and TEMP_DATA_OBJECT_ID (not exposed in v$tempseg_usage).
We can also see that names of LOB object and index are composed by table OBJECT_ID (4080456) with prefix "SYS_LOB"/"SYS_IL" and suffix "C00002$$".

-- sys.x_ktsso is x$ktsso
select s.sid, s.serial#, g.ktssoobjn temp_object_id, g.ktssoobjd temp_data_object_id, o.object_name, g.ktssosegt segtype, g.ktssoblks blocks,
       temporary, generated, g.ktssosqlid sql_id_tempseg --, a.sql_text, g.*
  from sys.x_ktsso g, v$session s, v$sqlarea a, dba_objects o
where g.ktssoses=s.saddr and g.ktssosqlid = a.sql_id and g.ktssoobjn=o.object_id(+) and s.sid=123
 order by g.ktssoobjn, g.ktssoblks;
 
  SID SERIAL# TEMP_OBJECT_ID TEMP_DATA_OBJECT_ID OBJECT_NAME               SEGTYPE     BLOCKS TEMPORARY GENERATED SQL_ID_TEMPSEG
  --- ------- -------------- ------------------- ------------------------- ------- ---------- --------- --------- ---------------
  123   10733              0             4311680                                 7        256                     d61k5majnv4mb
  123   10733        4080456             4349312 TEMP_TAB                        3        256 Y         N         d61k5majnv4mb
  123   10733        4080457             4376960 SYS_LOB0004080456C00002$$       5       1024 Y         Y         d61k5majnv4mb
  123   10733        4080458             4333440 SYS_IL0004080456C00002$$        4        256 Y         Y         d61k5majnv4mb
  123   10733        4080459             4327040 TEMP_TAB#I1                     4        256 Y         N         d61k5majnv4mb
In the above output, TEMP_DATA_OBJECT_ID=4311680 is with SEGTYPE=7, that is 'TEMP_UNDO' according to GV_$SORT_USAGE (there will be no 'TEMP_UNDO' if set "temp_undo_enabled = false").

Each temp segment is session specific, and is created as a distinct object in each session.
It has a unique DATA_OBJECT_ID, that means one different DATA_OBJECT_ID per session
(non-temp segment has the same DATA_OBJECT_ID in DB-wide for all the session). It is calculated as follows:

-- traditional (Smallfile) temp tablesapce
  DATA_OBJECT_ID = "relative file number" * power(2, 22) + "seg_header_block_number"

-- Bigfile temp tablesapce
  DATA_OBJECT_ID = "seg_header_block_number"
We can verify them as follows:

-- sys.x_ktsso is x$ktsso
select s.sid, s.serial#, g.ktssoobjn temp_object_id, 
       g.ktssoobjd temp_data_object_id, 
      (ktssorfno*power(2, 22) + ktssobno) temp_data_object_id_calc,
       ktssorfno rfile#, ktssobno seg_header_block,
       o.object_name, g.ktssosegt segtype, g.ktssoblks blocks, 
       temporary, generated
       --g.ktssosqlid sql_id_tempseg, a.sql_text, g.*
  from sys.x_ktsso g, v$session s, v$sqlarea a, dba_objects o
where g.ktssoses=s.saddr and g.ktssosqlid = a.sql_id and g.ktssoobjn=o.object_id(+) and s.sid=913 
 order by g.ktssoobjn, g.ktssoblks;

  SID SERIAL# TEMP_OBJECT_ID TEMP_DATA_OBJECT_ID  TEMP_DATA_OBJECT_ID_CALC     RFILE# SEG_HEADER_BLOCK  OBJECT_NAME               SEGTYPE     BLOCKS TEMPORARY GENERATED
  --- ------- -------------- -------------------  ------------------------ ---------- ----------------  ------------------------- ------- ---------- --------- ---------
  123   10733              0             4311680                   4311680          1           117376                                  7        256                    
  123   10733        4080456             4349312                   4349312          1           155008  TEMP_TAB                        3        256 Y         N        
  123   10733        4080457             4376960                   4376960          1           182656  SYS_LOB0004080456C00002$$       5       1024 Y         Y        
  123   10733        4080458             4333440                   4333440          1           139136  SYS_IL0004080456C00002$$        4        256 Y         Y        
  123   10733        4080459             4327040                   4327040          1           132736  TEMP_TAB#I1                     4        256 Y         N        

-- Note: v$bh.FILE# = x$bh.FILE# is absolute file number.
--       TEMP_DATA_OBJECT_ID_CALC is computed with relative file number, 
--       which can be caculated by v$bh.OBJD and v$bh.block# as follows.
--       relative file number is exposed in x$bh.DBARFIL, see next query

select file#, block#, class#, temp, objd 
      ,((OBJD - block#)/power(2, 22)) relative_file_number  --, b.*
  from v$bh b
 where (   OBJD = block#                             -- Bigfile   Tablespace
        or mod((OBJD - block#), power(2, 22)) = 0    -- Smallfile Tablespace
       )
 --and block# in (117376, 155008, 182656, 139136, 132736)
 --and OBJD in (4311680, 4349312, 4376960, 4333440, 4327040)
 and (class# = 4 or class# >= 17 and mod(class#,2)=1)   -- class# = 4: Segment Header, class# >= 17 odd number: Temp Undo Segment Header
 order by b.OBJD;
 
  FILE#  BLOCK#  class#  TEMP  OBJD     RELATIVE_FILE_NUMBER
  -----  ------  ------  ----  -------  --------------------
      1  117376      33     Y  4311680   1                     -- class# >= 17 odd number is Temp Undo Segment Header
      1  155008       4     Y  4349312   1                     -- class#  = 4 is normal Segment Header
      1  182656       4     Y  4376960   1                     
      1  139136       4     Y  4333440   1                     
      1  132736       4     Y  4327040   1  
  
-- The correct way is to use relative file number x$bh.DBARFIL instead of v$bh.FILE# = x$bh.FILE#, which is absolute file number

select * 
  from sys.x_bh 
 where dbarfil = 1    -- relative file number 
   --and dbablk in (117376, 155008, 182656, 139136, 132736)
   --and obj in (4311680, 4349312, 4376960, 4333440, 4327040)
   and (class = 4 or class >= 17 and mod(class,2)=1);   -- class = 4: Segment Header, class >= 17 odd number: Temp Undo Segment Header

-- Computation Verification. 
-- class is either  = 4 (Segment Header Block) 
--              or >= 17 odd number (Temp Undo Segment Header Block)
select OBJ, DBARFIL, DBABLK, class, b.* 
  from sys.x_bh b
 where OBJ = DBABLK                             -- Bigfile   Tablespace
    or OBJ = DBARFIL * power(2, 22) + dbablk    -- Smallfile Tablespace
;
TEMP_OBJECT_ID number (except 'TEMP_UNDO') also suggests the creating sequence of temp objects.

Following query displays the temp blocks kept in buffer cache.

select objd, count(*)
  from v$bh
where objd in (select ktssoobjd from sys.x_ktsso g, v$session s where g.ktssoses=s.saddr and s.sid=123)
 group by objd order by objd;
 
   OBJD COUNT(*)
------- --------
4311680       18
4327040        7
4333440        2
4349312        3
4376960        3
In the above output, "SYS_IL0004080456C00002$$" is marked as SEGTYPE=4, but according to definition of VIEW SYS.GV_$SORT_USAGE, SEGTYPE=4 is for normal 'INDEX', 'LOB_INDEX' should be ktssosegt (SEGTYPE) 6, specially for temp LOB_INDEX.

With following test code, we can see SEGTYPE: LOB_INDEX (ktssosegt: 6).

In one session (SID 123), we run following code to create a xmltype from bfile (sleeping for 60 seconds to observe):

declare
  l_xml    xmltype;
begin
  l_xml := xmltype(bfilename('TEST_DIR', 'TEST.XML'), nls_charset_id('utf8'));

  dbms_session.sleep(60);
end;
/
In another montoring session, we can see SEGTYPE: LOB_DATA (ktssosegt: 5) and LOB_INDEX (ktssosegt: 6):

select s.sid, tablespace, contents, segtype, segfile#, segblk#, blocks, segrfno#
 from v$tempseg_usage t, v$session s where t.session_addr=s.saddr;

   SID TABLESPACE CONTENTS  SEGTYPE   SEGFILE# SEGBLK# BLOCKS SEGRFNO#
  ---- ---------- --------- --------- -------- ------- ------ --------
   123 TEMP1      TEMPORARY LOB_DATA      3074  128384    768        1
   123 TEMP1      TEMPORARY LOB_INDEX     3074  126336    256        1
      
      
-- It creates 1 temp CACHE_LOBS, and 2 temp ABSTRACT_LOBS.
select * from v$temporary_lobs;

   SID CACHE_LOBS NOCACHE_LOBS ABSTRACT_LOBS
  ---- ---------- ------------ -------------
   123          1            0             2
Following queries confirm that "SYS_IL0004080456C00002$$" is a 'LOB_INDEX'.

select table_name, column_name, segment_name, tablespace_name, index_name
  from dba_lobs where table_name='TEMP_TAB';
 
  TABLE_NAME COLUMN_NAM SEGMENT_NAME              TABLESPACE INDEX_NAME
  ---------- ---------- ------------------------- ---------- ------------------------
  TEMP_TAB   B          SYS_LOB0004080456C00002$$ TEMP       SYS_IL0004080456C00002$$
 
select index_name, index_type, table_name, uniqueness, temporary, generated, duration
  from dba_indexes where table_name='TEMP_TAB';
 
  INDEX_NAME               INDEX_TYPE TABLE_NAME UNIQUENES TEMPORARY GENERATED DURATION
  ------------------------ ---------- ---------- --------- --------- --------- ---------------
  SYS_IL0004080456C00002$$ LOB        TEMP_TAB   UNIQUE    Y         Y         SYS$TRANSACTION
  TEMP_TAB#I1              NORMAL     TEMP_TAB   NONUNIQUE Y         N         SYS$SESSION
 
select object_name, object_id, data_object_id, object_type, temporary, generated
  from dba_objects where object_name in (select index_name from dba_lobs where table_name='TEMP_TAB');
 
  OBJECT_NAME              OBJECT_ID DATA_OBJECT_ID OBJECT_TYPE TEMPORARY GENERATED
  ------------------------ --------- -------------- ----------- --------- ---------
  SYS_IL0004080456C00002$$   4080458        4080458 INDEX       Y         Y

3. SQL Executions


Now we run a query and list different run-time temp segments during SQL execution.

At first create test objects:

truncate table temp_tab_3k;
drop table temp_tab_3k;
create global temporary table temp_tab_3k(a number) on commit preserve rows;
 
truncate table temp_tab_30k;
drop table temp_tab_30k;
create global temporary table temp_tab_30k(a number, b varchar2(1000)) on commit preserve rows;
 
truncate table temp_tab_4k;
drop table temp_tab_4k;
create global temporary table temp_tab_4k(a number) on commit preserve rows;
 
truncate table temp_tab_40k;
drop table temp_tab_40k;
create global temporary table temp_tab_40k(a number, b varchar2(1000)) on commit preserve rows;
Then run a Plsql code consisting of two queries, one is in main block, another is in autonomous_transaction block. Both are active during the Plsql code execution (so more than one Sql statements can be active at the same time in one single session).

(Note: we set "_optimizer_gather_stats_on_load"=true to enable Online Statistics Gathering.
See Blog: 12c Online Statistics Gathering for Direct Path Insert and 3 Side Effects)

alter session set max_dump_file_size = unlimited;
alter system set events '1652 trace name errorstack level 3';
  -- ORA-01652 trace file is appended at the end of the Blog
 
alter session set "_optimizer_gather_stats_on_load"=true;
 
declare
  l_cnt_temp_tab_4k  number;
  procedure sort is
    pragma autonomous_transaction;
    l_cnt_temp_tab_3k  number;
  begin
    begin
      select count(*) into l_cnt_temp_tab_3k from temp_tab_3k;
      dbms_output.put_line('temp_tab_3k rows='||l_cnt_temp_tab_3k);
     
      insert /*+ append */ into temp_tab_4k select level from dual connect by level <= 4e3;
      insert /*+ append */ into temp_tab_40k select level, rpad('ABC', 1000, 'X') from dual connect by level <= 4e4;
      commit;
      for c in (select /*+ GATHER_PLAN_STATISTICS MONITOR sort_2 */ t22.a from temp_tab_4k t21, temp_tab_40k t22 order by t22.a)
      loop
        if mod(c.a, 1e3)=0 then
          dbms_session.sleep(0.01);
        end if;     
      end loop;     
      commit;
    end;
  end sort;
begin
  insert /*+ append */ into temp_tab_3k select level from dual connect by level <= 3e3;
  insert /*+ append */ into temp_tab_30k select level, rpad('ABC', 1000, 'X') from dual connect by level <= 3e4;
  commit;
  for c in (select /*+ GATHER_PLAN_STATISTICS MONITOR sort_1 */ t12.a from temp_tab_3k t11, temp_tab_30k t12 order by t12.a)
  loop
    sort;
    select count(*) into l_cnt_temp_tab_4k from temp_tab_4k;
    dbms_output.put_line('temp_tab_4k rows='||l_cnt_temp_tab_4k);
    if mod(c.a, 1e3)=0 then
      dbms_session.sleep(0.01);
    end if;     
  end loop;
end;
/
Now we can list all created temp segments (test session id = 789):

select s.sid, t.tablespace, t.segtype, t.blocks, t.sql_id_tempseg, substr(a.sql_text, 1, 110) sql_text --, t.*
  from v$tempseg_usage t, v$session s, v$sqlarea a
 where t.session_addr=s.saddr and t.sql_id_tempseg = a.sql_id and sid = 789
 order by t.segtype desc, t.blocks;
 
  SID TABL SEGTYPE   BLOCKS SQL_ID_TEMPSEG SQL_TEXT
  --- ---- --------- ------ -------------- --------------------------------------------------------------------------------------------------------------
  789 TEMP TEMP_UNDO    256 c06dqvb5b20x0  declare   procedure sort is     pragma autonomous_transaction;   begin     begin       insert /*+ append */ in
  789 TEMP SORT      132608 6tuxpu523m1hf  SELECT /*+ GATHER_PLAN_STATISTICS MONITOR sort_1 */ T12.A FROM TEMP_TAB_3K T11, TEMP_TAB_30K T12 ORDER BY T12.
  789 TEMP SORT      235776 dmuyj6fvhjqp6  SELECT /*+ GATHER_PLAN_STATISTICS MONITOR sort_2 */ T22.A FROM TEMP_TAB_4K T21, TEMP_TAB_40K T22 ORDER BY T22.
  789 TEMP DATA         256 824w6gbdj9mr2  INSERT /*+ append */ INTO TEMP_TAB_3K SELECT LEVEL FROM DUAL CONNECT BY LEVEL <= 3E3
  789 TEMP DATA         256 fxqyxs94y5vk3  INSERT /*+ append */ INTO TEMP_TAB_4K SELECT LEVEL FROM DUAL CONNECT BY LEVEL <= 4E3
  789 TEMP DATA        4352 fatjqb5a1czd7  INSERT /*+ append */ INTO TEMP_TAB_30K SELECT LEVEL, RPAD('ABC', 1000, 'X') FROM DUAL CONNECT BY LEVEL <= 3E4
  789 TEMP DATA        5888 3m9nu19xxyr2d  INSERT /*+ append */ INTO TEMP_TAB_40K SELECT LEVEL, RPAD('ABC', 1000, 'X') FROM DUAL CONNECT BY LEVEL <= 4E4
We can also query underlined x$ktsso to reveal TEMP_OBJECT_ID and TEMP_DATA_OBJECT_ID.

select s.sid, s.serial#, g.ktssoobjn temp_object_id, g.ktssoobjd temp_data_object_id, o.object_name, g.ktssosegt segtype, g.ktssoblks blocks,
       temporary, generated, g.ktssosqlid sql_id_tempseg --, a.sql_text, g.*
  from sys.x_ktsso g, v$session s, v$sqlarea a, dba_objects o
where g.ktssoses=s.saddr and g.ktssosqlid = a.sql_id and g.ktssoobjn=o.object_id(+) and s.sid=789
 order by g.ktssoobjn, g.ktssoblks;
 
  SID SERIAL# TEMP_OBJECT_ID TEMP_DATA_OBJECT_ID OBJECT_NAME  SEGTYPE BLOCKS TEMPORARY GENERATED SQL_ID_TEMPSEG
  --- ------- -------------- ------------------- ------------ ------- ------ --------- --------- --------------
  789   40183              0             4287616                    7    256                     c06dqvb5b20x0
  789   40183              0             4285824                    1 132608                     6tuxpu523m1hf
  789   40183              0             4296832                    1 235776                     dmuyj6fvhjqp6
  789   40183        4080448             4218240 TEMP_TAB_3K        3    256 Y         N         824w6gbdj9mr2
  789   40183        4080449             4196992 TEMP_TAB_30K       3   4352 Y         N         fatjqb5a1czd7
  789   40183        4080450             4306304 TEMP_TAB_4K        3    256 Y         N         fxqyxs94y5vk3
  789   40183        4080451             4306048 TEMP_TAB_40K       3   5888 Y         N         3m9nu19xxyr2d
Query v$sql_workarea_active to show work areas currently allocated for the session:

select sid, sql_id, operation_type, expected_size, actual_mem_used, tempseg_size, tablespace, active_time --, v.*
  from v$sql_workarea_active v
 where sid = 789
 order by v.sql_id, v.operation_type;
 
  SID SQL_ID        OPERATION_TYPE EXPECTED_SIZE ACTUAL_MEM_USED TEMPSEG_SIZE TABL ACTIVE_TIME
  --- ------------- -------------- ------------- --------------- ------------ ---- -----------
  789 6tuxpu523m1hf BUFFER                 75776           75776                    1165320603
  789 6tuxpu523m1hf SORT (v2)            1245184          508928   1086324736 TEMP  1165320158
  789 dmuyj6fvhjqp6 BUFFER                104448          104448                    1150366036
  789 dmuyj6fvhjqp6 SORT (v2)            1245184          508928   1931476992 TEMP  1150365532
SQL_ID: 6tuxpu523m1hf is the query in main block, dmuyj6fvhjqp6 is that in autonomous_transaction block. Above output consists of two "BUFFER" areas and two "SORT (v2)" respectively for two queries. "BUFFER" is used for "BUFFER SORT" in xplan. "SORT (v2)" is used for "SORT ORDER BY" in xplan (see later "SQL Monitoring Report").

According to v$sql_workarea_active Oracle Docu, if a work area spills to disk, then this view contains information for the temporary segment created on behalf of this work area.

Due to the size of sorting, 1GB (1086324736) temp space is allocated for main block query; 2GB (1931476992) temp space is allocated for autonomous_transaction block query. (see later two SQL Monitoring Reports).

By looking v$sqlarea output, direct_writes indicates that TEMP is used due to "SORT (v2)". rows_processed shows number of rows fetched (PLSQL arraysize=100).

 
select sql_id, last_active_time, executions, disk_reads, direct_writes, buffer_gets, rows_processed, substr(sql_text, 1, 78) sql_text --, v.*
  from v$sqlarea v
  where sql_text like 'SELECT%MONITOR sort%' and sql_id in ('6tuxpu523m1hf','dmuyj6fvhjqp6')
  order by last_active_time;
 
  SQL_ID        LAST_ACTIVE_TIME     EXECUTIONS DISK_READS DIRECT_WRITES BUFFER_GETS ROWS_PROCESSED SQL_TEXT
  ------------- -------------------- ---------- ---------- ------------- ----------- -------------- ------------------------------------------------------------------------------
  6tuxpu523m1hf 2021-OCT-19 06:32:01          1       4353        132447        4304            100 SELECT /*+ GATHER_PLAN_STATISTICS MONITOR sort_1 */ T12.A FROM TEMP_TAB_3K T11
  dmuyj6fvhjqp6 2021-OCT-19 06:36:04          1      40918        235592        5736       23996500 SELECT /*+ GATHER_PLAN_STATISTICS MONITOR sort_2 */ T22.A FROM TEMP_TAB_4K T21
v$bh shows that TEMP_TAB_30K (OBJD 4196992) has 4287 blocks in buffer cache, and TEMP_TAB_40K (OBJD 4306048) has 5716 blocks in buffer cache. Both are close to their number of blocks in v$tempseg_usage.

select objd, count(*)
  from v$bh
where objd in (select ktssoobjd from sys.x_ktsso g, v$session s where g.ktssoses=s.saddr and s.sid=789)
 group by objd order by objd;
 
     OBJD COUNT(*)
  ------- --------
  4196992     4287
  4218240       15
  4285824        1
  4287616        2
  4296832        1
  4306048     5716
  4306304        8
After half hour, the session throws ORA-01652 (256 is number of DB blocks, i.e. one TEMP extent in the test. extent is TEMP minimum unit):

  temp_tab_3k rows=3000
  temp_tab_4k rows=4000
  temp_tab_3k rows=3000
  declare
  *
  ERROR at line 1:
  ORA-01652: unable to extend temp segment by 256 in tablespace TEMP
  ORA-06512: at line 14
  ORA-06512: at line 14
  ORA-06512: at line 29
  ORA-06512: at line 29
 
  Elapsed: 00:30:38.31
By the way, output line "temp_tab_3k rows=3000" is used to show that all committed rows (temp table is created as "commit preserve rows") in main block is visible inside autonomous_transaction block (no committed rows still not visible). "temp_tab_4k rows=4000" shows that of autonomous_transaction block is visible inside main block. So temp table is session private, not transaction private.

The first output line "temp_tab_3k rows=3000" indicates that the first execution of autonomous_transaction block is successful since it requires only 2GB Temp space (see next section "SQL Monitoring Report").

The second output line "temp_tab_3k rows=3000" ended with ORA-01652 because temp_tab_4k and temp_tab_40k (defined as "commit preserve rows") are filled again with 4k and 40k rows (the size doubled to 8K and 80K rows respectively), and it demands 7GB Temp space (see next section "SQL Monitoring Report"). Together with 1GB already allocated for the main block query, whole 8GB are used up, and it results in ORA-01652.


4. SQL Monitoring Report


Now we can look SQL Monitoring Report for two above queries.


4.1 Query_1: 6tuxpu523m1hf (in main block)


We can see "BUFFER SORT" uses 75776 bytes memory, "SORT ORDER BY" needs 497KB bytes memory and additionally demands 1GB Temp space to meet sorting request. (BTW, "BUFFER SORT" is a BUFFER area, not used for "SORT").

Refer to above v$sql_workarea_active query for SQL_ID: 6tuxpu523m1hf, it shows that "BUFFER" uses ACTUAL_MEM_USED=75776; "SORT ORDER BY" takes ACTUAL_MEM_USED=508928 (508928/1024=497KB) plus additionally TEMPSEG_SIZE=1086324736 (1GB).

Both pair of (Read Reqs=136, Read Bytes=34MB) and (Write Reqs=4502, Write Bytes=1GB) stats indicate that each IO Read / Write is about 32 8K blocks.

Almost similar Elapsed Time(s) and Cpu Time(s) values suggest that the Sql executions is CPU intensive (sorting).

Before the test, we set "_optimizer_gather_stats_on_load"=true, so Rows (Estim) is exactly Rows (Actual).

select sys.dbms_sqltune.report_sql_monitor('6tuxpu523m1hf', report_level=>'all' , type=>'TEXT') from dual;
 
SQL Text
------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS MONITOR sort_1 */ T12.A FROM TEMP_TAB_3K T11, TEMP_TAB_30K T12 ORDER BY T12.A
 
Global Information
------------------------------
Status              :  EXECUTING          
 SQL ID              :  6tuxpu523m1hf      
 SQL Execution ID    :  16777216           
 Execution Started   :  10/19/2021 06:31:46
 First Refresh Time  :  10/19/2021 06:31:46
 Last Refresh Time   :  10/19/2021 06:32:01
 Duration            :  455s               
 Fetch Calls         :  1                  
 
Global Stats
================================================================================
| Elapsed |   Cpu   |    IO    | Fetch | Buffer | Read | Read  | Write | Write |
| Time(s) | Time(s) | Waits(s) | Calls |  Gets  | Reqs | Bytes | Reqs  | Bytes |
================================================================================
|      15 |      15 |     0.10 |     1 |   4304 |  136 |  34MB |  4502 |   1GB |
================================================================================
 
SQL Plan Monitoring Details (Plan Hash Value=2630616599)
======================================================================================================================================================================================
| Id   |        Operation        |     Name     |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read | Read  | Write | Write |  Mem  | Temp | Activity | Activity Detail |
|      |                         |              | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs | Bytes | Reqs  | Bytes |       |      |   (%)    |   (# samples)   |
======================================================================================================================================================================================
| -> 0 | SELECT STATEMENT        |              |         |       |        14 |     +2 |     1 |      100 |      |       |       |       |     . |    . |          |                 |
| -> 1 |   SORT ORDER BY         |              |     90M |  316K |        15 |     +1 |     1 |      100 |    1 | 248KB |  4502 |   1GB | 497KB |  1GB |    13.64 | Cpu (9)         |
| -> 2 |    MERGE JOIN CARTESIAN |              |     90M | 44087 |        14 |     +2 |     1 |      90M |      |       |       |       |     . |    . |     6.06 | Cpu (4)         |
| -> 3 |     TABLE ACCESS FULL   | TEMP_TAB_30K |   30000 |  1226 |        14 |     +2 |     1 |    30000 |  134 |  33MB |       |       |     . |    . |     1.52 | Cpu (1)         |
| -> 4 |     BUFFER SORT         |              |    3000 |  315K |        14 |     +2 | 30000 |      90M |      |       |       |       | 75776 |    . |     1.52 | Cpu (1)         |
|    5 |      TABLE ACCESS FULL  | TEMP_TAB_3K  |    3000 |     1 |         1 |     +2 |     1 |     3000 |    1 | 40960 |       |       |     . |    . |          |                 |
======================================================================================================================================================================================

4.2 Query_2: dmuyj6fvhjqp6 (in autonomous_transaction block)


We can see "BUFFER SORT" uses 102KB bytes memory, "SORT ORDER BY" needs 497KB bytes memory and additionally demands 2GB Temp space to meet sorting request ("BUFFER SORT" is a BUFFER area, not used for "SORT").

Refer to above v$sql_workarea_active query for SQL_ID: dmuyj6fvhjqp6, it shows that "BUFFER" uses ACTUAL_MEM_USED=104448 (104448/1024=102KB); "SORT ORDER BY" takes ACTUAL_MEM_USED=508928 (508928/1024=497KB) plus additionally TEMPSEG_SIZE=1931476992 (2GB).

Last column "Progress = 27%" can be used to estimate the remaining time.

select sys.dbms_sqltune.report_sql_monitor('dmuyj6fvhjqp6', report_level=>'all' , type=>'TEXT') from dual;
 
SQL Text
------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS MONITOR sort_2 */ T22.A FROM TEMP_TAB_4K T21, TEMP_TAB_40K T22 ORDER BY T22.A
 
Global Information
------------------------------
Status              :  EXECUTING          
 SQL ID              :  dmuyj6fvhjqp6      
 SQL Execution ID    :  16777216           
 Execution Started   :  10/19/2021 06:32:01
 First Refresh Time  :  10/19/2021 06:32:01
 Last Refresh Time   :  10/19/2021 06:39:54
 Duration            :  474s               
 Fetch Calls         :  439981              
 
Global Stats
================================================================================
| Elapsed |   Cpu   |    IO    | Fetch | Buffer | Read | Read  | Write | Write |
| Time(s) | Time(s) | Waits(s) | Calls |  Gets  | Reqs | Bytes | Reqs  | Bytes |
================================================================================
|      42 |      37 |     4.93 |  440K |   5736 | 2448 | 549MB |  8113 |   2GB |
================================================================================
 
SQL Plan Monitoring Details (Plan Hash Value=1990735535)
===========================================================================================================================================================================================================
| Id   |        Operation        |     Name     |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read | Read  | Write | Write |  Mem  | Temp | Activity |      Activity Detail      | Progress |
|      |                         |              | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs | Bytes | Reqs  | Bytes |       |      |   (%)    |        (# samples)        |          |
===========================================================================================================================================================================================================
| -> 0 | SELECT STATEMENT        |              |         |       |       473 |     +1 |     1 |      44M |      |       |       |       |     . |    . |     7.55 | Cpu (4)                   |          |
| -> 1 |   SORT ORDER BY         |              |    160M |  565K |       473 |     +1 |     1 |      44M | 2268 | 504MB |  8113 |   2GB | 497KB |  2GB |    47.17 | Cpu (22)                  |      27% |
|      |                         |              |         |       |           |        |       |          |      |       |       |       |       |      |          | direct path read temp (3) |          |
|    2 |    MERGE JOIN CARTESIAN |              |    160M | 81639 |        27 |     +1 |     1 |     160M |      |       |       |       |     . |    . |     3.77 | Cpu (2)                   |          |
|    3 |     TABLE ACCESS FULL   | TEMP_TAB_40K |   40000 |  1635 |        27 |     +1 |     1 |    40000 |  179 |  45MB |       |       |     . |    . |          |                           |          |
|    4 |     BUFFER SORT         |              |    4000 |  563K |        27 |     +1 | 40000 |     160M |      |       |       |       | 102KB |    . |    13.21 | Cpu (7)                   |          |
|    5 |      TABLE ACCESS FULL  | TEMP_TAB_4K  |    4000 |     2 |         1 |     +1 |     1 |     4000 |    1 | 57344 |       |       |     . |    . |          |                           |          |
===========================================================================================================================================================================================================
During the first run of both queries, they together consume about 3GB Temp space (37% of 8GB).

06:56:32 SQL > select t.*, round(100*(tablespace_size - free_space)/tablespace_size) percent_used from dba_temp_free_space t;
 
  TABLESPACE_NAME TABLESPACE_SIZE ALLOCATED_SPACE FREE_SPACE SHARED INST_ID PERCENT_USED
  --------------- --------------- --------------- ---------- ------ ------- ------------
  TEMP                 8388608000      8388608000 5268045824 SHARED                   37
In the second run, immediately before:

  ORA-01652: unable to extend temp segment by 256 in tablespace TEMP
 
    (Note: 256 is number of DB blocks, i.e, one TEMP extent in the test. extent is TEMP minimum unit).
Whole 8GB is consumed.

07:02:24 SQL > select t.*, round(100*(tablespace_size - free_space)/tablespace_size) percent_used from dba_temp_free_space t;
 
  TABLESPACE_NAME TABLESPACE_SIZE ALLOCATED_SPACE FREE_SPACE SHARED INST_ID PERCENT_USED
  --------------- --------------- --------------- ---------- ------ ------- ------------
  TEMP                 8388608000      8388608000          0 SHARED                  100
SQL Monitoring Report shows that the second run of this query consumes Mem(Max) is 80MB. And Temp(Max) reaches 7GB, which means that total 8GB are all used, hence ORA-01652.
(Note: the first and second run of SQL_ID: dmuyj6fvhjqp6 are designated by different SQL Execution ID: 16777216 and 16777217)

Global Information
------------------------------
Status              :  DONE (ERROR)       
 SQL ID              :  dmuyj6fvhjqp6      
 SQL Execution ID    :  16777217            
 Execution Started   :  10/19/2021 07:00:44
 First Refresh Time  :  10/19/2021 07:00:44
 Last Refresh Time   :  10/19/2021 07:02:36
 Duration            :  112s               
 Fetch Calls         :  1                  
 
Global Stats
===========================================================================================
| Elapsed |   Cpu   |    IO    |  Other   | Fetch | Buffer | Read | Read  | Write | Write |
| Time(s) | Time(s) | Waits(s) | Waits(s) | Calls |  Gets  | Reqs | Bytes | Reqs  | Bytes |
===========================================================================================
|     113 |     105 |     2.97 |     5.05 |     1 |  10691 | 3311 |   3GB | 29367 |   7GB |
===========================================================================================
 
SQL Plan Monitoring Details (Plan Hash Value=1990735535)
==================================================================================================================================================================================================
| Id |        Operation        |     Name     |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read | Read  | Write | Write |  Mem  | Temp  | Activity |       Activity Detail        |
|    |                         |              | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs | Bytes | Reqs  | Bytes | (Max) | (Max) |   (%)    |         (# samples)          |
==================================================================================================================================================================================================
|  0 | SELECT STATEMENT        |              |         |       |       113 |     +1 |     1 |        0 |      |       |       |       |     . |     . |    11.50 | Cpu (5)                      |
|    |                         |              |         |       |           |        |       |          |      |       |       |       |       |       |          | log file sequential read (8) |
|  1 |   SORT ORDER BY         |              |    160M |  565K |       101 |     +1 |     1 |        0 |      |       | 29370 |   7GB |  80MB |   7GB |    61.06 | Cpu (68)                     |
|    |                         |              |         |       |           |        |       |          |      |       |       |       |       |       |          | direct path write temp (1)   |
|  2 |    MERGE JOIN CARTESIAN |              |    160M | 81639 |       101 |     +1 |     1 |     597M |      |       |       |       |     . |     . |     9.73 | Cpu (11)                     |
|  3 |     TABLE ACCESS FULL   | TEMP_TAB_40K |   40000 |  1635 |       112 |     +1 |     1 |    74581 |  156 |  39MB |       |       |     . |     . |          |                              |
|  4 |     BUFFER SORT         |              |    4000 |  563K |       101 |     +1 | 74581 |     597M |      |       |       |       | 190KB |     . |    17.70 | Cpu (20)                     |
|  5 |      TABLE ACCESS FULL  | TEMP_TAB_4K  |    4000 |     2 |         1 |     +1 |     1 |     8000 |    1 | 57344 |       |       |     . |     . |          |                              |
==================================================================================================================================================================================================
After ORA-01652, Plsql block execution is terminated, and Temp space is released:
       
07:04:28 SQL > select t.*, round(100*(tablespace_size - free_space)/tablespace_size) percent_used from dba_temp_free_space t;
 
  TABLESPACE_NAME TABLESPACE_SIZE ALLOCATED_SPACE FREE_SPACE SHARED INST_ID PERCENT_USED
  --------------- --------------- --------------- ---------- ------ ------- ------------
  TEMP                 8388608000      8388608000 8241807360 SHARED                    2
By the way, with following query, we can observe both "SORT (v2)" and "HASH-JOIN" (SEGTYPE: 1 and 2) using Temp Space.
   
alter session set "_optimizer_gather_stats_on_load"=true;
insert /*+ append */ into temp_tab_40k select level, rpad('ABC', 1000, 'X') from dual connect by level <= 4e5;
commit;
 
select /*+ use_hash GATHER_PLAN_STATISTICS MONITOR sort_2 */ t1.a from temp_tab_40k t1, temp_tab_40k t2
 where t1.b = t2.b order by t1.a;
 
select sid, sql_id, operation_type, expected_size, actual_mem_used, tempseg_size, tablespace, active_time --, v.*
  from v$sql_workarea_active v
where sid = 195
order by v.sql_id, v.operation_type;

  SID SQL_ID        OPERATION_ EXPECTED_SIZE ACTUAL_MEM_USED  TEMPSEG_SIZE TABLESPACE ACTIVE_TIME
  --- ------------- ---------- ------------- --------------- ------------- ---------- -----------
  195 aj3f52afjwua7 HASH-JOIN     83,886,080      89,966,592   840,957,952 TEMP       136,327,805
  195 aj3f52afjwua7 SORT (v2)     64,120,832      34,364,416 6,035,603,456 TEMP       134,897,069

5. Shrinking Temporary Tablespaces


Oracle TEMP space can be allocated, but not de-allocated as explained in MOS "Queries To Monitor Temporary Tablespace Usage (Doc ID 289894.1)":

  Multiple transactions which need a sort on disk, can share the same sort segment, however, they cannot share the same extent.
  The sort segment expands by allocating new extents. The sort extents are not de-allocated while the instance is running,
  but are marked as free and can be re-used as required. Therefore, the sort segment grows to a certain steady state.
During the first run of both queries, although we are using less than 3GB Temp space, we get ORA-03297 when shrinking it down to 6G after 9 minutes.

ORA-03297 said "file contains used data beyond requested RESIZE value", which probably means that there exist temp extents distributed beyond 6G boundary inside 8G temp file (we will see it later in Tempfile Dump).

06:42:32 SQL > ALTER TABLESPACE temp SHRINK SPACE KEEP 6G;
 
  ERROR at line 1:
  ORA-03297: file contains used data beyond requested RESIZE value
  Elapsed: 00:09:41.83
DB alert.log shows only 1032K (8192000 - 8190968) are reduced.

  2021-10-19T06:42:32.351933+02:00
  ALTER TABLESPACE temp SHRINK SPACE KEEP 6G
  2021-10-19T06:42:32.381387+02:00
  Resize operation completed for file# 3073, old size 8192000K, new size 8190968K
  2021-10-19T06:52:14.174181+02:00
  ORA-3297 signalled during: ALTER TABLESPACE temp SHRINK SPACE KEEP 6G...
Unix also shows this only 1032k change (8388616192-8387559424=1056768=1032K):

  $ > ls -ltr temp01.dbf
    -rw-r----- 1 oracle dba 8388616192 Oct 19 06:40 temp01.dbf
 
  $ > ls -ltr temp01.dbf
    -rw-r----- 1 oracle dba 8387559424 Oct 19 06:53 temp01.dbf
11.2.0.4 introduced ORA-03297 is documented as:
(see MOS "Resize Of Temporary Tablespace with KEEP keyword not working (Doc ID 1663229.1)")

  03297, 00000, "file contains used data beyond requested RESIZE value"
  // *Cause:  Some portion of the file in the region to be trimmed is
  //          currently in use by a database object
  // *Action: Drop or move segments containing extents in this region prior to
  //          resizing the file, or choose a resize value such that only free
  //          space is in the trimmed.
After Plsql block running terminated (with ORA-01652), Temp space is released, only 2% of 8GB is used.
           
07:04:28 SQL > select t.*, round(100*(tablespace_size - free_space)/tablespace_size) percent_used from dba_temp_free_space t;
 
  TABLESPACE_NAME TABLESPACE_SIZE ALLOCATED_SPACE FREE_SPACE SHARED INST_ID PERCENT_USED
  --------------- --------------- --------------- ---------- ------ ------- ------------
  TEMP                 8388608000      8388608000 8241807360 SHARED                    2
Shrinking it again, Temp space is quickly resized down to 6GB.

07:05:04 SQL > ALTER TABLESPACE temp SHRINK SPACE KEEP 6G;
 
  Tablespace altered.
  Elapsed: 00:00:00.15
DB alert.log shows that Resize operation completed with new size 6409208K (about 6GB):

  2021-10-19T07:05:04.268529+02:00
  ALTER TABLESPACE temp SHRINK SPACE KEEP 6G
  2021-10-19T07:05:04.418246+02:00
  Resize operation completed for file# 3073, old size 8190968K, new size 6291456K
  Completed: ALTER TABLESPACE temp SHRINK SPACE KEEP 6G
Unix shows:

  $ > ls -ltr temp01.dbf
    -rw-r----- 1 oracle dba 6442459136 Oct 19 07:05 temp01.dbf
To efficiently shrink TABLESPACE temp, it is advisable to set database in restricted mode, then run above shrink command.


6. Monitoring Queries


Database Journal: "Monitoring Oracle's Temp Usage" gives some queries to monitor temp usage.

Oracle MOS:

   "Queries To Monitor Temporary Tablespace Usage (Doc ID 289894.1)",
   "How Can Temporary Segment Usage Be Monitored Over Time? (Doc ID 364417.1)"
   "Primary Note: Overview of Oracle Temporary Tablespaces (Doc ID 1498442.1)"
have descriptions about temporary (sort) segment and monitoring.

In this section, we collect a few often used Temp monitoring queries and classify them by their behaviours.

In order to correlate different stats in the queries, we first shrink Temp tablespace to 5000M (temp file maxsize 8000M).
Then fill a temp table with 1622M (see temp_tab in Section 2).

ALTER TABLESPACE temp SHRINK SPACE KEEP 5000M;
 
insert /*+ append */ into temp_tab select level, rpad('ABC', 7000, 'X') from dual connect by level <= 2e5;
commit;

--==== 1. List TEMP Settings ====--
 
$ > ls -ltr temp*
  -rw-r----- 1 oracle dba 5242888192 Oct 19 17:50 temp01.dbf
 
select * from dba_temp_files;
 
  FILE_NAME  FILE_ID TABLESPACE      BYTES BLOCKS STATUS RELATIVE_FNO AUTOE   MAXBYTES  MAXBLOCKS INCREMENT_BY USER_BYTES USER_BLOCKS SHARED
  ---------- ------- ---------- ---------- ------ ------ ------------ ----- ---------- ---------- ------------ ---------- ----------- ------
  temp01.dbf       1 TEMP       5242880000 640000 ONLINE            1 YES   8388608000    1024000        12800 5240782848      639744 SHARED

--==== 2. Monitor TEMP Allocations ====--
-- list current used space per session within allocated space (total USED_EXTENTS=811)
select session_addr, sum(blocks) used_blocks, (sum(blocks)*8192/1024/1024) used_mb, sum(blocks)*8192 used_bytes, sum(blocks)/256 used_extents
  from v$tempseg_usage
 group by rollup(session_addr);
 
  SESSION_ADDR     USED_BLOCKS USED_MB USED_BYTES USED_EXTENTS
  ---------------- ----------- ------- ---------- ------------
  00000000B70D97B8         256       2    2097152            1
  00000000B72A7418      206336    1612 1690304512          806     -- session which filled temp_tab
  00000000B8B6FF78         256       2    2097152            1
  00000000B8B92E28         256       2    2097152            1
  00000000B8D18540         256       2    2097152            1
  00000000B8D42BA8         256       2    2097152            1
                        207616    1622 1700790272          811     -- total used
 
-- list total, allocated, and free space
select t.*, (tablespace_size/8192) tablespace_blocks, (tablespace_size/8192/256) tablespace_extents,
      (allocated_space/8192) allocated_blocks, (allocated_space/8192/256) allocated_extents
  from dba_temp_free_space t;
 
  TABLESPACE TABLESPACE_SIZE ALLOCATED_SPACE FREE_SPACE SHARED TABLESPACE_BLOCKS TABLESPACE_EXTENTS ALLOCATED_BLOCKS ALLOCATED_EXTENTS
  ---------- --------------- --------------- ---------- ------ ----------------- ------------------ ---------------- -----------------
  TEMP            5242880000      1707081728 3539992576 SHARED            640000               2500           208384               814
 
-- list total TEMP used and free
select * from v$temp_space_header;
 
  TABLESPACE FILE_ID BYTES_USED BLOCKS_USED BYTES_FREE BLOCKS_FREE RELATIVE_FNO
  ---------- ------- ---------- ----------- ---------- ----------- ------------
  TEMP             1 1707081728      208384 3535798272      431616            1
 
-- list used and total
select s.tot_used_blocks, f.total_blocks, (s.tot_used_blocks/f.total_blocks)*100 as "percent used"
from (select sum(used_blocks) tot_used_blocks from v$sort_segment where tablespace_name='TEMP') s,
      (select sum(blocks)      total_blocks    from dba_temp_files where tablespace_name='TEMP') f;
 
  TOT_USED_BLOCKS TOTAL_BLOCKS percent used
  --------------- ------------ ------------
           207616       640000        32.44
 
-- list Allocated and Un-Allocated TEMP         
--   owner=0 are free extents (un-allocated), owner!=0 are allocated extents.
--   For details, see later section: Tempfile Dump
select owner, count(*), sum(bytes), sum(blocks) from v$temp_extent_map group by owner;
 
  OWNER COUNT(*) SUM(BYTES) SUM(BLOCKS)
  ----- -------- ---------- -----------
      1      813 1704984576      208128     -- Allocated (CACHED) TEMP
      0     1686 3535798272      431616     -- Un-Allocated       TEMP

--==== 3. Monitor Allocated TEMP Utilization (Efficiency: Used against Allocated, or Used against Cached) ====--
-- list cached stats and utilization (cache efficiency)    
--   cached temporary space usage (not clear if they are blocks of v$bh.temp='Y'. See following query Note of v$sort_segment)
select t.*, 100*blocks_used/blocks_cached block_percent_used from v$temp_extent_pool t;
 
  TABLESPACE FILE_ID EXTENTS_CACHED EXTENTS_USED BLOCKS_CACHED BLOCKS_USED BYTES_CACHED BYTES_USED RELATIVE_FNO BLOCK_PERCENT_USED
  ---------- ------- -------------- ------------ ------------- ----------- ------------ ---------- ------------ ------------------
  TEMP             1            813          811        208128      207616   1704984576 1700790272            1         99.7539975
 
-- list sort_segment stats and utilization of allocated blocks (TOTAL_BLOCKS). TOTAL_BLOCKS is allocated blocks in v$temp_extent_map
--   (EXTENT_HITS: Number of times an unused extent was found in the pool. 
--                 Probably accumulated (EXTENTS_ALLOCATED - EXTENTS_USED) in v$temp_extent_pool)
select current_users, total_blocks, used_blocks, free_blocks, max_blocks, max_used_blocks, max_sort_blocks
     ,extent_hits, freed_extents, free_requests
     ,(select sum(blocks) from v$tempseg_usage) tempseg_blocks
     ,100*used_blocks/total_blocks block_percent_used --, t.*
  from v$sort_segment t;
 
  CURRENT_USERS TOTAL_BLOCKS USED_BLOCKS FREE_BLOCKS MAX_BLOCKS MAX_USED_BLOCKS MAX_SORT_BLOCKS EXTENT_HITS FREED_EXTENTS FREE_REQUESTS TEMPSEG_BLOCKS BLOCK_PERCENT_USED
  ------------- ------------ ----------- ----------- ---------- --------------- --------------- ----------- ------------- ------------- -------------- ------------------
             10       208128      207616         512    1023744         1023744          951040      104014         15045            61         207616         99.7539975
                   
--==== 4. Monitor Workarea ====--
--Connect v$sql_workarea_active with v$sql_workarea
select *
  from v$sql_workarea_active a, v$sql_workarea w
 where a.workarea_address = w.workarea_address and a.sid = 789;
 
-- Connect v$sql_workarea_active with v$sql_workarea and v$tempseg_usage
select * from v$sql_workarea_active a, v$sql_workarea w, v$tempseg_usage t
 where a.workarea_address = w.workarea_address
   and a.tablespace = t.tablespace(+) and a.segrfno# = t.segrfno#(+) and a.segblk# = t.segblk#(+) and a.sid = 789;

--==== 5. Monitor executions. direct_writes indicates TEMP used; rows_processed indicates number of fetches (PLSQL arraysize=100) ====--
select sql_id, first_load_time, last_active_time, executions, disk_reads, direct_writes, buffer_gets, rows_processed, v.*
  from v$sqlarea v
  where sql_text like 'SELECT%MONITOR sort%' or sql_id in ('6tuxpu523m1hf','dmuyj6fvhjqp6');

--==== 6. Monitor PGA and TEMP allocation ====--
select pga_allocated, temp_space_allocated, v.*
  from v$active_session_history v where session_id = 789
  order by sample_time desc;

--==== 7. Monitor UNDO ====--
-- show tempundo stats, for example, 
--   SSOLDERRCNT   for count of ORA-01555 snapshot too old, 
--   NOSPACEERRCNT for count of ORA-1652: unable to extend temp segment

select * from v$tempundostat; --where ssolderrcnt > 0;


-- normal permanent undo segment (not including temp undo segment), which are displayed in AWR 
--(hence AWR does not contain temp undo segment stats, 
-- for example: ORA-01555 count (SSOLDERRCNT) for tempundo is not showed in AWR)

select * from v$undostat;
select * from dba_hist_undostat;

-- contain SYSTEM undo segment and normal permanent undo segment (not including temp undo segment) 

select * from dba_rollback_segs;

7. ORA-01652 Trace File


In section "SQL Executions", if we turn on ORA-01652 trace:

  alter system set events '1652 trace name errorstack level 3';
Here some excerpted sections:

ORA-01652: unable to extend temp segment by 256 in tablespace TEMP
 
----- Current SQL Statement for this session (sql_id=dmuyj6fvhjqp6) -----
SELECT /*+ GATHER_PLAN_STATISTICS MONITOR sort_2 */ T22.A FROM TEMP_TAB_4K T21, TEMP_TAB_40K T22 ORDER BY T22.A
 
----- Call Stack Trace -----
  FRAME [16] (ksesec2()+205 -> kgeselv())
  FRAME [17] (ktsxterr()+592 -> ksesec2())
  FRAME [18] (ktfttgex()+953 -> ktsxterr())
  FRAME [19] (ktstallocext()+396 -> ktfttgex())
  FRAME [20] (ktstgrowseg()+857 -> ktstallocext())
  FRAME [21] (ktstadr_addextent()+757 -> ktstgrowseg())
  FRAME [22] (ktrsexecExecuteInExcepHdlr()+327 -> ktstadr_addextent())
  FRAME [23] (ktrsexec()+193 -> ktrsexecExecuteInExcepHdlr())
  FRAME [24] (ktataddextent()+122 -> ktrsexec())
  FRAME [25] (stsGetExtent()+76 -> ktataddextent())
  FRAME [26] (ssmalc()+1232 -> stsGetExtent())
  FRAME [27] (sdbinb()+259 -> ssmalc())
  FRAME [28] (sdbputqb()+931 -> sdbinb())
  FRAME [29] (smboWrt2()+202 -> sdbputqb())
  FRAME [30] (smboFinishSpill()+194 -> smboWrt2())
  FRAME [31] (smboSort()+377 -> smboFinishSpill())
  FRAME [32] (smboWrt()+540 -> smboSort())
  FRAME [33] (smboPut()+1273 -> smboWrt())
  FRAME [34] (qersoSORowP()+242 -> smboPut())
  FRAME [35] (qerstRowP()+737 -> qersoSORowP())
  FRAME [36] (qerstRowP()+737 -> qerstRowP())
  FRAME [37] (qersoFetchSimple()+428 -> qerstRowP())
  FRAME [38] (qersoFetch()+210 -> qersoFetchSimple())
  FRAME [39] (qerstFetch()+449 -> qersoFetch())
  FRAME [40] (qerjoCartesianFetch()+3305 -> qerstFetch())
  FRAME [41] (qerstFetch()+449 -> qerjoCartesianFetch())
 
----- Session Wait History: ----- 
        elapsed time of 0.729063 sec since last wait
     0: waited for 'direct path write temp'
        file number=0xc01, first dba=0xf9e79, block cnt=0x1f
        wait_id=165501 seq_num=34439 snap_id=1
        wait times: snap=0.006435 sec, exc=0.006435 sec, total=0.006435 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.003242 sec of elapsed time
     1: waited for 'local write wait'
        file#=0xc01, block#=0x2, =0x0
        wait_id=165500 seq_num=34438 snap_id=1
        wait times: snap=0.000235 sec, exc=0.000235 sec, total=0.000235 sec
        wait times: max=infinite
        wait counts: calls=1 os=1
        occurred after 0.000002 sec of elapsed time
    ...
     9: waited for 'local write wait'
        file#=0xc01, block#=0x4, =0x0
        wait_id=165492 seq_num=34430 snap_id=1
        wait times: snap=0.000118 sec, exc=0.000118 sec, total=0.000118 sec
        wait times: max=infinite
        wait counts: calls=1 os=1
        occurred after 0.000012 sec of elapsed time
       
    sample interval: 1 sec, max history 120 sec
    ---------------------------------------------------
      [43 samples,                                         07:02:03 - 07:02:25]
        not in wait at each sample
      [1 sample,                                                      07:02:02]
        waited for 'direct path write temp', seq_num: 31433
         p1: 'file number'=0xc01
          p2: 'first dba'=0x4051b
          p3: 'block cnt'=0x1f
          time_waited: 0.035970 sec (sample interval: 0 sec)
   
----- data_block_dump ----- 
BH (0x15cf77238) file#: 3073 rdba: 0x00488e20 (1/560672) class: 1 ba: 0x15c3aa000
  set: 13 pool: 3 bsz: 8192 bsi: 0 sflg: 0 pwc: 0,25
  dbwrid: 0 obj: 4306048 objn: 4080451 tsn: [0/3] afn: 3073 hint: f
 
BH (0x145fc3b10) file#: 3073 rdba: 0x00400002 (1/2) class: 13 ba: 0x145a78000
  set: 15 pool: 3 bsz: 8192 bsi: 0 sflg: 0 pwc: 0,25
  dbwrid: 0 obj: -1 objn: 972 tsn: [0/3] afn: 3073 hint: f
 
BH (0x13ef72378) file#: 3073 rdba: 0x00400004 (1/4) class: 12 ba: 0x13e33a000
  set: 13 pool: 3 bsz: 8192 bsi: 0 sflg: 0 pwc: 0,25
  dbwrid: 0 obj: -1 objn: 0 tsn: [0/3] afn: 3073 hint: f
 
  File Space Bitmap Block:
  BitMap Control:
  RelFno: 1, BeginBlock: 1015936, Flag: 1, First: 31, Free: 3937
  Bit number 0 - owned by instance 1
  Bit number 1 - owned by instance 1
  ...
  Bit number 30 - owned by instance 1
  Bit number 31 is free
  ...
  Bit number 3967 is free
 
BH (0xfff87228) file#: 3073 rdba: 0x00400003 (1/3) class: 12 ba: 0xff516000
  set: 16 pool: 3 bsz: 8192 bsi: 0 sflg: 0 pwc: 0,25
  dbwrid: 0 obj: -1 objn: 0 tsn: [0/3] afn: 3073 hint: f
 
  File Space Bitmap Block:
  BitMap Control:
  RelFno: 1, BeginBlock: 128, Flag: 1, First: 3968, Free: 0
  Bit number 0 - owned by instance 1
  Bit number 1 - owned by instance 1
  ...
  Bit number 3966 - owned by instance 1

8. Tempfile Dump


To further understand Temp Space allocation, we can dump temp File Space blocks (1-6).

alter system dump tempfile 1 BLOCK MIN 1 BLOCK MAX 6;
Here some parts excerpted from dump file:

BH (0x145fc3b10) file#: 3073 rdba: 0x00400002 (1/2) class: 13 ba: 0x145a78000
  frmt: 0x02 chkval: 0x0000 type: 0x1d=KTFB Bitmapped File Space Header
  File Space Header Block:
  Header Control:
  RelFno: 1, Unit: 256, Size: 640000, Flag: 9(0x9)
  AutoExtend: YES, Increment: 12800, MaxSize: 1024000
  Initial Area: 126, Tail: 639871, First: 811, Free: 1686
           
BH (0xfff87228) file#: 3073 rdba: 0x00400003 (1/3) class: 12 ba: 0xff516000
  frmt: 0x02 chkval: 0x0000 type: 0x1e=KTFB Bitmapped File Space Bitmap
  File Space Bitmap Block:
  BitMap Control:
  RelFno: 1, BeginBlock: 128, Flag: 1, First: 811, Free: 3155
    Bit number 0 - owned by instance 1
    Bit number 1 - owned by instance 1
    ...
    Bit number 810 - owned by instance 1
    Bit number 811 is free
    ...
    Bit number 1450 - owned by instance 1
    ...
    Bit number 1532 - owned by instance 1
    ...
    Bit number 3967 is free
 
BH (0x13ef72378) file#: 3073 rdba: 0x00400004 (1/4) class: 12 ba: 0x13e33a000
  frmt: 0x02 chkval: 0x0000 type: 0x1e=KTFB Bitmapped File Space Bitmap
  File Space Bitmap Block:
  BitMap Control:
  RelFno: 1, BeginBlock: 1015936, Flag: 1, First: 0, Free: 3968
    Bit number 0 is free
    ...
    Bit number 3967 is free
 
buffer tsn: 3 rdba: 0x00400005 (1/5)
  frmt: 0x02 chkval: 0xbf44 type: 0x1e=KTFB Bitmapped File Space Bitmap
  File Space Bitmap Block:
  BitMap Control:
  RelFno: 1, BeginBlock: 2031744, Flag: 1, First: 0, Free: 3968
    Bit number 0 is free
    ...
    Bit number 3967 is free
 
buffer tsn: 3 rdba: 0x00400006 (1/6)
  frmt: 0x02 chkval: 0x3f76 type: 0x1e=KTFB Bitmapped File Space Bitmap
  File Space Bitmap Block:
  BitMap Control:
  RelFno: 1, BeginBlock: 3047552, Flag: 1, First: 0, Free: 3968
    Bit number 0 is free
    ...
    Bit number 3967 is free
Block 2 is File Space Header (rdba: 0x00400002 (1/2) class: 13), which stores temp file settings (exposed in dba_temp_files), for example, file size is 640000 blocks, first free extent is 811, free extents is 1686 (exposed in v$temp_extent_map).

Blocks 3 to 6 are File Space Bitmap (class: 12), which stores extent Bitmap. For example, in Block 3 (rdba: 0x00400003 (1/3) class: 12), Allocated extents are marked as "owned by instance 1" (that probably points to "owner" column of v$temp_extent_map). UN-Allocated extents are marked as "free". Allocated Bits spread over three ranges: from 0 to 810 (811 Bits), and Bit 1450, 1532. 3155 Bits are free (3967-810-2, 2 are Bit number 1450 and 1532). Each Bitmap block can have 3968 rows.

File Space Bitmap is exposed in v$temp_extent_map, we can run a query to list all allocated extents:

select * from (
select rownum-1 rn, t.* from v$temp_extent_map t
) where owner=1;
 
  Bit number TABLESPACE FILE_ID BLOCK_ID   BYTES BLOCKS OWNER RELATIVE_FNO
  ---------- ---------- ------- -------- ------- ------ ----- ------------
           0 TEMP             1      128 2097152    256     1            1
           1 TEMP             1      384 2097152    256     1            1
           2 TEMP             1      640 2097152    256     1            1
        ....  
         810 TEMP             1   207488 2097152    256     1            1
        1450 TEMP             1   371328 2097152    256     1            1
        1532 TEMP             1   392320 2097152    256     1            1
 
  813 rows selected.
From Block dump and above query output, we can see that extents are not contiguously allocated, there exist 720 (1533-813) free extents under last allocated Bit number 1532. That is probably why we cannot shrink down to only allocated extents, and even get ORA-3297 when shrinking TABLESPACE temp.