Friday, January 2, 2026

Blog List

"Disk file operations I/O" with (FileOperation: 8, fileno: 0, filetype: 8) Test

In this blog, we will make a test to demonstrate Oracle Wait Event "Disk file operations I/O" with
(P1/FileOperation: 8, P2/fileno: 0, P3/filetype: 8).

Note: Tested in Oracle 19.27


1. Test Setup


The database auditing is configured as follows:

NAME                      TYPE        VALUE
------------------------- ----------- --------------------------------
audit_file_dest           string      /home/oracle/suk/slow_dir3/adump
audit_sys_operations      boolean     TRUE
audit_trail               string      OS
audit_file_dest is mounted on a slow file system with 5000 ms read/write delay.
(See Blog: Oracle Database Stuck on DB Writer Wait Event: "oradebug request completion" ).


2. Wait Event Tracing


2.1 10046 Trace


We start system-wide 10046 trace by:

alter system set events '10046 trace name context forever, level 12';  

2.2 strace


Find TNS listener process and strace it:

oracle@db3audit:db3audit > ps -efl |grep tns
  0 S oracle    728055       1  0  80   0 - 40462 do_epo Sep27 ?        00:00:15 /orabin/app/oracle/product/19.27.0.0./bin/tnslsnr LISTENER_oradb3audit             

oracle@db3audit:db3audit >  strace -ttTfy -p 728055 -o strace_output_2.txt

2.3 Sqlplus Session


Now we open a Sqlplus session by

SQL > sqlplus k/k@db3audit

2.4 10046 Trace Output


Trace file /orabin/app/oracle/admin/db3audit/diag/rdbms/db3audit/db3audit/trace/db3audit_ora_1072963.trc
Build label:    RDBMS_19.27.0.0.0DBRU_LINUX.X64_250331
Oracle process number: 57
Unix process pid: 1072963, image: oracle@db3audit
*** MODULE NAME:(sqlplus.exe) 2025-12-23T10:11:56.689481+01:00

*** 2025-12-23T10:12:01.847101+01:00
WAIT #0: nam='Disk file operations I/O' ela= 5100597 FileOperation=8 fileno=0 filetype=8 obj#=-1 tim=18667193313139
It shows "ela= 5100597" microseconds with "FileOperation=8 fileno=0 filetype=8".
So we can say that in this case, "FileOperation=8" is file open, and "filetype=8" is OS Audit file.

After test, stop 10046 trace by (otherwise too many trace files):

  alter system set events '10046 trace name context off';

2.5 strace Output


We can see that Sqlplus Process 1072963 was spawned by TNS Process 728055:

oracle@db3audit:db3audit > strace -ttTfy -p 728055 -o strace_output_3.txt
./strace: Process 728055 attached
./strace: Process 1072962 attached
./strace: Process 1072963 attached
Open strace_output_2.txt, it shows:

1072963 10:11:56.746551 openat(AT_FDCWD, "/home/oracle/suk/slow_dir3/adump/db3audit_ora_1072963_20251223101156746537740094.aud", O_RDWR|O_CREAT|O_EXCL, 0640) = 10
                                 <5.098572>
1072963 10:12:01.845204 write(10, "Audit file ", 11) = 11 <0.000058>
1072963 10:12:01.845315 write(10, "/home/oracle/suk/slow_dir3/adump"..., 84) = 84 <0.000023>
1072963 10:12:01.845372 write(10, "\n", 1) = 1 <0.000022>
1072963 10:12:01.845433 write(10, "Oracle Database 19c Enterprise E"..., 90) = 90 <0.000021>
1072963 10:12:01.845482 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.845533 write(10, "Build label:    RDBMS_19.27.0.0."..., 54) = 54 <0.000020>
1072963 10:12:01.845582 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.845639 write(10, "ORACLE_HOME:    /orabin/app/orac"..., 64) = 64 <0.000020>
1072963 10:12:01.845689 write(10, "System name:\tLinux\n", 19) = 19 <0.000020>
1072963 10:12:01.845738 write(10, "Node name:\tdb3audit\n", 20) = 20 <0.000021>
1072963 10:12:01.845796 write(10, "Release:\t4.18.0-372.9.1.el8.x86_"..., 35) = 35 <0.000020>
1072963 10:12:01.845845 write(10, "Version:\t#1 SMP Fri Apr 15 22:12"..., 45) = 45 <0.000021>
1072963 10:12:01.845895 write(10, "Machine:\tx86_64\n", 16) = 16 <0.000021>
1072963 10:12:01.845953 write(10, "CLID:\tP\n", 8) = 8 <0.000021>
1072963 10:12:01.846010 write(10, "Instance name: db3audit\n", 24) = 24 <0.000062>
1072963 10:12:01.846105 write(10, "Instance number: 1\n", 19) = 19 <0.000020>
1072963 10:12:01.846161 write(10, "Database name: db3audit ", 24) = 24 <0.000023>
1072963 10:12:01.846215 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846264 write(10, "Database unique name: db3audit ", 31) = 31 <0.000020>
1072963 10:12:01.846313 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846388 write(10, "Database id: 1059538969\n", 24) = 24 <0.000021>
1072963 10:12:01.846442 write(10, "Database role: PRIMARY\n", 23) = 23 <0.000020>
1072963 10:12:01.846491 write(10, "Redo thread mounted by this inst"..., 40) = 40 <0.000020>
1072963 10:12:01.846541 write(10, "Oracle process number: 57\n", 26) = 26 <0.000021>
1072963 10:12:01.846592 write(10, "Unix process pid: 1072963, image"..., 49) = 49 <0.000019>
1072963 10:12:01.846640 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846689 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846762 write(10, "Tue Dec 23 10:12:01 2025 +01:00\n", 32) = 32 <0.000021>
1072963 10:12:01.846812 write(10, "LENGTH: \"408\"", 13) = 13 <0.000021>
1072963 10:12:01.846860 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846909 write(10, "SESSIONID:[9] \"102147891\" ENTRYI"..., 406) = 406 <0.000030>
1072963 10:12:01.846973 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.847023 write(10, "\n", 1) = 1 <0.000021>
The above output shows that openat "/home/oracle/suk/slow_dir3/adump/db3audit_ora_1072963_20251223101156746537740094.aud" returns FD = 10,
and it takes 5.098572 seconds from 10:11:56 to 10:12:01.

Look OS audit file, its name is composed by DB SID: db3audit, Unix process pid: 1072963, and creating timestamp: 20251223101156, which is 2025-12-23 10:11:56.
In the OS audit file, the first timestamp "10:12:01" is recorded for session LOGON, which has about 5 seconds delay (from 10:11:56 to 10:12:01),
and the second timestamp "10:28:44" is for session LOGOFF. Here the full audit file:

oracle@db3audit:db3audit > cat /home/oracle/suk/slow_dir3/adump/db3audit_ora_1072963_20251223101156746537740094.aud
Audit file /home/oracle/suk/slow_dir3/adump/db3audit_ora_1072963_20251223101156746537740094.aud
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
Build label:    RDBMS_19.27.0.0.0DBRU_LINUX.X64_250331
ORACLE_HOME:    /orabin/app/oracle/product/19.27.0.0.
System name:	Linux
Node name:	db3audit
Release:	4.18.0-372.9.1.el8.x86_64
Version:	#1 SMP Fri Apr 15 22:12:19 EDT 2022
Machine:	x86_64
CLID:	P
Instance name: db3audit
Instance number: 1
Database name: db3audit 
Database unique name: db3audit 
Database id: 1059538969
Database role: PRIMARY
Redo thread mounted by this instance: 1
Oracle process number: 57
Unix process pid: 1072963, image: oracle@db3audit

Tue Dec 23 10:12:01 2025 +01:00
LENGTH: "408"
SESSIONID:[9] "102147891" ENTRYID:[1] "1" STATEMENT:[1] "1" USERID:[1] "K" USERHOST:[16] "SYS\MACHINE12345" TERMINAL:[12] "MACHINE12345" ACTION:[3] "100" RETURNCODE:[1] "0" COMMENT$TEXT:[126] "Authenticated by: DATABASE;AUTHENTICATED IDENTITY: K; Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.44.105.100)(PORT=50160))" OS$USERID:[7] "KS00383" DBID:[10] "1059538969" PRIV$USED:[1] "5" CURRENT_USER:[1] "K"

Thu Dec 23 10:28:44 2025 +01:00
LENGTH: "231"
SESSIONID:[9] "102147891" ENTRYID:[1] "1" USERID:[1] "K" ACTION:[3] "101" RETURNCODE:[1] "0" LOGOFF$PREAD:[5] "35037" LOGOFF$LREAD:[6] "166495" LOGOFF$LWRITE:[1] "4" LOGOFF$DEAD:[1] "0" DBID:[10] "1059538969" SESSIONCPU:[3] "258"

2.6 v$active_session_history and Oracle Performance Views


select sample_time, program, event, p1, p2, p3, p1text, p2text, p3text
  from v$active_session_history t 
 where event = 'Disk file operations I/O' and p1=8 and p2=0 and p3=8 
   and sample_time >= timestamp '2025-12-28 10:31:22.549'
 order by t.sample_time;
 
 
  SAMPLE_TIME         PROGRAM      EVENT                     P1  P2  P3  P1TEXT         P2TEXT  P3TEXT
  ------------------- ------------ ------------------------ --- --- --- --------------- ------- --------
  2025-12-22 10:11:57 sqlplus.exe  Disk file operations I/O   8   0   8  FileOperation  fileno  filetype
  2025-12-22 10:11:58 sqlplus.exe  Disk file operations I/O   8   0   8  FileOperation  fileno  filetype
  2025-12-22 10:11:59 sqlplus.exe  Disk file operations I/O   8   0   8  FileOperation  fileno  filetype
  2025-12-22 10:12:00 sqlplus.exe  Disk file operations I/O   8   0   8  FileOperation  fileno  filetype
  2025-12-22 10:12:01 sqlplus.exe  Disk file operations I/O   8   0   8  FileOperation  fileno  filetype


select filetype_id, filetype_name, function_id, function_name, sum(small_read_reqs), sum(small_write_reqs), sum(number_of_waits), sum(wait_time) 
from  v$iostat_function_detail 
where filetype_id = 8
group by filetype_id, filetype_name, function_id, function_name
order by filetype_id, function_id;

  FILETYPE_ID FILETYPE FUNCTION_ID FUNCTION_NAME        SUM(SMALL_READ_REQS) SUM(SMALL_WRITE_REQS) SUM(NUMBER_OF_WAITS) SUM(WAIT_TIME)
  ----------- -------- ----------- -------------------- -------------------- --------------------- -------------------- --------------
            8 Other              2 LGWR                                    0                     0               136398          50459
            8 Other              7 Recovery                                0                     0                    1              0
            8 Other              8 Buffer Cache Reads                      0                     0                  245            283
            8 Other             13 Others                                  0                     0               273272         120046


-- also in v$session_event, dba_hist_system_event
select event, total_waits, time_waited, time_waited_micro, event_id, wait_class_id, wait_class#, wait_class from v$system_event 
 where event = 'Disk file operations I/O';

  EVENT                     TOTAL_WAITS TIME_WAITED TIME_WAITED_MICRO   EVENT_ID WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS
  ------------------------- ----------- ----------- ----------------- ---------- ------------- ----------- ----------
  Disk file operations I/O        67226       28853         288534923  166678035    1740759767           8 User I/O

-- no rows in v$iostat_file
select * from v$iostat_file where filetype_id = 8;
	no rows selected

2.7 AWR and ASH


TOP Wait Event "Disk file operations I/O" in AWR and ASH looks like:

Top 10 Foreground Events by Total Wait Time

    Event Waits Total Wait Time (sec) Avg Wait % DB time Wait Class
    Disk file operations I/O 2,706 11.4 4.20ms 35.4 User I/O

    Top Event P1/P2/P3 Values

      Event % Event P1, P2, P3 Values % Activity Parameter 1 Parameter 2 Parameter 3
      Disk file operations I/O 45.45 "8","0","8" 45.45 FileOperation fileno filetype


      3. bpftrace


      Start following bpftrace script and then open a Sqlplus session.

      The script output shows file open (sys_enter_openat) "Duration (us): 5089844", and its Oracle CallStack.
      
      BPFTRACE_STRLEN=128 bpftrace -e 'tracepoint:syscalls:sys_enter_openat /strncmp("oracle", comm, 6) == 0 && strncmp("/home/oracle/suk/slow_dir3", str(args->filename), 26) == 0 / {
                     time("%H:%M:%S "); @start[tid] = nsecs;
                     printf("comm: %s, pid: %d, tid: %d, __syscall_nr: %d, dfd: %d, flags: %d, mode: %d \n============== filename: %s =======\n", 
                             comm, pid, tid, args->__syscall_nr, args->dfd, args->flags, args->mode, str(args->filename)); }
                   tracepoint:syscalls:sys_exit_openat  /strncmp("oracle", comm, 6) == 0 && @start[tid] / {
                     time("%H:%M:%S "); @fd = (uint64)args->ret; $duration_us = ( nsecs - @start[tid] ) / 1000; 
                     printf("===================== File Descriptor: %d, Duration (us): %d ===================== \n", @fd, $duration_us); 
                     printf("CallStack: %s\n", ustack(30));
                     delete(@start[tid]); }
                   tracepoint:syscalls:sys_enter_write /strncmp("oracle", comm, 6) == 0 && @fd == args->fd / {
                     time("%H:%M:%S "); $length = args->count; $line = str(args->buf);
                     printf("[pid: %d, fd: %d, length: %3d] %s \n", pid, args->fd, $length, $line); }'
      
      13:12:01 comm: oracle_1086521_, pid: 1086521, tid: 1086521, __syscall_nr: 257, dfd: -100, flags: 194, mode: 416
      ============== filename: /home/oracle/suk/slow_dir3/adump/db3audit_ora_1086521_20251228131201036719868850.aud =======
      13:12:06 ===================== File Descriptor: 8, Duration (us): 5089844 =====================
      CallStack:
              __open+79
              szaudinit+506
              szaud+1416
              szaud_mkrec+1774
              audios+2936
              audins+13805
              audlon0+1983
              audlon+526
              auddft_internal+2853
              audStatement+1168
              kpolnb+4729
              kpoauth+1917
              opiodr+1264
              ttcpip+1219
              opitsk+1925
              opiino+939
              opiodr+1264
              opidrv+1094
              sou2o+165
              opimai_real+422
              ssthrdmain+417
              main+256
              __libc_start_main+243
              0x5541f689495641d7
              
      13:12:06 [pid: 1086521, fd: 8, length:  11] Audit file
      13:12:06 [pid: 1086521, fd: 8, length:  84] /home/oracle/suk/slow_dir3/adump/db3audit_ora_1086521_20251228131201036719868850.aud
      13:12:06 [pid: 1086521, fd: 8, length:  90] Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
      Version 19.27.0.0.0
      ......
      13:12:06 [pid: 1086521, fd: 8, length: 406] SESSIONID:[9] "102148556" ENTRYID:[1] "1" STATEMENT:[1] "1" USERID:[1] "K" USERHOST:[16] "SYS\MACHINE12345" TERMINAL:[12] "MACH
      
      In above test, Audit Parameter: audit_syslog_level is not set, and audit files (and audit info) are created under directory audit_file_dest.
      If we set "audit_syslog_level='LOCAL0.INFO'" and re-start DB, audit info is written into /var/log/messages (see /etc/rsyslog.conf).
      
      alter system set audit_syslog_level='LOCAL0.INFO' scope=spfile;
      --DB re-start
      
      -- reset by alter system reset audit_syslog_level scope=spfile;
      
      Blog: Disk file operations I/O (July 17, 2024 Dinesh Bandelkar) shows one case of "Disk file operations I/O" with (FileOperation: 8, fileno: 0, filetype: 8) when Oracle Redaction is OS audited.