-
January 2026 (1)
-
December 2025 (1)
-
September 2025 (3)
-
August 2025 (1)
-
July 2025 (3)
-
June 2025 (1)
-
May 2025 (1)
-
February 2025 (1)
-
November 2024 (1)
-
October 2024 (1)
-
September 2024 (1)
-
April 2024 (3)
-
January 2024 (1)
-
October 2023 (1)
-
September 2023 (3)
-
August 2023 (1)
-
June 2023 (1)
-
April 2023 (3)
-
March 2023 (2)
-
February 2023 (1)
-
January 2023 (1)
-
December 2022 (2)
-
October 2022 (2)
-
September 2022 (2)
-
August 2022 (2)
-
July 2022 (1)
-
June 2022 (1)
-
May 2022 (2)
-
April 2022 (2)
-
March 2022 (1)
-
February 2022 (2)
-
January 2022 (1)
-
December 2021 (1)
-
November 2021 (1)
-
October 2021 (2)
-
July 2021 (1)
-
June 2021 (1)
-
May 2021 (1)
-
April 2021 (3)
-
March 2021 (2)
-
January 2021 (1)
-
November 2020 (3)
-
September 2020 (1)
-
August 2020 (1)
-
May 2020 (3)
-
April 2020 (3)
-
February 2020 (2)
-
January 2020 (1)
-
December 2019 (2)
-
August 2019 (2)
-
April 2019 (1)
-
November 2018 (5)
- Oracle row cache objects Event: 10222, Dtrace Script (I)
- Row Cache Objects, Row Cache Latch on Object Type: Plsql vs Java Call (Part-1) (II)
- Row Cache Objects, Row Cache Latch on Object Type: Plsql vs Java Call (Part-2) (III)
- Row Cache and Sql Executions (IV)
- Latch: row cache objects Contentions and Scalability (V)
-
October 2018 (2)
-
July 2018 (3)
-
April 2018 (1)
-
March 2018 (2)
-
February 2018 (1)
-
January 2018 (4)
-
October 2017 (2)
-
September 2017 (2)
-
July 2017 (3)
-
May 2017 (8)
- JDBC, Oracle object/collection, dbms_pickler, NOPARALLEL sys.type$ query
- PLSQL Context Switch Functions and Cost
- Oracle Datetime (1) - Concepts
- Oracle Datetime (2) - Examples
- Oracle Datetime (3) - Assignments
- Oracle Datetime (4) - Comparisons
- Oracle Datetime (5) - SQL Arithmetic
- Oracle Datetime (6) - PLSQL Arithmetic
-
March 2017 (3)
-
February 2017 (1)
-
January 2017 (1)
-
November 2016 (1)
-
September 2016 (2)
-
August 2016 (1)
-
June 2016 (1)
-
May 2016 (1)
-
April 2016 (1)
-
February 2016 (1)
-
January 2016 (3)
-
December 2015 (1)
-
November 2015 (1)
-
September 2015 (2)
-
August 2015 (1)
-
July 2015 (2)
-
June 2015 (1)
-
April 2015 (2)
-
January 2015 (1)
-
December 2014 (1)
-
November 2014 (2)
-
May 2014 (3)
-
March 2014 (2)
-
November 2013 (3)
-
September 2013 (1)
-
June 2013 (2)
-
April 2013 (2)
-
March 2013 (3)
-
December 2012 (1)
-
November 2012 (2)
-
July 2012 (1)
-
May 2012 (1)
-
April 2012 (1)
-
February 2012 (1)
-
November 2011 (2)
-
July 2011 (1)
-
May 2011 (3)
-
April 2011 (1)
Friday, January 2, 2026
"Disk file operations I/O" with (FileOperation: 8, fileno: 0, filetype: 8) Test
In this blog, we will make a test to demonstrate Oracle Wait Event "Disk file operations I/O" with
(P1/FileOperation: 8, P2/fileno: 0, P3/filetype: 8).
Note: Tested in Oracle 19.27
The database auditing is configured as follows:
(See Blog: Oracle Database Stuck on DB Writer Wait Event: "oradebug request completion" ).
We start system-wide 10046 trace by:
Find TNS listener process and strace it:
Now we open a Sqlplus session by
So we can say that in this case, "FileOperation=8" is file open, and "filetype=8" is OS Audit file.
After test, stop 10046 trace by (otherwise too many trace files):
We can see that Sqlplus Process 1072963 was spawned by TNS Process 728055:
and it takes 5.098572 seconds from 10:11:56 to 10:12:01.
Look OS audit file, its name is composed by DB SID: db3audit, Unix process pid: 1072963, and creating timestamp: 20251223101156, which is 2025-12-23 10:11:56.
In the OS audit file, the first timestamp "10:12:01" is recorded for session LOGON, which has about 5 seconds delay (from 10:11:56 to 10:12:01),
and the second timestamp "10:28:44" is for session LOGOFF. Here the full audit file:
TOP Wait Event "Disk file operations I/O" in AWR and ASH looks like:
Start following bpftrace script and then open a Sqlplus session.
The script output shows file open (sys_enter_openat) "Duration (us): 5089844", and its Oracle CallStack.
If we set "audit_syslog_level='LOCAL0.INFO'" and re-start DB, audit info is written into /var/log/messages (see /etc/rsyslog.conf).
(P1/FileOperation: 8, P2/fileno: 0, P3/filetype: 8).
Note: Tested in Oracle 19.27
1. Test Setup
The database auditing is configured as follows:
NAME TYPE VALUE
------------------------- ----------- --------------------------------
audit_file_dest string /home/oracle/suk/slow_dir3/adump
audit_sys_operations boolean TRUE
audit_trail string OS
audit_file_dest is mounted on a slow file system with 5000 ms read/write delay.(See Blog: Oracle Database Stuck on DB Writer Wait Event: "oradebug request completion" ).
2. Wait Event Tracing
2.1 10046 Trace
We start system-wide 10046 trace by:
alter system set events '10046 trace name context forever, level 12';
2.2 strace
Find TNS listener process and strace it:
oracle@db3audit:db3audit > ps -efl |grep tns
0 S oracle 728055 1 0 80 0 - 40462 do_epo Sep27 ? 00:00:15 /orabin/app/oracle/product/19.27.0.0./bin/tnslsnr LISTENER_oradb3audit
oracle@db3audit:db3audit > strace -ttTfy -p 728055 -o strace_output_2.txt
2.3 Sqlplus Session
Now we open a Sqlplus session by
SQL > sqlplus k/k@db3audit
2.4 10046 Trace Output
Trace file /orabin/app/oracle/admin/db3audit/diag/rdbms/db3audit/db3audit/trace/db3audit_ora_1072963.trc
Build label: RDBMS_19.27.0.0.0DBRU_LINUX.X64_250331
Oracle process number: 57
Unix process pid: 1072963, image: oracle@db3audit
*** MODULE NAME:(sqlplus.exe) 2025-12-23T10:11:56.689481+01:00
*** 2025-12-23T10:12:01.847101+01:00
WAIT #0: nam='Disk file operations I/O' ela= 5100597 FileOperation=8 fileno=0 filetype=8 obj#=-1 tim=18667193313139
It shows "ela= 5100597" microseconds with "FileOperation=8 fileno=0 filetype=8".So we can say that in this case, "FileOperation=8" is file open, and "filetype=8" is OS Audit file.
After test, stop 10046 trace by (otherwise too many trace files):
alter system set events '10046 trace name context off';
2.5 strace Output
We can see that Sqlplus Process 1072963 was spawned by TNS Process 728055:
oracle@db3audit:db3audit > strace -ttTfy -p 728055 -o strace_output_3.txt
./strace: Process 728055 attached
./strace: Process 1072962 attached
./strace: Process 1072963 attached
Open strace_output_2.txt, it shows:
1072963 10:11:56.746551 openat(AT_FDCWD, "/home/oracle/suk/slow_dir3/adump/db3audit_ora_1072963_20251223101156746537740094.aud", O_RDWR|O_CREAT|O_EXCL, 0640) = 10
<5.098572>
1072963 10:12:01.845204 write(10, "Audit file ", 11) = 11 <0.000058>
1072963 10:12:01.845315 write(10, "/home/oracle/suk/slow_dir3/adump"..., 84) = 84 <0.000023>
1072963 10:12:01.845372 write(10, "\n", 1) = 1 <0.000022>
1072963 10:12:01.845433 write(10, "Oracle Database 19c Enterprise E"..., 90) = 90 <0.000021>
1072963 10:12:01.845482 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.845533 write(10, "Build label: RDBMS_19.27.0.0."..., 54) = 54 <0.000020>
1072963 10:12:01.845582 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.845639 write(10, "ORACLE_HOME: /orabin/app/orac"..., 64) = 64 <0.000020>
1072963 10:12:01.845689 write(10, "System name:\tLinux\n", 19) = 19 <0.000020>
1072963 10:12:01.845738 write(10, "Node name:\tdb3audit\n", 20) = 20 <0.000021>
1072963 10:12:01.845796 write(10, "Release:\t4.18.0-372.9.1.el8.x86_"..., 35) = 35 <0.000020>
1072963 10:12:01.845845 write(10, "Version:\t#1 SMP Fri Apr 15 22:12"..., 45) = 45 <0.000021>
1072963 10:12:01.845895 write(10, "Machine:\tx86_64\n", 16) = 16 <0.000021>
1072963 10:12:01.845953 write(10, "CLID:\tP\n", 8) = 8 <0.000021>
1072963 10:12:01.846010 write(10, "Instance name: db3audit\n", 24) = 24 <0.000062>
1072963 10:12:01.846105 write(10, "Instance number: 1\n", 19) = 19 <0.000020>
1072963 10:12:01.846161 write(10, "Database name: db3audit ", 24) = 24 <0.000023>
1072963 10:12:01.846215 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846264 write(10, "Database unique name: db3audit ", 31) = 31 <0.000020>
1072963 10:12:01.846313 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846388 write(10, "Database id: 1059538969\n", 24) = 24 <0.000021>
1072963 10:12:01.846442 write(10, "Database role: PRIMARY\n", 23) = 23 <0.000020>
1072963 10:12:01.846491 write(10, "Redo thread mounted by this inst"..., 40) = 40 <0.000020>
1072963 10:12:01.846541 write(10, "Oracle process number: 57\n", 26) = 26 <0.000021>
1072963 10:12:01.846592 write(10, "Unix process pid: 1072963, image"..., 49) = 49 <0.000019>
1072963 10:12:01.846640 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846689 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846762 write(10, "Tue Dec 23 10:12:01 2025 +01:00\n", 32) = 32 <0.000021>
1072963 10:12:01.846812 write(10, "LENGTH: \"408\"", 13) = 13 <0.000021>
1072963 10:12:01.846860 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.846909 write(10, "SESSIONID:[9] \"102147891\" ENTRYI"..., 406) = 406 <0.000030>
1072963 10:12:01.846973 write(10, "\n", 1) = 1 <0.000020>
1072963 10:12:01.847023 write(10, "\n", 1) = 1 <0.000021>
The above output shows that openat "/home/oracle/suk/slow_dir3/adump/db3audit_ora_1072963_20251223101156746537740094.aud" returns FD = 10,and it takes 5.098572 seconds from 10:11:56 to 10:12:01.
Look OS audit file, its name is composed by DB SID: db3audit, Unix process pid: 1072963, and creating timestamp: 20251223101156, which is 2025-12-23 10:11:56.
In the OS audit file, the first timestamp "10:12:01" is recorded for session LOGON, which has about 5 seconds delay (from 10:11:56 to 10:12:01),
and the second timestamp "10:28:44" is for session LOGOFF. Here the full audit file:
oracle@db3audit:db3audit > cat /home/oracle/suk/slow_dir3/adump/db3audit_ora_1072963_20251223101156746537740094.aud
Audit file /home/oracle/suk/slow_dir3/adump/db3audit_ora_1072963_20251223101156746537740094.aud
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
Build label: RDBMS_19.27.0.0.0DBRU_LINUX.X64_250331
ORACLE_HOME: /orabin/app/oracle/product/19.27.0.0.
System name: Linux
Node name: db3audit
Release: 4.18.0-372.9.1.el8.x86_64
Version: #1 SMP Fri Apr 15 22:12:19 EDT 2022
Machine: x86_64
CLID: P
Instance name: db3audit
Instance number: 1
Database name: db3audit
Database unique name: db3audit
Database id: 1059538969
Database role: PRIMARY
Redo thread mounted by this instance: 1
Oracle process number: 57
Unix process pid: 1072963, image: oracle@db3audit
Tue Dec 23 10:12:01 2025 +01:00
LENGTH: "408"
SESSIONID:[9] "102147891" ENTRYID:[1] "1" STATEMENT:[1] "1" USERID:[1] "K" USERHOST:[16] "SYS\MACHINE12345" TERMINAL:[12] "MACHINE12345" ACTION:[3] "100" RETURNCODE:[1] "0" COMMENT$TEXT:[126] "Authenticated by: DATABASE;AUTHENTICATED IDENTITY: K; Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.44.105.100)(PORT=50160))" OS$USERID:[7] "KS00383" DBID:[10] "1059538969" PRIV$USED:[1] "5" CURRENT_USER:[1] "K"
Thu Dec 23 10:28:44 2025 +01:00
LENGTH: "231"
SESSIONID:[9] "102147891" ENTRYID:[1] "1" USERID:[1] "K" ACTION:[3] "101" RETURNCODE:[1] "0" LOGOFF$PREAD:[5] "35037" LOGOFF$LREAD:[6] "166495" LOGOFF$LWRITE:[1] "4" LOGOFF$DEAD:[1] "0" DBID:[10] "1059538969" SESSIONCPU:[3] "258"
2.6 v$active_session_history and Oracle Performance Views
select sample_time, program, event, p1, p2, p3, p1text, p2text, p3text
from v$active_session_history t
where event = 'Disk file operations I/O' and p1=8 and p2=0 and p3=8
and sample_time >= timestamp '2025-12-28 10:31:22.549'
order by t.sample_time;
SAMPLE_TIME PROGRAM EVENT P1 P2 P3 P1TEXT P2TEXT P3TEXT
------------------- ------------ ------------------------ --- --- --- --------------- ------- --------
2025-12-22 10:11:57 sqlplus.exe Disk file operations I/O 8 0 8 FileOperation fileno filetype
2025-12-22 10:11:58 sqlplus.exe Disk file operations I/O 8 0 8 FileOperation fileno filetype
2025-12-22 10:11:59 sqlplus.exe Disk file operations I/O 8 0 8 FileOperation fileno filetype
2025-12-22 10:12:00 sqlplus.exe Disk file operations I/O 8 0 8 FileOperation fileno filetype
2025-12-22 10:12:01 sqlplus.exe Disk file operations I/O 8 0 8 FileOperation fileno filetype
select filetype_id, filetype_name, function_id, function_name, sum(small_read_reqs), sum(small_write_reqs), sum(number_of_waits), sum(wait_time)
from v$iostat_function_detail
where filetype_id = 8
group by filetype_id, filetype_name, function_id, function_name
order by filetype_id, function_id;
FILETYPE_ID FILETYPE FUNCTION_ID FUNCTION_NAME SUM(SMALL_READ_REQS) SUM(SMALL_WRITE_REQS) SUM(NUMBER_OF_WAITS) SUM(WAIT_TIME)
----------- -------- ----------- -------------------- -------------------- --------------------- -------------------- --------------
8 Other 2 LGWR 0 0 136398 50459
8 Other 7 Recovery 0 0 1 0
8 Other 8 Buffer Cache Reads 0 0 245 283
8 Other 13 Others 0 0 273272 120046
-- also in v$session_event, dba_hist_system_event
select event, total_waits, time_waited, time_waited_micro, event_id, wait_class_id, wait_class#, wait_class from v$system_event
where event = 'Disk file operations I/O';
EVENT TOTAL_WAITS TIME_WAITED TIME_WAITED_MICRO EVENT_ID WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS
------------------------- ----------- ----------- ----------------- ---------- ------------- ----------- ----------
Disk file operations I/O 67226 28853 288534923 166678035 1740759767 8 User I/O
-- no rows in v$iostat_file
select * from v$iostat_file where filetype_id = 8;
no rows selected
2.7 AWR and ASH
TOP Wait Event "Disk file operations I/O" in AWR and ASH looks like:
Top 10 Foreground Events by Total Wait Time
| Event | Waits | Total Wait Time (sec) | Avg Wait | % DB time | Wait Class |
|---|---|---|---|---|---|
| Disk file operations I/O | 2,706 | 11.4 | 4.20ms | 35.4 | User I/O |
Top Event P1/P2/P3 Values
| Event | % Event | P1, P2, P3 Values | % Activity | Parameter 1 | Parameter 2 | Parameter 3 |
|---|---|---|---|---|---|---|
| Disk file operations I/O | 45.45 | "8","0","8" | 45.45 | FileOperation | fileno | filetype |
3. bpftrace
Start following bpftrace script and then open a Sqlplus session.
The script output shows file open (sys_enter_openat) "Duration (us): 5089844", and its Oracle CallStack.
BPFTRACE_STRLEN=128 bpftrace -e 'tracepoint:syscalls:sys_enter_openat /strncmp("oracle", comm, 6) == 0 && strncmp("/home/oracle/suk/slow_dir3", str(args->filename), 26) == 0 / {
time("%H:%M:%S "); @start[tid] = nsecs;
printf("comm: %s, pid: %d, tid: %d, __syscall_nr: %d, dfd: %d, flags: %d, mode: %d \n============== filename: %s =======\n",
comm, pid, tid, args->__syscall_nr, args->dfd, args->flags, args->mode, str(args->filename)); }
tracepoint:syscalls:sys_exit_openat /strncmp("oracle", comm, 6) == 0 && @start[tid] / {
time("%H:%M:%S "); @fd = (uint64)args->ret; $duration_us = ( nsecs - @start[tid] ) / 1000;
printf("===================== File Descriptor: %d, Duration (us): %d ===================== \n", @fd, $duration_us);
printf("CallStack: %s\n", ustack(30));
delete(@start[tid]); }
tracepoint:syscalls:sys_enter_write /strncmp("oracle", comm, 6) == 0 && @fd == args->fd / {
time("%H:%M:%S "); $length = args->count; $line = str(args->buf);
printf("[pid: %d, fd: %d, length: %3d] %s \n", pid, args->fd, $length, $line); }'
13:12:01 comm: oracle_1086521_, pid: 1086521, tid: 1086521, __syscall_nr: 257, dfd: -100, flags: 194, mode: 416
============== filename: /home/oracle/suk/slow_dir3/adump/db3audit_ora_1086521_20251228131201036719868850.aud =======
13:12:06 ===================== File Descriptor: 8, Duration (us): 5089844 =====================
CallStack:
__open+79
szaudinit+506
szaud+1416
szaud_mkrec+1774
audios+2936
audins+13805
audlon0+1983
audlon+526
auddft_internal+2853
audStatement+1168
kpolnb+4729
kpoauth+1917
opiodr+1264
ttcpip+1219
opitsk+1925
opiino+939
opiodr+1264
opidrv+1094
sou2o+165
opimai_real+422
ssthrdmain+417
main+256
__libc_start_main+243
0x5541f689495641d7
13:12:06 [pid: 1086521, fd: 8, length: 11] Audit file
13:12:06 [pid: 1086521, fd: 8, length: 84] /home/oracle/suk/slow_dir3/adump/db3audit_ora_1086521_20251228131201036719868850.aud
13:12:06 [pid: 1086521, fd: 8, length: 90] Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
......
13:12:06 [pid: 1086521, fd: 8, length: 406] SESSIONID:[9] "102148556" ENTRYID:[1] "1" STATEMENT:[1] "1" USERID:[1] "K" USERHOST:[16] "SYS\MACHINE12345" TERMINAL:[12] "MACH
In above test, Audit Parameter: audit_syslog_level is not set, and audit files (and audit info) are created under directory audit_file_dest.If we set "audit_syslog_level='LOCAL0.INFO'" and re-start DB, audit info is written into /var/log/messages (see /etc/rsyslog.conf).
alter system set audit_syslog_level='LOCAL0.INFO' scope=spfile;
--DB re-start
-- reset by alter system reset audit_syslog_level scope=spfile;
Blog:
Disk file operations I/O (July 17, 2024 Dinesh Bandelkar)
shows one case of "Disk file operations I/O" with (FileOperation: 8, fileno: 0, filetype: 8) when Oracle Redaction is OS audited.
Subscribe to:
Comments (Atom)