PeopleSoft App Server Freeze Troubleshooting Checklist

1. Identify stuck processes

ps -ef | grep PSAPPSRV

Better:

ps -eo pid,ppid,state,wchan,etime,cmd | grep PSAPPSRV

Important things:

• STATE

  • D = uninterruptible I/O wait
  • S = sleeping
  • R = running

• WCHAN

  • shows kernel wait point

Examples:

• futex_wait
• io_schedule
• pipe_wait
• do_poll

2. Check overall system pressure

CPU / load

top -H

Look for:

• high load average with low CPU usage
• blocked threads

VM / blocked tasks

vmstat 1

Important columns:

• b

  • blocked processes

• wa

  • I/O wait

Disk I/O

iostat -xz 1

Important:

• await
• %util
• queue depth

High await with low CPU = storage problem.

3. Attach to stuck process

strace

strace -tt -p <PID>

Look for repeated:

• futex()
• poll()
• recv()
• read()
• write()
• semop()
• msgrcv()

Examples:

Mutex wait

futex(... FUTEX_WAIT ...)

Socket/network wait

poll(...)
recvfrom(...)

File lock wait

fcntl(... F_SETLKW ...)

4. Get process stack

gstack

gstack <PID>

or:

pstack <PID>

This is often the most valuable command.

Look for:

• Oracle OCI calls
• Tuxedo calls
• semaphores
• socket waits
• file I/O
• mutex/futex waits

5. Check open files/resources

lsof

lsof -p <PID>

Look for:

• shared logs
• deleted files
• NFS mounts
• pipes
• sockets
• temp files

6. Check Tuxedo IPC resources

IPC inventory

ipcs -a

Look for:

• semaphore exhaustion
• huge shared memory
• large message queues

Tuxedo status

tmadmin

Useful commands:

psc
pq
printserver

Look for:

• servers stuck BUSY
• queue buildup
• same request forever
• no idle handlers

7. Check process wait channels

Very useful:

ps -eo pid,state,wchan:32,cmd | grep PSAPPSRV

Examples:

• futex_wait_queue_me
• io_schedule
• pipe_wait
• do_select

This often immediately identifies the subsystem.

8. Check kernel logs

dmesg -T | tail -100

Look for:

• storage resets
• NFS issues
• filesystem hangs
• OOM
• network errors

9. Capture before killing process

IMPORTANT: Always gather:

• strace
• gstack
• ps
• lsof

BEFORE killing the stuck process.

Killing clears the evidence.

10. Most valuable quick combo

When incident happens:

ps -eo pid,state,wchan:32,cmd | grep PSAPPSRV

strace -tt -p <PID>

gstack <PID>

lsof -p <PID>

vmstat 1

iostat -xz 1

Those six commands will direct you to the blockage.

You could also turn up the logging, log fence, peoplecode and sql trace. That would also likely get you to the problem.