IP packet tracing is a vital part of troubleshooting problems in TCP/IP networks. You can use IBM’s PKTTRACE utility to copy IP packets as they enter or leave a z/OS TCP/IP stack. It works well, provided that:
- You correctly execute each of the steps to start, stop, then view the trace.
- You know how to write a long, complex PKTTRACE command to precisely select the few packets you need from the thousands you might get.
- The problem involves one TCP/IP stack, and one stack only.
- No one else is using PKTTRACE at the same time.
Let me explain.
Assuming you are using the native IBM z/OS Comm Server PKTTRACE with no additional software, you must perform many steps to gather the trace data. Perhaps you have personally experienced the frustration of starting a trace, stopping and formatting it to verify the correct packets are being traced, realizing the PKTTRACE command needs to be changed, correcting the PKTTRACE command, restarting the trace, and then stopping and formatting it again for another verification check. That is a lot of effort just to set the stage to perform what should be a relatively simple troubleshooting task.
The PKTTRACE command by itself has no context and allows many parameters to be coded. It is the operator’s task to determine the correct set of parameters and values to specify. A simple mistake can produce undesired results such as the wrong, too many, or no packets being traced. The operator is frustrated in knowing he has to determine the mistake then resubmit the trace.
PKTTRACE is stack oriented. It does not have any concept of multiple stacks or sysplex environments. What does a user do when he needs to gather packet traces on traffic that can traverse multiple stacks which can run on more than one system?
And to make matters more complicated, did you know that the PKTTRACE utility was not designed to support multiple users working independently? A PKTTRACE command entered by one user can impact other active PKTTRACEs.
For example, using VARY PKTTRACE, Colin starts a trace for TCP traffic on port 21, and requests 1500 bytes from each packet:
V TCPIP,,PKTTRACE,PROT=TCP,PORTNUM=21,ABBREV=1500
Later, Deb starts a trace for all TCP traffic, but requests only the first 50 bytes of payload:
V TCPIP,,PKTTRACE,PROT=TCP,ABBREV=50
And later on before Colin stops his trace, Deb then stops her trace:
V TCPIP,,PKTTRACE,PROT=TCP,OFF
Finally, the problem Colin needs to trace occurs and he stops his trace. When Colin examines his trace, he notices that it suddenly started tracing only 50 bytes at the time that Deb started her trace and the trace stopped before the event he needed trace occurred! Why? The last PKTTRACE command entered becomes the first filter the stack looks at when matching new IP packets. Because Deb’s trace was started after Colin’s, the set of filters on Deb’s PKTTRACE command was the first to match the criteria for both her packets and Colin’s, so its ABBREV value (100 bytes) was the one used by the stack to determine how much to trace. And because Colin’s trace filter was a “subset” of Deb’s, the stack removed both filters when she stopped her trace.
What can Colin do to prevent future headaches using PKTTRACE? He could try coordinating with Deb and anyone else who might start a packet trace. Not an exciting proposition especially if Colin is in a hurry to gather a trace. He could issue a NETSTAT command to see if any packet traces are active. Once again, more effort and work on Colin’s part.
Colin needs a solution which addresses the PKTTRACE shortcomings.
- Make the process of starting and viewing traces simple with few steps as possible.
- Provide context and fill in the blanks yet allow the operator to override parameters if necessary.
- Allow the operator to start and view traces simultaneously on multiple stacks on multiple systems.
- Coordinate multiple active PKTTRACE commands so that each user sees the expected trace data. I.e. Colin needs a software package which addresses each of these shortcomings.
In the next newsletter, I will provide information on a solution which addresses these shortcomings. That solution is the VIP monitor.
Dennis Mruz, tech support engineer, SDS, Jan. 2012
Free Demo/Trial
We offer individualized product demonstrations by request. Your organization can also try SDS Software on your system for 30 days, free of charge.
Contact SDS
Phone: (800) 443-6183
Phone: (763) 571-9000
200 Village Center Drive, Suite 250
North Oaks, MN 55127
USA