Joshua Zhu’s Blog

For a long time, I was not satisfied with vmstat, because it does not generate timestamps. Then I came across a cool program named Dstat yesterday, which can be an excellent replacement for vmstat.

A screenshot of Dstat

By using Dstat, now not only all my needs can be met but also it is pretty easy to do resource usage analysis and graphing.

As the author described, “Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.” So why use three or more tools when one tool can give you everything you need?

Here is an example showing how to use it.

First, capture the resource usage information (CPU and memory) to a file, e.g. stat.dat:

$ dstat -tcmn > stat.dat

Then use the scripts below to create CPU and memory usage graphs:

#!/usr/bin/gnuplot
 
set terminal png
set output "cpu.png"
set title "CPU usage"
set xlabel "time"
set ylabel "percent"
set xdata time
set timefmt "%d-%m %H:%M:%S"
set format x "%H:%M"
plot "stat.dat" using 1:4 title "system" with lines, 
"stat.dat" using 1:3 title "user" with lines, 
"stat.dat" using 1:5 title "idle" with lines

#!/usr/bin/gnuplot
 
set terminal png
set output "memory.png"
set title "memory usage"
set xlabel "time"
set ylabel "size(M Bytes)"
set xdata time
set timefmt "%d-%m %H:%M:%S"
set format x "%H:%M"
plot "stat.dat" using 1:9 title "used" with lines, 
"stat.dat" using 1:10 title "buff" with lines, 
"stat.dat" using 1:11 title "cach" with lines, 
"stat.dat" using 1:12 title "free" with lines

Resource usage graph examples:

Download the scripts.
http://www.zhuzhaoyuan.com/download/dstat/cpu.sh
http://www.zhuzhaoyuan.com/download/dstat/memory.sh

I’m surprised by the fact that so many network programmers don’t know TIME_WAIT and CLOSE_WAIT well, particularly those who use Java, C#, Python and etc.

It is true, however, TIME_WAIT and CLOSE_WAIT are the most confusing two among TCP’s 11 states (CLOSED, LISTEN, SYN_SENT, SYN_RECV, ESTABLISHED, CLOSE_WAIT, LAST_ACK, FIN_WAIT1, FIN_WAIT2, CLOSING, and TIME_WAIT), which are displayed by netstat(1).

Before going into detail, let me explain the terms first:
Active Open: An end sends a SYN segment to the other end by calling connect(2). This end is usually called a client.
Passive Open: An end issues a passive open by calling socket(2), bind(2), and listen(2) so that it can accept(2) the clients’ connections. This end is usually called a server.
Active Close: An end performs active close when it calls close(2) first. It results a FIN segment being sent.
Passive Close: The other end receives the FIN segment performs the passive close.
MSL: Maximum Segment Lifetime, the maximum time a segment can live in the network before being discarded.

Now take a look at the TCP state transition diagram.

TCP State Transition Diagram 1

TCP State Transition Diagram 2

In the diagrams above, I denote the active open/close transitions with a green line and the passive open/close transitions with a blue line (Normal transitions are in bold). The end that performs the active close goes through the TIME_WAIT state; while the end performs the passive close enters the CLOSE_WAIT state. Note that either end (the server or the client) can perform the active close!

What’s the TIME_WAIT state for?
1) When the final ACK being sent by the end that performed the active close was lost, the other end would resend a FIN. TCP maintains the information of the connection, hence it would resend the lost ACK rather than respond with an RST (interpreted as error).
2) To allow wandering duplicates to expire in the network, so that the duplicate segments would not corrupt later connections.

The value of TIME_WAIT delay is 2MSL, which is TCP implementation dependent and is generally about 1-4 minutes.

CLOSE_WAIT happens when an end has received a FIN segment from the other end, to wait for the program to close the socket. Long duration of CLOSE_WAIT suggests that there might be a bug in your program. In fact, a TCP connection can stay in the CLOSE_WAIT state forever unless it is explicitly closed. It is the application’s responsibility to close its socket after use, to release the resource of the connection.

Conclusion:
In most cases, there is no need to worry about TIME_WAIT. It is a safety feature of TCP. But you really should take care of CLOSE_WAIT. Also note, if a connection is in CLOSE_WAIT, it would never end up in the TIME_WAIT state, and vice versa. BTW, if you haven’t read Richard Stevens’ Unix Network Programming, go out and buy one. It is a must-have for all network programmers, even if you do Windows programming.

Download the dot files and msc file here:
http://www.zhuzhaoyuan.com/download/tcp/tcp-state1.dot
http://www.zhuzhaoyuan.com/download/tcp/tcp-state2.dot
http://www.zhuzhaoyuan.com/download/tcp/tcp_close.msc