Applications that handle a huge number of TCP connections, either as a server, or as a client, will often see a large number of sockets in TIME_WAIT state. The sockets in TIME_WAIT can be seen with the 'netstat -t' command. If the number of TIME_WAIT sockets gets too large, your address space will be exhausted, causing a disruption of TCP traffic. This page describes the TIME_WAIT issue in a Linux context. For a generic source of information, see this FAQ entry on UNIX socket programming.
Only one of the two partners communicating with TCP will get the TIME_WAIT status. This happens to the side that initiated the closing of the connection.
GNU/Linux seems to be lacking a proper configuration option for the TIME_WAIT status (like BSD offers). For linux, it is hard-coded to 60 seconds in the kernel in include/net/tcp.h as:
#define TCP_TIMEWAIT_LEN (60*HZ)
Do not be confused by the /proc/sys/net/ipv4/tcp_fin_timeout config item. The FIN TIMEOUT is not the same as the TIMEWAIT length.
Also, I find the /proc/sys/net/ipv4/tcp_max_tw_buckets confusing. Although the default is set at 180000, I see a TCP disruption when I have 32K TIME_WAIT sockets on my system, regardless of the max tw buckets.
You can force the kernel to reuse the TIME_WAIT buckets though, by setting the /proc/sys/net/ipv4/tcp_tw_recycle entry to 1. I have tested this, and it works: very few TIME_WAIT entries will be present when using this setting. It could be however, that using this could affect the reliability of TCP connections. I believe problems may arrise in NAT environments.
An application like a client of the linuxtuples system, easily exhausts the available socket address space. If this happens, I see a 'connect failed: Cannot assign requested address'.