IAX is the Inter-Asterisk eXchange protocol native to Asterisk PBX and supported by a number of other softswitches and PBXs. It is used to enable VoIP connections between servers as well as client-server communication.
IAX now most commonly refers to IAX2, the second version of the IAX protocol. The original IAX protocol has been deprecated almost universally in favor of IAX2.
IAX2 is a VoIP protocol that usually carries both signalling and data on the same path. The commands and parameters are sent binary and any extension has to have a new numeric code allocated. Historically this was modeled after the internal data passing of Asterisk modules.
IAX2 uses a single UDP data stream (usually on port 4569) to communicate between endpoints, both for signaling and data. The voice traffic is transmitted in-band, making IAX2 easier to firewall and more likely to work behind network address translation. This is in contrast to SIP, H.323 and Media Gateway Control Protocol which are using an out-of-band RTP stream to deliver information.
IAX2 supports trunking, multiplexing channels over a single link. When trunking, data from multiple calls are merged into a single set of packets, meaning that one IP datagram can deliver information for more than one call, reducing the effective IP overhead without creating additional latency. This is a big advantage for VoIP users, where IP headers are large percentage of the bandwidth usage.