traceroute is a tool that cleverly combines two characteristics of the protocols that make Internet possible. These are:
TTL or packet expiration
In order to protect the Internet from the effect of packets trapped in a routing loop, TCP/IP's designers added a counter to each datagram. This counter is called TTL, short for Time To Live. This is a number that limits how many hops a datagram can go through before being discarded by the network.
Whitout this protection, a packet sent to an unexistant destination would be trapped jumping between two routers somewhere in the Internet, wasting CPU and bandwidth forever.
When an IP datagram in injected into the network, the TTL field is populated with the maximum number of hops that define the lifetime of that datagram. Each router through which the datagram passes, decrements this counter. When this counter reaches zero, the datagram is discarded.
ICMP or Internet Control Message Protocol
Another hit from the Internet's designers, was the incorporation of a protocol called ICMP, which handles control messages. These are administrative messages that are sent among Internet nodes.
ICMP packets serve a number of purposes: Note the congestion of a network device or link, the choice of a sub-optimal routing path, not being able to reach a given site, etc. Uno of these messages is specially useful for traceroute: Time to live exceeded.
By combining these two tools, traceroute allows for the construction of a network map as seen from a particular node. Below is an example of the traceroute command from my ADSL line at home.
$ traceroute www.yahoo.com traceroute to www.yahoo.akadns.net (64.58.76.178), 30 hops max, 40 byte packets 1 dyn-0.rbe.lac00-nrp6.cha.dsl.cantv.net (200.84.20.1) 75.447 ms 32.067 ms 67.297 ms 2 fe1-1-0.core-00.cha.dsl.cantv.net (172.16.11.1) 36.428 ms 56.477 ms 76.693 ms 3 172.16.68.1 (172.16.68.1) 54.544 ms 41.076 ms 77.019 ms 4 172.16.67.1 (172.16.67.1) 91.147 ms 45.139 ms 60.265 ms 5 200.44.43.216 (200.44.43.216) 246.248 ms 317.591 ms 208.779 ms 6 so3-2-0.miabb2.miami.opentransit.net (193.251.252.81) 166.349 ms 145.007 ms 146.57 ms 7 so0-3-0.miacr2.miami.opentransit.net (193.251.242.221) 175.159 ms 160.761 ms 152.51 ms 8 so2-0-0.atlcr2.atlanta.opentransit.net (193.251.128.130) 141.717 ms 134.771 ms 140.719 ms 9 so2-0-0.atlcr1.atlanta.opentransit.net (193.251.128.126) 128.368 ms 131.772 ms 133.031 ms 10 p12-0.oakcr2.oakhill.opentransit.net (193.251.128.166) 125.632 ms 124.872 ms 122.715 ms 11 p3-0.oakcr1.oakhill.opentransit.net (193.251.243.81) 130.962 ms 122.494 ms 123.683 ms 12 p4-0.nykcr3.new-york.opentransit.net (193.251.242.253) 124.902 ms 121.155 ms 135.26 ms 13 p11-0.nykcr2.new-york.opentransit.net (193.251.241.217) 123.852 ms 143.848 ms 122.225 ms 14 p6-0.nykbb1.new-york.opentransit.net (193.251.241.234) 124.196 ms 120.325 ms 121.752 ms 15 iar1-so-3-3-0.newyork.cw.net (208.173.135.49) 119.919 ms 123.62 ms 126.001 ms 16 agr1-loopback.newyork.cw.net (206.24.194.101) 144.715 ms 133.117 ms 121.273 ms 17 dcr1-so-7-0-0.newyork.cw.net (206.24.207.65) 121.781 ms 138.223 ms 117.651 ms 18 dcr1-loopback.washington.cw.net (206.24.226.99) 131.868 ms 133.121 ms 151.216 ms 19 bhr1-pos-0-0.sterling1dc2.cw.net (206.24.238.34) 153.02 ms 157.421 ms 153.004 ms 20 csr03-ve242.stng01.exodus.net (216.33.98.219) 146.549 ms 150.491 ms 124.85 ms 21 216.35.210.122 (216.35.210.122) 129.982 ms 131.641 ms 131.883 ms 22 www9.dcx.yahoo.com (64.58.76.178) 180.406 ms 160.385 ms 123.175 ms
Here, every hop that a packet must go through when following the path from my computer to www.yahoo.com is shown. The direction of the packet flow is very important, as in Internet there's no guarantee that the path followed in the return trip is the same. The quantity of hops can give an idea of the network's complexity.
For each hop, which is conveniently enumerated by traceroute in the first column of its output, three packets with a TTL that is gradually incremented are sent. The name of the node that answers, be it a succesful response indicating that the packet arrived at its final destination or an error reporting the expiration of the packet while in transit, is shown along the times taken by the answers to be received. This gives an idea of the condition of the network either in that or in a previous hop.

The previous example allows a better visualization of the inner workings of this process. In the first hop, towards the node 1, traceroute sets the TTL to 1 and sends the packet towards the destination node. When the node 1 decrements the value of the TTL and obtains zero, it returns an error message to the origin node stating the expiration of the packet while in transit. This process is repeated various times and the times are registered.
For the next hop, traceroute increases the TTL by one and sends the packet towards its destination. The node 1 decrements the TTL to one and forwards the packet to node 2. Node 2 then receives the packet with TTL one and decrements it, obtaining a zero TTL which triggers the corresponding error message towards the origin node. This process is repeated for ever increasing values of the TTL to find gradually farther hops, until the final node is reached or a maximum TTL is reached. This maximum limit is usually 30 but can be up to 255.
Other ways and variations
By combining the gradual increment of the TTL and the generation of some type of answer, it is possible to vary the way in which traceroute generates the map of the network. A common variation consists in the sending of UDP datagrams,that may be answered with differend kinds of errors depending on wether a node was reached or not in this hop.

