One of the natural questions after reading this article about NetAddr::IP, is a comparison between other CPAN modules performing similar functions, such as Net::Netmask and Net::CIDR.
After perusing the documentation of said modules, I found some overlap in functionality (i.e., representation and parsing, to some degree). NetAddr::IP supports more functions that either Net::CIDR or Net::Netmask, so comparing functionalities was a bit moot. I went ahead and prepared a set of benchmarks designed to compare the relative speeds of the tasks I mentioned on my tutorial and then some. I left some of the obvious tasks such as comparison, conversion, etc. outside because either it was not worth it or the other modules did not directly support similar functionality.
A bit of caution is in order. In the results shown below, I cite execution times. You should consider these relative to each other. Better yet, since the test scripts are included too, you might want to download them to your machine and test them yourself. This will give you an idea of the actual speed you can expect from any of the covered modules.
Another bit of caution is also in order. I wrote NetAddr::IP, so I am much more familiar with it than with the other modules. This also may cause some bias in my assessment of what constitutes an important feature. With this in mind, let's get busy...
General methodology of the comparison
I used the tasks from my prior article that can be performed by more than one module as a basis to compare performance. The only exception to this is splitting the address space, as I consider this functionality important. So important in fact, that this was one of the reasons for writing NetAddr::IP in the first place.
I wrote scripts using each of the modules involved in this comparison and used Benchmark and Benchmark::Timer to compare the relevant pieces of code so as to disregard the overhead of non-related operations such as loading the example datasets. The result of each test along the code I used is discussed in the corresponding section below.
Below, I show the version of everything used in this article.
$ perl -MBenchmark -e 'print $Benchmark::VERSION, "\n";' 1.04 $ perl -MBenchmark::Timer -e 'print $Benchmark::Timer::VERSION, "\n";' 0.5 $ perl -MNet::Netmask -e 'print $Net::Netmask::VERSION, "\n";' 1.9002 $ perl -MNet::CIDR -e 'print $Net::CIDR::VERSION, "\n";' 0.04 $ perl -MNetAddr::IP -e 'print $NetAddr::IP::VERSION, "\n";' 3.14_3 $ perl -v | egrep This This is perl, v5.8.0 built for darwin
Grouping of the address space
For this test, I want to convert a large set of IP addresses into the most compact CIDR representation possible. This would be one or more prefixes representing exactly all the addresses fed as input.
The first action was to write a script that would generate the ranges for the test case. This simple tool called ranger is shown below:
1: #!/usr/bin/perl
2:
3: use warnings;
4: use strict;
5: use NetAddr::IP;
6:
7: for my $r (@ARGV) {
8: my $ip = new NetAddr::IP $r;
9:
10: for (my $h = $ip->first;
11: $h <= $ip->last;
12: $h ++)
13: {
14: print $h->addr, "\n";
15: }
16: }
The following datasets were generated:
- dataset.0 A /24 worth of IP space (254 addresses).
- dataset.1 A /18 worth of IP space (16382 addresses).
- dataset.2 A /17 worth of IP space (32766 addresses).
- dataset.3 A /16 worth of IP space (65534 addresses).
- dataset.4 32 /25s taken from 32 /24s (4032 addresses).
I guess the shell code depicts a better explanation for the last dataset:
$ ./ranger `for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 \
19 20 21 22 23 24 25 26 27 28 29 30 31 \
32; do echo -n "172.16.$i.0/25 "; done` > dataset.4
However, note that the largest test cases were used only with NetAddr::IP due to the amount of time that Net::CIDR took in this particular benchmark.
For the NetAddr::IP solution, I used the compactref() function. The Net::CIDR solution merges the parsing and compaction of the address range in a single step, through the cidradd() function. For completeness, the NetAddr::IP solution is presented in two variants: One that parses and compacts the whole set of addresses and another, that only loads the set once. The loading and parsing process is also timed separatedly, for reference purposes.
It seems that Net::Netmask does not provide the functionality required for this test, so no solution or results can be presented.
The complete code for the benchmark, is listed below.
1: #!/usr/bin/perl
2:
3: use strict;
4: use warnings;
5: use Net::CIDR;
6: use NetAddr::IP;
7: use Benchmark qw(cmpthese timethese);
8:
9: my @space = ();
10:
11: my @ips = <ARGV>;
12:
13: sub net_ip_load
14: {
15: @space = ();
16: @space = map { NetAddr::IP->new($_) } @ips;
17: }
18:
19: sub net_ip_compact
20: {
21: my $r_compact = NetAddr::IP::compactref \@space;
22: }
23:
24: sub net_cidr
25: {
26: @space = ();
27: for (@ips) {
28: @space = Net::CIDR::cidradd($_, @space);
29: }
30: }
31:
32: print "Benchmarking with ", scalar @ips, " addresses...\n";
33:
34: my $loaded = 0;
35:
36: timethese(100, {
37: 'N::IP load' => sub { net_ip_load; },
38: });
39:
40: cmpthese(10, {
41: 'N::IP+load' => sub { net_ip_load; net_ip_compact; },
42: 'N::IP' => sub { unless ($loaded) { net_ip_load; $loaded ++; }
43: net_ip_compact; },
44: 'CIDR' => \&net_cidr,
45: });
The following output shows the result of the benchmark for the two smallest data sets. As you can see, the results seem to indicate that NetAddr::IP::new() is expensive. This is probably due to the number of formats supported by NetAddr::IP. I didn't use all of the datasets with all of the modules due to the time it took for the test cases to run.
$ ./bench-1 ./dataset.0
Benchmarking with 254 addresses...
Benchmark: timing 100 iterations of N::IP load...
N::IP load: 6 wallclock secs ( 5.34 usr + 0.00 sys = 5.34 CPU) @ 18.73/s (n=100)
Benchmark: timing 10 iterations of CIDR, N::IP, N::IP+load...
CIDR: 36 wallclock secs (33.61 usr + 0.00 sys = 33.61 CPU) @ 0.30/s (n=10)
N::IP: 1 wallclock secs ( 0.71 usr + 0.00 sys = 0.71 CPU) @ 14.08/s (n=10)
N::IP+load: 1 wallclock secs ( 1.21 usr + 0.00 sys = 1.21 CPU) @ 8.26/s (n=10)
Rate CIDR N::IP+load N::IP
CIDR 0.298/s -- -96% -98%
N::IP+load 8.26/s 2678% -- -41%
N::IP 14.1/s 4634% 70% --
$ ./bench-1 ./dataset.1
Benchmarking with 16382 addresses...
Benchmark: timing 100 iterations of N::IP load...
N::IP load: 390 wallclock secs (350.75 usr + 0.00 sys = 350.75 CPU) @ 0.29/s (n=100)
Benchmark: timing 10 iterations of CIDR, N::IP, N::IP+load...
CIDR: 5081 wallclock secs (4479.64 usr + 0.00 sys = 4479.64 CPU) @ 0.00/s (n=10)
N::IP: 57 wallclock secs (49.74 usr + 0.00 sys = 49.74 CPU) @ 0.20/s (n=10)
N::IP+load: 94 wallclock secs (84.50 usr + 0.00 sys = 84.50 CPU) @ 0.12/s (n=10)
s/iter CIDR N::IP+load N::IP
CIDR 448 -- -98% -99%
N::IP+load 8.45 5201% -- -41%
N::IP 4.97 8906% 70% --
$ ./bench-1 ./dataset.2
Benchmarking with 32766 addresses...
Benchmark: timing 10 iterations of N::IP, N::IP+load...
N::IP: 121 wallclock secs (109.56 usr + 0.00 sys = 109.56 CPU) @ 0.09/s (n=10)
N::IP+load: 197 wallclock secs (167.16 usr + 0.00 sys = 167.16 CPU) @ 0.06/s (n=10)
s/iter N::IP+load N::IP
N::IP+load 16.7 -- -34%
N::IP 11.0 53% --
$ ./bench-1 ./dataset.3
Benchmarking with 65534 addresses...
Benchmark: timing 10 iterations of N::IP, N::IP+load...
N::IP: 255 wallclock secs (240.24 usr + 0.00 sys = 240.24 CPU) @ 0.04/s (n=10)
N::IP+load: 468 wallclock secs (383.92 usr + 0.00 sys = 383.92 CPU) @ 0.03/s (n=10)
s/iter N::IP+load N::IP
N::IP+load 38.4 -- -37%
N::IP 24.0 60% --
$ ./bench-1 ./dataset.4
Benchmarking with 4032 addresses...
Benchmark: timing 100 iterations of N::IP load...
N::IP load: 99 wallclock secs (87.09 usr + 0.00 sys = 87.09 CPU) @ 1.15/s (n=100)
Benchmark: timing 10 iterations of CIDR, N::IP, N::IP+load...
CIDR: 11805 wallclock secs (9650.87 usr + 0.00 sys = 9650.87 CPU) @ 0.00/s (n=10)
N::IP: 14 wallclock secs (11.71 usr + 0.00 sys = 11.71 CPU) @ 0.85/s (n=10)
N::IP+load: 23 wallclock secs (19.92 usr + 0.00 sys = 19.92 CPU) @ 0.50/s (n=10)
s/iter CIDR N::IP+load N::IP
CIDR 965 -- -100% -100%
N::IP+load 1.99 48348% -- -41%
N::IP 1.17 82316% 70% --
Just to be sure, I randomly shuffled the the data and run the same benchmark again. The results seemed to hurt Net::CIDR the most, as the time increased significantly (this is why only the smallest dataset was used, but feel free to try with the other datasets in your spare time). Since NetAddr::IP and presumably Net::CIDR seem to depend on sorting its arguments, you should have this in mind when writing code with any of these alternatives. It seems that feeding pre-ordered addresses will reduce the execution time. I believe this has more to do with Perl's sort than with anything else.
$ awk '{ print rand(), " ", $1 }' < dataset.0 | sort -n | awk '{print $2}' | ./bench-1
Benchmarking with 254 addresses...
Benchmark: timing 100 iterations of N::IP load...
N::IP load: 7 wallclock secs ( 5.05 usr + 0.00 sys = 5.05 CPU) @ 19.80/s (n=100)
Benchmark: timing 10 iterations of CIDR, N::IP, N::IP+load...
CIDR: 295 wallclock secs (258.31 usr + 0.00 sys = 258.31 CPU) @ 0.04/s (n=10)
N::IP: 1 wallclock secs ( 1.32 usr + 0.00 sys = 1.32 CPU) @ 7.58/s (n=10)
N::IP+load: 3 wallclock secs ( 1.72 usr + 0.00 sys = 1.72 CPU) @ 5.81/s (n=10)
Rate CIDR N::IP+load N::IP
CIDR 3.87e-02/s -- -99% -99%
N::IP+load 5.81/s 14918% -- -23%
N::IP 7.58/s 19469% 30% --
As can be drawn from the earlier results, the execution time of the test based on NetAddr::IP seems to increase linearly with the number of addresses, while Net::CIDR seem to increase quadratically. This was also present in historical versions of NetAddr::IP, but it's no longer the case.
Splitting the address space
This test deals with segmenting a large block of address space in smaller subnets according to CIDR. For instance, this is what you would do when a new network must be installed. For this test, I will split a /16 in increasingly small pieces (from /17 to /26). The same /16 is used.
It seems that neither Net::Netmask nor Net::CIDR provide the functionality required for this test, so no solution or results can be presented. The code used for the NetAddr::IP solution is depicted below:
1: #!/usr/bin/perl
2:
3: use strict;
4: use warnings;
5: use NetAddr::IP;
6: use Benchmark::Timer;
7:
8: my $ip = new NetAddr::IP "172.16.0.0/16";
9:
10: my $t = Benchmark::Timer->new();
11:
12: for my $len (17 .. 26) {
13: for my $times (0 .. 99) {
14: $t->start("split/$len");
15: $ip->splitref($len);
16: $t->stop("split/$len");
17: }
18: }
19:
20: $t->report;
These are the results of running this benchmark.
$ ./bench-2 100 trials of split/17 (54.992ms total), 549us/trial 100 trials of split/18 (67.422ms total), 674us/trial 100 trials of split/19 (90.040ms total), 900us/trial 100 trials of split/20 (140.712ms total), 1.407ms/trial 100 trials of split/21 (223.532ms total), 2.235ms/trial 100 trials of split/22 (409.919ms total), 4.099ms/trial 100 trials of split/23 (880.705ms total), 8.807ms/trial 100 trials of split/24 (1.567s total), 15.674ms/trial 100 trials of split/25 (3.173s total), 31.732ms/trial 100 trials of split/26 (6.751s total), 67.514ms/trial
These results show that the execution time increases with the number of subnets being produced by the splitting process, which in itself is not surprising. Since no array passing is taking place, I think we could safely assume that this increase comes directly from the splitref() method.
Range to CIDR conversion
This test deals with conversion of a range of IP addresses to the corresponding CIDR block. I will choose a few subnets, namely:
| 172.16.0.0 - 172.16.255.255 |
| 127.16.0.0 - 172.16.0.255 |
| 10.0.0.0 - 10.255.255.255 |
And perform the conversion a few times, in order to compare the execution times. The code for the benchmark is reproduced below. The NetAddr::IP solution uses the cidr() to return the subnet in the desired format. The Net::Netmask solution is based in the range2cidrlist function and range2cidr() is used in the Net::CIDR solution.
1: #!/usr/bin/perl
2:
3: use strict;
4: use warnings;
5: use Net::CIDR;
6: use NetAddr::IP;
7: use Net::Netmask;
8: use Benchmark qw(cmpthese);
9:
10: cmpthese
11: (
12: 1000,
13: {
14: 'Net::IP /16' => sub
15: {
16: NetAddr::IP->new('172.16.0.0-172.16.255.255')->cidr;
17: },
18: 'Net::Mask /16' => sub
19: {
20: Net::Netmask::range2cidrlist('172.16.0.0', '172.16.255.255');
21: },
22: 'Net::CIDR /16' => sub
23: {
24: Net::CIDR::range2cidr('172.16.0.0 - 172.16.255.255');
25: },
26: });
27:
28: cmpthese
29: (
30: 1000,
31: {
32: 'Net::IP /24' => sub
33: {
34: NetAddr::IP->new('172.16.0.0-172.16.0.255')->cidr;
35: },
36: 'Net::Mask /24' => sub
37: {
38: Net::Netmask::range2cidrlist('172.16.0.0', '172.16.0.255');
39: },
40: 'Net::CIDR /24' => sub
41: {
42: Net::CIDR::range2cidr('172.16.0.0 - 172.16.0.255');
43: },
44: });
45:
46: cmpthese
47: (
48: 1000,
49: {
50: 'Net::IP /8' => sub
51: {
52: NetAddr::IP->new('10.0.0.0 - 10.255.255.255')->cidr;
53: },
54: 'Net::Mask /8' => sub
55: {
56: Net::Netmask::range2cidrlist('10.0.0.0', '10.255.255.255');
57: },
58: 'Net::CIDR /8' => sub
59: {
60: Net::CIDR::range2cidr('10.0.0.0 - 10.255.255.255');
61: },
62: });
The results for this test are shown below:
$ ./bench-3
Benchmark: timing 1000 iterations of Net::CIDR /16, Net::IP /16, Net::Mask /16...
Net::CIDR /16: 0 wallclock secs ( 0.47 usr + 0.00 sys = 0.47 CPU) @ 2127.66/s (n=1000)
Net::IP /16: 1 wallclock secs ( 0.62 usr + 0.00 sys = 0.62 CPU) @ 1612.90/s (n=1000)
Net::Mask /16: 1 wallclock secs ( 0.41 usr + 0.00 sys = 0.41 CPU) @ 2439.02/s (n=1000)
Rate Net::IP /16 Net::CIDR /16 Net::Mask /16
Net::IP /16 1613/s -- -24% -34%
Net::CIDR /16 2128/s 32% -- -13%
Net::Mask /16 2439/s 51% 15% --
Benchmark: timing 1000 iterations of Net::CIDR /24, Net::IP /24, Net::Mask /24...
Net::CIDR /24: 0 wallclock secs ( 0.47 usr + 0.00 sys = 0.47 CPU) @ 2127.66/s (n=1000)
Net::IP /24: 1 wallclock secs ( 0.61 usr + 0.00 sys = 0.61 CPU) @ 1639.34/s (n=1000)
Net::Mask /24: 1 wallclock secs ( 0.37 usr + 0.00 sys = 0.37 CPU) @ 2702.70/s (n=1000)
(warning: too few iterations for a reliable count)
Rate Net::IP /24 Net::CIDR /24 Net::Mask /24
Net::IP /24 1639/s -- -23% -39%
Net::CIDR /24 2128/s 30% -- -21%
Net::Mask /24 2703/s 65% 27% --
Benchmark: timing 1000 iterations of Net::CIDR /8, Net::IP /8, Net::Mask /8...
Net::CIDR /8: 0 wallclock secs ( 0.42 usr + 0.00 sys = 0.42 CPU) @ 2380.95/s (n=1000)
Net::IP /8: 1 wallclock secs ( 0.63 usr + 0.00 sys = 0.63 CPU) @ 1587.30/s (n=1000)
Net::Mask /8: 1 wallclock secs ( 0.43 usr + 0.00 sys = 0.43 CPU) @ 2325.58/s (n=1000)
Rate Net::IP /8 Net::Mask /8 Net::CIDR /8
Net::IP /8 1587/s -- -32% -33%
Net::Mask /8 2326/s 47% -- -2%
Net::CIDR /8 2381/s 50% 2% --
Note that the throughput shown for the three modules remain failry constant for all the cases. NetAddr::IP is consistently, the slowest performer, probably because of the parsing.
Sorting a set of IP addresses
This test is not specifically mentioned in my original article but seems to be a strong point to Net::Netmask, so I decided to include it anyway. Besides perhaps this should have been included in my earlier article, as this seems like a common enough application. I will use the IP addresses contained in the dataset.4 data file (in reverse order) for sorting.
The code for the benchmark can be seen below. The NetAddr::IP solution relies in the implicit overloading of operators such as <=> and cmp that this module provides and that allows direct comparison among subnets. The Net::Netmask solution will be using the built-in sort_by_ip_address(). The Net::CIDR module does not seem to include functions to implement this.
1: #!/usr/bin/perl
2:
3: use strict;
4: use warnings;
5: use Net::CIDR;
6: use NetAddr::IP;
7: use Net::Netmask;
8: use Benchmark qw(cmpthese timethis);
9:
10: my @ips = reverse <ARGV>;
11: my @space;
12:
13: timethis(5, sub { @space = map { new NetAddr::IP $_ } @ips; },
14: "NetAddr::IP new");
15:
16: my $thing;
17:
18: cmpthese
19: (
20: 1000,
21: {
22: 'NetAddr::IP' => sub
23: {
24: no warnings;
25: $thing = sort @space;
26: },
27: 'N::IP + new' => sub
28: {
29: no warnings;
30: $thing = sort map { new NetAddr::IP $_ } @ips;;
31: },
32: 'Net::Netmask' => sub
33: {
34: $thing = Net::Netmask::sort_by_ip_address(@ips);
35: },
36: });
The results are shown below. Note that the results for the call to NetAddr::IP::new() have been separately timed in order to identify where the difference lies.
$ ./bench-4 ./dataset.4
NetAddr::IP new: 5 wallclock secs ( 4.49 usr + 0.00 sys = 4.49 CPU) @ 1.11/s (n=5)
Benchmark: timing 1000 iterations of N::IP + new, Net::Netmask, NetAddr::IP...
N::IP + new: 948 wallclock secs (796.85 usr + 0.00 sys = 796.85 CPU) @ 1.25/s (n=1000)
Net::Netmask: 1 wallclock secs ( 0.12 usr + 0.00 sys = 0.12 CPU) @ 8333.33/s (n=1000)
(warning: too few iterations for a reliable count)
NetAddr::IP: 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU) @ 50000.00/s (n=1000)
(warning: too few iterations for a reliable count)
Rate N::IP + new Net::Netmask NetAddr::IP
N::IP + new 1.25/s -- -100% -100%
Net::Netmask 8333/s 663942% -- -83%
NetAddr::IP 50000/s 3984150% 500% --
As you see, again the parsing that happens within NetAddr::IP cause a huge penalty in performance.
Conclussions
In the first place, it would seem that the parsing of the plethora of formats accepted by NetAddr::IP::new() is a burden that neither Net::Netmask nor Net::CIDR have. Their respective authors decided to support a couple of formats (the most common ones I would say) and stayed there. This is the most likely cause for the comparative performance in the range to CIDR conversion test. This means that if you need only simple representation and light manipulations of addresses in the formats these modules handle, you might be better served by them.
Perhaps eventually I will address the parsing performance issue in NetAddr::IP and improve it. However, performance was never a goal for it. The focus was rather in functionality and completeness.
Complex operations such as compaction and splitting of the address space are either much faster with NetAddr::IP or not doable with the other modules. NetAddr::IP provides a richer set of functions spanning much more practical uses, making the handling of subnets in Perl almost as easy as with any other "native type". This was indeed its reason to be.
Note that many of the features of NetAddr::IP were not excercised. The most important reason is that none of the other modules provide equivalent methods of achieving the same results.
I chose not to make any comparison among the interfaces that the different modules provide because this, to some degree, is either a personal preference discussion or a religious war.
As a final note, this article is meant to provide only a relative reference. This shows that some tasks are faster with one module or the other, but as versions and functionality evolve, you should check yourself which module is more convenient for your particular application.
