A visit to Telecom's Network Integration Lab

Published Thu 22 November 2007 21:06 (+1300)

New IEEE Central Section

Today I went along to the first AGM for the IEEE New Zealand Central section. Central was recently promoted from a subsection of South to a section in its own right, thanks to all the committee ppl for their work towards that.

Being a full section means basically more resources to build up the local IEEE activities, hopefully a bunch of things will be happening next year. The timing was actually lucky – shortly after it was declared there'd be no more Region 10 sections for the immediate future (as much as anything due to the continually increasing cost of flying all the extra section heads in the Asia-Pacific region up to Canada for the training conferences!)

Anyway, our AGM was kindly hosted by Mike Brown of the Telecom Network Integration Lab here in Wellington, and he gave us a tour of the network testbed facilities there.

I would tell you that if you're not into telecommunications hardware you're not going to be interested in the rest of the post, but something tells me you've figured that out own your own already ;).

Telecom's Network Integration Lab

It was interesting to hear what they're working on, and what they're not. And, hey, lots flashing lights and network cards and stuff. We like it.

The whole facility, incidentally, has about NZ$30 million worth of gear in it – it's not just a few bits and pieces, it's a complete makeup of the entire network systems that we have around the country. And that includes all extant generations too – lots of big ATM boxes still lumbering around (well, lumbering in a stationary way… screwed into cabinets).

It's mostly telecommunications equipment, though that's changing – lots of Sun boxes there (and a few from other vendors), mostly for the new service stuff – as software becomes a bigger part of what they do (and related companies like EDS want to get closer in on the lab action).

As well as that sort of expansion of what's tested there, they are looking at how to build in more batch testing. Originally it was sufficient to thoroughly test systems before the first units go out; now, they're finding nasty surprises when subsequent batches are shipped to them.

Mike gave ADSL filters from China as an example – the first batches they got were all produced in a factory and so had very consistent quality. However some time later, the batches started failing; turns out they were instead being hand-wound by farmers god knows where, and so each unit behaved differently depending on which farmer happened to wind it, making a certain percentage unusable.

So for the future, they may have to test all the batches of equipment that the company orders in, rather than just the initial shipments.


Dad will be slightly pleased to hear that they had a small pile of very compact 8-port DSLAMs; these little babies are going to go into the roadside telco cabinets, whereas previous generations were still focussed around the big ugly cabinets in the exchanges.

(Dad will be pleased because it means they might actually get a real internet connection on the farmlet they recently semi-retired to… but only slightly pleased as they still have the same problem of Telecom refusing to put one in until there's a bunch of customers on the street who want one, which was the real blocker last time he asked.)

Fiber to the home

4th-gen prototype fiber-to-home CPE (not much to look at, but..)
4th-gen prototype fiber-to-home CPE (not much to look at, but..)

The grey plastic boxes under Mike's hand in this photo wouldn't even qualify as nerdporn, let alone interesting to look at for the world at large, but we will still be very pleased if and when they start showing up on our houses: they're fiber-to-the-home termination boxes, and when you have one, your days of full tubes are over.

Will we ever get there? We damn well hope so, but I don't suggest putting bets on it – people thought we'd all have it by now, and only a small number of places in the world with high-density urban areas use it at all, let alone as the norm.

Basically, we keep being able to get more speed out of copper, which combined with the usual chicken-and-egg problem of spurring higher utilisation (especially with all the dumb actions of the content cartels), so the economics of putting in fiber terminated at each home just don't work out yet. One day…

Next-Generation Network testbed

Telecom Next-Generation Network gear
Telecom Next-Generation Network gear
Not-the-next-generation network switch
Not-the-next-generation network switch

The gear shown here is part of Telecom's Next-Generation Network testbed (compare and contrast with the not-the-next-generation-network equipment in the foyer). The NGN project is pretty much what you'd expect, all-IP core, replacing traditional phone services with VOIP, looking at home-to-air roaming, ditching legacy ATM and pigeon carriers, all that kinda stuff.

I asked about IPv6 and was surprised to hear that they aren't going to that as part of the NGN – no customer requirement for it yet. Of course, they are making sure that all the gear that they're buying does it, for future extendability.

Incidentally, they have already been migrating a lot of stuff of legacy platforms, often without requiring any real reintegration on the customer part – eg. the decades-old frame relay services don't actually run over native frame relay across the country any more, they get mapped onto the more modern backbone services.

Given the amount of money at stake in big decades-long technical solutions, it's not surprising that like all big NZ electrical engineering & networking companies (hi Kordia!) they put a lot of effort into full testing and integrating all their gear; there's no blue-sky flash-cutover bollocks like you get when software resellers are contracted to come in and replace a company's IT systems, everything is fully made up and tested, full migration plans are made and tested, backup options are provisioned, etc.

All this gives their customers a big warm fuzzy feeling and gives them a pretty high sale rate once they get customers in the door of the lab :).

Technology lifecycle changes

IMHO much more important than the specific technology changes is that they're having to move to completely different lifecycles (with completely different equipment and technology lifetimes).

The NEAX exchanges that still run the bulk of our phone system were put in on the basis of a 25-year lifespan; all the new stuff has a lifetime of just 3-5 years – a completely different ballgame when it comes to investing in infrastructure!

Getting return on investment is going to be a huge problem – I mentioned earlier the new smaller DSLAMs, for low-density areas; in cities we'll still use (later-gen) cabinet DSLAMs but while as you'd expect these are much higher-density than those old Nokia cabinets, the price per port has shifted downwards quite little.

That's bad. No cheap DSL for you, at least not under the current adoption & utilization rates… but more about that another time ;).

It also means that they need to be able to experiment with, trial deploy, debug, and scale out on a vastly faster timescale – if you spend a few years dicking around it'd be obsolete by the time you get it out there. So the whole business needs to be based on a continuous evolution, moving-target technology adoption model. Balancing that with the need for speed is difficult.

High capacity

Incidentally, one of those new half-cabinet DSLAM racks pulls down up to 125 amps, which in hand-waving terms suitable for everyday conversation, is equivalent to 6 bananas a second or 20 lanes of traffic.

I think you'll agree that's a lot of bananas, cars, or electricity.

Modern telco equipment rooms are fed with hand-width busbars (busbars are basically like big cables, but the metal conductor is so big it's just a big solid beam of metal), the kind I've only seen previously in Dad's TV transmission station switchgear; processing performance and density is continually going up, but power use is rocketing through the ceiling.

And yeah, you will not be able to lift one of these on your own :). They are effectively working closer and closer towards being intricate lumps of metal, semiconductor, substrate, and coolant, so they are damn heavy.

Managing contention

The lab isn't just used for one thing at a time; there will be a number of projects on at all times, and they won't just be local or in-house Telecom projects – in addition to all the integration work with other NZ companies, the testbed facility sees use by vendors and customer companies all across the world. It's valuable simply because it's a real representation of what gets used in production, not a miniaturized cut-down.

All this means they have to manage efficiently. Every day they have an almost Scrum-style meeting to lay out who's working on what that day, what's going to be achieved, and lay down the schedule for who's using what gear when; this works so well that IIRC they've had about 4 instances of unresolvable schedule conflicts in the time they've been running the lab.

The engineer's grotto

Part of the engineer's grotto
Part of the engineer's grotto

The bit where you'll find the 40-or-so engineers that're there on any given day is quite amusing, Mike joked that it was modelled to be like Mum's basement – low ceilings and dim lights. And lots of screens!

Human ear model testing
Human ear model testing

This little guy is a very accurate and very expensive human hearing modeller. By carefully positioning the cellphone at international standard distance from the head – you do hold your cellphone at the ISO standard cellphone distance, right? – you can get a very precise model of how clear it sounds, taking into account the complex shape and material of the ear, even the resonance from the body.

(Sorry about the crap quality photos – didn't think to take my real camera.)

Out of the lab room and back in the foyer, there's several clusters of PS3s, Xbox 360s, PSPs, etc. I think they're there to slowly lure the engineers out of the cave so that at some point they can lock it and make them all go home instead of playing with the scopes and stuff all night. Anyway, it's clearly a fun place for tech geeks to work!

And a supercomputer

Oh, and they've got a supercomputer out back. BTW.

Put together in the last harried weeks of the LOTR production – with less than 2 months to go Weta ran out of capacity to do the final battle stuff (which turned out to have a huge number of characters) bad; they needed an entire new array of render servers in addition to the not-at-all-small one they'd already put together, but they maxed out the space/power/HVAC capacity of their Miramir installation – they were already on a half-megawatt transformer and couldn't get any more!

So, a joint venture with Telecom ensued, and within weeks there were enough warm CPUs to make it the 4th most powerful at the time (though, actually, being a supercomputer has more to do with interconnect speed, not just the CPU speed this “supercomputer” provides, BUT anyway).

What's it doing now, 4000 CPUs and LOTR long over? Oh, you know, this and that… it's hired out by the minute, bit of biotech – AIDS protein stuff, that sort of thing, bit of modelling, some movies, started learning yoga. You know, keeping busy… just regular supercomputer stuff. Reading a lot of books lately, that's been really great. Might try out for the relocated Square Kilometer Array processing job next year…