by gg3 » Sat 22 Jul 2006, 03:53:49
Tsakach is pretty well on target for causes. Another possible cause would be a heat-driven breakdown in cable insulation. Either way the result is that underground cable goes short or open, or has a ground cross, and in the latter case it might even spot-weld itself to the inside of the cable ducts (heavy conduits) under the street (oh joy!).
Bleep, re. incompetence: what do you do for a living...?
I work in telecoms; have done more than my share of cable pulls and cable fault testing & troubleshooting over the years. And while it's true that 48 volts DC is hardly 14 KV AC, the basic principles are similar.
I have no doubt about the conditions Con Ed crews are dealing with in New York right now. We have some slang terms for that when it occurs in telecom cable & cross-connects: "spiderweb," meaning a God-awful mess that's a royal pain in the arse to work on, and "string & glue," meaning, work that was done "expeditiously" without following standard practices, usually as a "temporary" installation but which has become permanent over time. (And here I might mention, thankfully our dude who is in charge of cable is absolutely meticulous, I can always count on his work to be 100% with no compromises, which makes for not only smooth PBX installations, but vastly simplified troubleshooting if repairs are ever needed. In fact we can basically rule out cable troubles a-priori on his installations, so when trouble comes up, we can start looking elsewhere than the cable plant. This saves time like crazy. Behind every reliable PBX is solid cable plant, no exceptions.)
So... If you were on the Con Ed field crew going after these blackouts, you'd be looking for opens, shorts, and ground crosses. Then upon finding one, you'd have to backtrack it until you could isolate the bad segment. Then you call in the truck with the large winch that would try to haul the bad cable out from under the street. Ideally you'd splice good cable to the bad segment, and use the latter as a "pull string" to tug the new cable into place as it went.
If your bad segment was a short or ground cross, it is entirely possible that it could have welded itself to the underground ducts, which are basically bundles of pipes that carry various cables, electrical and telecom and other (in separate pipes of course!). You would notice this because your truck, despite a diesel motor rated for a couple hundred horsepower, would start complaining. Then if you were having a particularly unlucky day, the bad cable would snap, the truck would jerk forward just a little, and the motor would purr along as it hauled up half of the proverbial worm in the apple, leaving the other half down there still stuck, jamming the conduit.
Plus or minus a few well-chosen cusswords, you'd get on the radio and call for a backhoe. This would show up and position itself at a likely spot, along with a couple of guys built like football players, the latter wielding jackhammers to help break up the pavement. When the jackhammer crew got a reasonable amount of pavement broken up, the backhoe operator would start digging into the street. After a while, the ducts would be exposed, and along would come additional workers with shovels to dig out the last little bit where the digging was too finicky for a backhoe to get at safely.
The backhoe operator might have air conditioning in the cab. The rest of the crew are out there broiling in the hot sun.
So they dig out the duct where the bad cable has refused to come out, and then presumably go at it with gasoline-powered chop-saws or similar, to physically cut into the duct and get at the jammed cable, and then grind down the spontaneous spot-weld until it comes free. By this time, the truck with the winch has been repositioned and is now ready to give a mighty tug on the newly-exposed free end to pull the rest of it the hell out of there.
At this point the crew are highly relieved because the trouble has been located and the heaviest work done. Someone climbs back into the hole in the street, welds or otherwise affixes a sleeve over the two ends of the conduit to knit them back together again, and now it's time to pull new cable. A flexible rod gets shoved through the duct in much the same manner as a drain snake. It pops out at the far end, gets attached to the new cable, and once again it's time for diesel-powered tug-of-war to pull the new segment into place.
That being done, another worker climbs in and splices the new segment back into the line, and upon cimbing back out, calls for a partial load test. If this succeeds, they go to a full load test.
If all is well, the backhoe operator will then pile the excavated earth back into the pit, or will use gravel conveniently delivered by a dump truck that would be considered large in any other city in the world. This operation is followed by another worker with a plate-compactor or a trench roller, to consolidate the backfill. And then, hopefully the same day, comes the concrete truck, all forty or fifty tons of it fully loaded (NYC allows 15 cubic yard loads of concrete, i.e. 30 tons, on its streets), and another handful of workers to pour the subgrade for the street. Last but not least this will be cordoned off while the concrete hardens, which should take at least a few days but is often sped up by the use of special additives so the road is passable the next day.
By this time, your lights have been working again for at least a few hours while you watch the rest of the construction operations going on down the street, and while impatient drivers hoot their horns at the inevitable traffic delays.
Or not.
And if not, then what we are looking at here, described in the media as "just as soon as one area is brought online, another blacks out, and they keep popping in and out like that, and it's a bloody mystery and no one knows what's going on....", what we are looking at there, is a state of cascading systems failure. And as I said elsewhere, there comes a point where it becomes too expensive to fix. And that, fellow citizens, is the first step down the slippery slope to the proverbial Olduvai Gorge.