Assuming the reader is not familiar with any or all of the terms below, it's OK to skim over the tech speak because this is a story about people. And cake. So if your job is tech ops, or management, or the counting of beans, well there's something for everyone.
Dear MSO,
With your permission I'd like to tell you a story about a cable system and the people who make it work. The cable system in our story was upgraded to 750 MHz in the forward path in 1994 offering 78 analog channels and a 200 MHz digital tier to its subscribers. Six years after upgrading the downstream the Multi System Operator (MSO) elected to further upgrade the network to include a 5-40 MHz return path for the upstream with the excellent goal of adding high speed internet to the company's existing offerings. Our system is loaded with 29 value taps.
In the planning stages for adding a return path to the network modem outputs of 60 dBmV were incorrectly assumed during design review resulting in an upstream design parameter of 20 dBmV at the amplifier housing. Now 20 dBmV at the housing plus 29 dB of loss through the 29 tap equals a design parameter of 49 dBmV at the tap port. A couple dB for drop loss and a 2 way split in the home result in a significant percentage of modems throughout the network operating at a 55 dBmV upstream output level. Anything more than a 2-way split in the home—well, those readers who understand the upstream can do the math. For those who do not—suffice it to say, things get worse.
The upgrade contractor charged with getting the return up and running in this county knows the design parameters are pushing the edge a bit. He even mentions it in a meeting or two but stops short of putting his objections on paper because he knows the project manager does not like delays. Both project manager and contractor are responsible for the schedule and the budget, not the system's performance thereafter. That's the system's problem. Besides, the modem's maximum output assuming a QPSK modulation scheme is 58 dBmV, 3 dB more than the 55 dBmV scenario described above. If the designer wants 29 value taps and 20 dBmV at the housing who are they to question the decision? In truth, they were the experts and the first to notice the issue but there are schedules to keep and the beast at marketing needs to be fed. So the contractor balances the network accordingly and performs the required certifications. The project manager releases the certified nodes to the marketing folks and the MSO begins to enjoy a return on investment. The miracle of capitalism repeats for the quadrillionth time. What a country. What a world.
Okay, but what about the Cake?
Subsequent to the initial sales push the MSO decides, rightly or wrongly, that modems capable of 58 dBmV outputs (QPSK) should not operate above 52 dBmV. Why 52 dBmV? Frankly I'm not sure but these are smart people so we shall take it on faith that they had good reasons for the decision and move along with our story. Now some of you may be asking yourselves why anyone would purchase a modem capable of a 58 dBmV output (QPSK) then limit that modem to a 52 dBmV output but as I said, these are smart people so let's just move along shall we. In our story line techs are dispatched to correct scenarios throughout the county where modems are transmitting above the 52 dBmV threshold. Keep in mind that the line techs in our two examples below are judged on the number of referrals they manage to complete in a given day.
Our first tech, let's call him Johnny (Gets a lot), Dunn, or Johnny D for short, is relatively unfamiliar with the upstream. Frankly he might have dozed a bit during the two day training class provided by the fine and forward thinking managers at our MSO prior to the launch of modems in the network. He's a hand's on type of guy so doesn't much like that "book learning." As far as he's concerned the trainer is a bit of an egghead who doesn't understand the challenges techs face in the field. Utilizing his tried and true method of trial and error Johnny D eventually discovers that lowering the return pad 5 dB at the active feeding this home clears up the high transmit issue. He calls dispatch to ask if the remaining subscribers fed by the active in question are still up and running and learns that indeed all of them are. Eureka! Problem solved. Maintaining the return, he concludes, is easy.
Of course the composite drop noise from all twenty five homes fed by this amplifier is now arriving at the headend 5 dB higher than it was prior to the pad change, and a couple of the homes fed by a low value tap near the end of the run are now operating in what the MSO calls "low transmit" but hey, that won't be noticed until next week's report comes out and there will be no way to connect the dots back to Johnny D.
Now as techs go in our cable system, Johnny D is the exception, not the rule. Seven out of ten line techs attending the upstream training classes that, once again, were provided by the fine and forward thinking managers at our MSO, learned the importance of unity gain. They know that in order to clear a high transmit call in a professional manner they need to first check that the return is balanced correctly. Next they need to visit the subscriber and determine if it is possible to reduce the passive loss within the subscriber's home. Of course all of this will take significantly more time than it would take to simply adjust the pad at the nearest active.
Now our second tech is Donny (Big on Ethics and a Bit of a Book Worm but Still Hunts and Fishes So I Guess He's OK) Jablonski. We shall call him Jablonski for short. At Jablonski's first high transmit call he has managed to reduced the modem's transmit level from 55 dBmV to 53 dBmV by reconfiguring the first passive in the subscriber's home but he is still 1 dB short. Keep in mind the MSO's policy, rightly or wrongly, calls for no modems transmitting above 52 dBmV. Jablonski does the math and learns that the 20 dBmV at the amplifier housing, plus the 29 dBmV of tap loss, plus the 2 dB of drop loss, plus the 2 dB for the DC-8 he installed as the first passive in the home, equals 53 dBmV. That's one dB above the MSO's arbitrary spec of 52.
It's worth noting that Jablonski is aware that he is not well liked by his supervisor whom we shall call Buddy (Looks Good on Paper) Kant or Buddy K for short. Buddy K is not terribly impressed with Jablonski because Jablonski clears only four calls per day compared to Johnny D who clears eight. Now Buddy K is an intelligent, hardnosed manager who wants his company to succeed and of course would like to succeed along with it. Unfortunately Buddy K, while being an excellent manager of people, is not terribly strong technically speaking, and to make things worse, he is afflicted with a not so rare disorder among managers whereby he thinks facts and excuses are one in the same thing. Being the type of guy that he is he will not accept excuses from his workers, especially since he knows he can send guys like Johnny D out there to get the job done. Given all of the above our friend Jablonski finds himself between the proverbial rock, and the freaking hard place. He needs to clear this call so on his way out of the neighborhood he stops by the active feeding his high transmit subscriber and lowers the return pad by 2 dB; the one dB he needs to meet the MSO's arbitrary spec of 52 and a second dB just to be safe.
At this point in the story it may be useful to zoom out a bit in order to see what all of this means for our network. If we assume there are three Johnny D's working in the system 50 weeks each year and further assume that each Johnny D encounters ten high transmit issues per week, the result is (3 Johnny D's x 10 pad changes x 50 weeks) equaling a 5 dB degradation of unity gain at 1500 actives in a given year. Add the lesser changes being made by the Jablonskis in the system and it's easy to see why unity gain can be an elusive goal when management prescribes unrealistic operational criteria. In this case the systemic failures are a design parameter equal to 49 dBmV at the tap port and an arbitrary operational limit of 52 dBmV for top end modem transmit levels, leaving only 3 dB for drop and passive loss within the home.
What does any of this have to do with Cake?
In my view it is safe to say the world is populated with a significant number of highly intelligent people. For the most part trains and planes run on time, buildings tower a thousand feet in the air, men have walked on the moon, I can send a message to a person half way around the world and they will be reading it within minutes, and my fourteen year old has over three hundred friends on Face Book even though she has only met 40 of them. These things are not possible in a world devoid of intelligence so it is reasonable to assume that a proportionate number of intelligent people are involved in the operation of our cable system. Given the likely level of intelligence within our system it is also reasonable to assume that the issue described above would be resolved by these folks in due time. Certainly attempts were made to do so. Folks at the bottom half of the ladder made enough noise to get the attention of a reasonable soul or two at the middle of the ladder who subsequently began to discuss the issue with a few of the folks at the top of the ladder. Unfortunately in the case of our network, the manager near the top of the ladder had "more important" things to consider that week and besides he was not hearing about this problem in his other systems—which of course were not designed for 49 dBmV at the tap port, kind of an apples and oranges type of thing, but I digress—so he began to think our system's middle management was a group of incompetent whiners and did not bother to disguise his feelings. Of course he can be forgiven for thinking this because the guy has a lot on his plate for crying out loud. Once middle management got wind of how they were being perceived by upper management our system's systemic design issue was run back down the flag pole, and left to the techs in the field to resolve.
Time goes by while our MSO's technicians struggle to reconcile management's impossible expectations with the reality based limits of their life in the field. Eventually some equilibrium is achieved between the high transmit modem issues and the low signal to noise issues that came about as a result of all the pad changes being made to correct the high transmits. They don't call it "balancing" for nothing. Of course unity gain, along with its inherent benefits, is out the window but the Johnny D's in the system eventually gain a better understanding of how things work in the field by talking more with the Jablonskis, and the Jablonskis learn some hard life lessons regarding reality vs. perception and the difference between the way the world should be and the way the world actually is. Slowly but surely things calm down in the field and life in our network reverts to a more predictable routine of generally happy subscribers, management, and technicians. Of course no one is as happy as they could be with more realistic design parameters but folks are relatively happy none the less.
Alas, life in our story is similar to life anywhere else; it has its ups and downs. Along comes 16 QAM. For those of you who don't know what that means, things are about to get worse, again. Nodes that barely met signal to noise minimums in the QPSK environment due to all the pad changes the boys and girls in the field were making to mitigate the high transmits, refused to work at all with 16 QAM so the entire process repeats for the next few weeks while subscriber frustration mounts. Of course Buddy K's weekly reports eventually began to reflect a higher percentage of modems in the system transmitting above 52 dBmV but at least the nodes were back up and running.
The months and years go by in a relatively smooth fashion until our MSO elects, in an intelligent manner I might add, to fund some improvements to the network before launching VoIP in the system. The project manager working on the budget for our MSO recognizes that 29 taps are bad without knowing exactly why so decides to replace each and every one of them with 26 taps. Doing so is certainly a step in the right direction however had our MSO's manager consulted with the design team while building the budget for VoIP launch he would have learned that the 29 value taps were only the tip of the iceberg and that the bigger issue is a design based on 49 dBmV at the tap port. Indeed in addition to locations with 29 taps there were thousands of additional locations throughout the network designed for 49 dBmV at the tap port. For example DC-12's with 17 taps on the high loss leg, DC- 9's with 20 taps on the high loss leg, 2 way splits with 26 taps etc, etc, and so on, and so on. The cost to change out the 29 taps reflected only a fraction of the cost to actually correct the issue at every location in the system. Basically what the system needed was a top down redesign or in the absence of an adequate budget for rebuild, a lower level at the tap port would have also worked. Indeed reducing the level at the tap port could have easily been accomplished as part of the sweep prior to launch of VoIP and would have allowed the MSO in our story to avoid both the cost for a rebuild and the less expensive cost for replacing the 29 taps. Wow!
Now I know what you're thinking—this is a no brainer. Could Milton Friedman have been wrong when he asserted that "There's no such thing as a free lunch?" Well, don't start writing your acceptance speech for the Nobel Prize in Economics just yet. Free market thinkers around the world can relax; there is a price to be paid in this scenario. It comes in the form of reduced signal to noise along the upstream and of course less downstream signal to work with in the home, but I digress, again. Once again it comes down to balancing. The alternative I propose means we are giving up signal to noise margin in order to lower our transmit level which in turn will allow us to avoid running our techs around in circles attempting to get two plus two to equal something other than four.
Back in our cable system the contractor providing the sweep prior to launching VoIP pointed the issue out at one of the MSO's weekly meetings but by this time the budget was set in stone and the project manager was not about to go back to the well for more money. The 29 taps were replaced with 26 taps as planned, the system was once again balanced for 20 dBmV at the amplifier housing, and the service calls began to roll in, albeit fewer than the first go round due to the change out of 29 taps. Of course the system techs were screaming to high heaven during all of this because they were forced to run from active to active to lower upstream pad values in order to clear the high transmits showing up on the MSO's weekly performance reports as the sweep contractor completed each node. In essence the MSO was paying the contractor to reestablish unity gain in the network before paying their in house guys, often in the same day, to disrupt unity gain in the same network in order to meet the system's arbitrary metric of 52 dBmV as a max transmit level. All the while the MSO's subscribers were caught in the middle. Does any of this sound familiar?
I'll get to the Cake Eventually I Swear
Is this a story about unity gain and high transmits, or a story about the importance of attaching the modem to the first split coming in to the house? I suppose to some extent it is, but from where I sit the moral of this story is how things can go bad within any large organization managed by human beings. One of my favorite Libertarian principles reads; Utopia is not an option, or as one of my favorite MSO managers likes to tell me, "we're not building a piano out here Mike". Both statements may be true but there is a vast difference between Utopia and a dysfunctional failure presided over by a leadership team with too many tasks on the plate and too few hours in a given day. Perhaps the budget will not allow for a top down network redesign but given the fact the MSO is planning a re-sweep prior to VoIP launch it certainly does not cost anything extra, in dollars anyway, to balance the network for 44 dBmV at the tap port as opposed to 49. Here at GoDirect we call this living within your realities. It sounds simple but fails to occur on a regular basis. The question is why?
- When the average tech in the field can't get things to work the way you want them to work, you've got a problem because super techs are in short supply.
- When middle managers are afraid to tell their bosses the truth, something is broken in your process.
- When compensation packages motivate project managers to bury systemic issues, something is wrong with the compensation package.
In all cases the MSO is losing money because they are paying multiple groups of people to accomplish conflicting goals and while this is going on they are frustrating their customers and likely losing a few in the process. In the world of RF, it always comes down to basic arithmetic. Overworked management needs someone they know they can rely on to tell them the truth even when they don't want to hear it. Two plus two will always equal four no matter how much pressure you bring to bear on the teams you manage.
Cake anyone?
If you're going to undertake a project, make sure to have representatives from the relevant equipment manufacturers on your site for at least the first few weeks. PPC, Gilbert, Motorola, Scientific Atlanta, Harmonic, CCor, Cisco, whomever. These folks are the experts. They know the limits of their gear. If you are planning an unrealistic set of operational parameters, they are likely to tell you so.
If you want to have a single Mega Contractor for the entire process that is understandable but consider paying a highly qualified, and independent group, say three to five techs plus manager, to set up the first batch of nodes. Pay them plenty of money by the hour but make sure they understand that the point of this mini project is to make things work as designed and to ensure happy subscribers post completion. This group could just as easily be pulled from your in house staff assuming they are qualified but that will only work if they are relieved of their current duties in order to focus 100% of their time and attention on the project at hand. Not only will this group work closely with your vendors they will also help you develop standards to measure the results. The most crucial part of this process will be running the service and line calls that will inevitably result post completion with the goal of finding out why they occur and what to do to prevent them in the future. In this way systemic issues that may arise from the project's activity will be discovered, documented, and resolved before you unleash the minions employed by your Mega Contractor. This group will work weekly with local project managers but should also have a conduit to, or simply answer to, someone higher up the food chain. Ideally all of the above will occur in a handful of nodes before you request bids for the bulk of the project.
Sure you will have to pay a premium to these groups and vendor reps but not having their involvement on the front end of the project may cost you tenfold more down the road when you are trying to band aid and gloss over systemic issues created by a few bad decisions made in the planning stages of the project. Once you find yourself in the heat of battle the need to meet the schedule will likely take precedence over the desire for a quality outcome.
Once you have achieved the desired results in the first group of nodes, and your subscribers in those nodes are happy, it's time to take bids from your mega contractors and get this ball rolling. The consultants, vendors, and in-house techs that helped you set up a working process may now be employed to help you keep your Mega Contractor's folks on the right page.
If it helps, ask your finance people to think of the additional cost on the front end as R&D and explain that there will be a return on the back end. What I propose is investing a lot of dollars in the first 2% of the project in order to save a far greater amount in the remaining 98%. In other words, it's cheaper this way. Poor planning is expensive. Systemic issues do not cure themselves. Eliminating dollars from the project's budget on the front end is no guarantee you won't spend three times more maintaining the dysfunctional network that results from poor planning and execution. And don't forget the losses to your top line when your competitors come to call on your frustrated customers.
Perhaps you really can have your cake, and eat it too? If that's true, do you really want to stiff the waiter?
Sincerely
Oct, 2009