|
|
|
Did anyone notice that many producer shows were not cablecast last Sunday (January 08, 2006) through mid-day Monday? There was a message screen displayed stating something to the effect that "Due to technical problems …". At the Board of Directors the next evening, Zane reported a hardware problem affected playback, but due to his diligence, programming resumed by the time the board met that evening. He can sure make himself smell like a rose, but truth be known, it was his direct actions and interference with the operations in playback that caused the outage.
When Access SF moved into their current facilities, they purchased a TiltRac robotic tape jukebox automated tape playback system made by Synergy Broadcast Systems. One unit handles VHS and SVHS, another handles DVCAM and (allegedly) MiniDV. The system also included a audio/video encoder and a four channel digital playback server, which I may discuss at another time. The system never could properly handle the MiniDV tapes. During automated playback, the robotic arm often dropped the tape, bringing down that unit. Synergy claimed it was an alignment problem, but they never got it aligned well enough to work reliably. This has been an ongoing problem since day one. Questions need to asked;
Whenever Synergy claimed to get it working and Tom Barkett tried to bring it back on-line with MiniDV tapes, the robot would end up dropping a tape in the middle of the night, the system would come down, shows wouldn't be run, and Tom would get paged in the middle of the night to come in and recover the system. Tom got tired of fighting it. He started working around the problem by dubbing all MiniDV tapes to DVCAM for playback. Tom setup the main studio control room, the flash studio, and the staff dubbing rack for digital dubbing of MiniDV to DVCAM. Unless a producer was using the decks, they were often being used to dub MiniDV to DVCAM for playback. This had been going ion as long as I had been working at the station. Each of these dubs were digital dubs over firewire with no loss of quality. Nearly half the tapes submitted for playback are MiniDV, and we've been dubbing each one for playback. Everyone knew Tom as a great person, but I wonder if you really understood how much effort he put in to get all of the producer tapes played. Please note that this problem did not affect VHS or SVHS playback, but they have their own problems which I may discuss at another time.
In recent months, the DVCAM TiltRac's robotic claw, which grabs the tapes, started to jam up. It would get stuck in the open position, so no tape could be grabbed. Synergy miss-diagnosed the problem and instructed Tom to drip some lubricating oil onto the claw's solenoid to get it to free up. The thought of dropping lubricating oil down an electrical solenoid makes the hair on the back of my neck stick up, but Tom followed the vendors directions. The problem would clear up for a couple of days and then fail again. Whenever the claw got stuck, TiltRac would fail, go off-line, page Tom, bypass shows that couldn't get loaded, and so on and so on. After Zane forced Tom out, I pressured TiltRac to correct the problem.
The plan was to first stabilize the system by fixing the claw, and then address the MiniDV problem. Synergy would not come on-site and demanded that we send them the unit for repair. They claimed to not have any replacement units or spare parts. After many phone calls they said they would send us a part by overnight express that we could install ourselves. I told them I needed the part, schematics, instructions, and a list of what tools were needed. Let me tell you, that never happened. We finally got the part after three weeks, and had to replace it ourselves with them talking us through it over the phone. We got the claw fixed, the system was stabilized, and we were dealing with the timeslot selection fiasco.
With the claw working, the plan was then to schedule a controlled shutdown of the DVCAM robot, realign the unit for MiniDV, and do rigorous testing prior to bringing it back on-line. We expected to have the unit down for a day or two, so we bought some new VHS tapes, so we could dub a couple days of MiniDV and DVCAM shows to VHS, and use only VHS and SVHS for playback during the maintenance and testing. We would be testing with house test tapes. Our audience wouldn't have seen any outage at all. At least that was the plan.
Zane was fully aware of the plan and actively participated in its formation.
After the beginning of the year, with Sam Long fully trained as the Programming Coordinator, I returned to my duties as a production facilitator. I was still Sam's backup and I was the primary on-call person since Sam didn't have dial-up capability at home. Regardless, Zane kept me out of the loop with what was about to take place. At the end of last week, Zane decided to fix the MiniDV problem himself, right then and now. He called Synergy technical support and had them realign the robot by remote access. Without the benefit of sight, they reset the alignment parameters. Zane ran some limited testing and decided it was all fixed. Zane had Sam stop dubbing MiniDV tapes and load Sunday's programming (our most heavily scheduled day) with the producers non-dubbed MiniDV tapes. The first one was scheduled to start at 7:00 a.m., Sunday morning (January 08, 2006), when nobody would be at the station to deal with problems, when Sam would be out-of-town with relatives visiting from out-of-town. Damn the torpedoes, full speed ahead!
Here's an excerpt of my problem report to Sam;
My pager went off at 6:50 this morning to wake me up. When I dialed into TiltRac, I found that my access was revoked. I called you and left voice mail. Then I called Zane and left voice mail. After trying your phone one more time, I got dressed and came in.
A MiniDV "Words & Power" (07:00) was had dropped below the claw on the robotic arm and had not played. Flatman 2 (itself) was down, as well as a couple decks. The DVCAM dub of "Gospel from the Roots Up" (07:30) had not played, and the DVCAM special "Memories of …" (08:00) had not loaded. I was about to manually start "Memories of …" as Zane arrived.
You may not know this, but TiltRac usually fails to unload tapes that are started manually, even when you manually correct the event and device statuses. Zane allowed me to stay long enough to bring the system back up, manually unload "Memories of …", dub "Words & Power" , reschedule it (and its 1:00 a.m. repeat), and reschedule "Gospel from the Roots Up".
Zane let me know that he didn't want me to be on-call or respond top TiltRac failures. He allowed me to remove my mobile and home phones from the event pager list.
As I left, I reminded Zane that there were additional MiniDV tapes scheduled, and there will likely be another failure.
As I'm writing this, I see that "Soul Full of Gospel" failed to play at 10:00. Oh, I see the 10:30 show failed to start, so I presume Flatman 2 is down. The show at 11:00 appears to be just static.
Here's an excerpt of my note to Zane;
When I was paged again this morning at 6:50 a.m., I found that my remote access ********** had been revoked. I'm still being woken up with pages, but I am unable to log in and investigate the problem. The last I knew, Sam doesn't have a land line allowing him to dial into TiltRac. I called Sam on the phone and got voice mail. I called you on the phone and got voice mail. I tried Sam one more time, and when I got voice email again, I dressed and came in.
As I was diagnosing the problem and preparing to manually load the next show, you arrived and told me that you had Sam remove my remote access and that I should leave. I was unsure whether you wanted me to leave immediately, or continue taking corrective action. After I manually started the next tape, you told me that I should leave the (MiniDV) tape (that caused the failure) on the tabletop and leave, since I no longer had any responsibility to respond to TiltRac failures.
On my way out, I double checked with you to let you know that unless corrective action was taken, Flatman 2 would continue to remain inoperative. You had me stay long enough to bring TiltRac back up and reschedule the two shows that had failed to play. Then, as I was leaving, I reminded you that there were more MiniDV tapes scheduled to play and so it was likely we would experience another failure.
It turned out that Zane removed my remote access and Synergy's remote access. Sam didn't have remote access yet because he couldn't dial in from home. No one now had remote access to diagnose or fix programming issues. Zane removed my remote access, but retained the paging which woke me up, caused me to drop my Sunday morning plans to come in to fix the problem, just to be told it wasn't my concern and sent home. Sam was in Sonoma County with visiting relatives. Zane was alone in playback.
Zane Blaney, the SFCTC Executive Director (management), tried to adjust TiltRac's alignment parameters himself. Tom Barkett, the person who knew more about TiltRac than anyone, was forced out by Zane two months earlier. I (Michael Faklis), being Tom's backup for a year and the person who trained Tom's replacement, was sent home and told this is no longer my concern. Sam Long, the new Program Coordinator was in Sonoma County with visiting relatives, and out of phone range. Synergy technical support is out of the office for the weekend. So Zane steps in where he has no expertise and tries to readjust alignment parameters himself. What he ends up doing is miss-aligning TiltRac so bad that it can no longer handle DVCAM tapes either.
When Sam finally received his voice mail, the damage was done. Zane forced Sam to abort his outing and come into the station to recover TiltRac. There was nothing Sam could do. TiltRac documentation is lacking (that's an understatement), and Synergy technical support was off for the weekend. The only thing Sam could do was take the DVCAM/MiniDV robot off-line and post the "Due to technical problems …" message. The only shows that ran, were those on VHS or SVHS. Sam had to wait until the next day to get the vendor on the phone and guide him through the realignment.
Had Zane followed the plan that he actively participated in preparing, this outage would never had occurred. Had Zane allowed me to dub the MiniDV tapes loaded for Sunday, we would have lost only two shows. Since I was terminated two days later, I can only presume that Sam had to contact all of the affected producers and arrange replays. Having done that job for a couple months, I imagine a few of them expressed their anger and frustrations at Sam, rather than at Zane, who bore full responsibility for this fiasco. Zane ended up smelling like a rose by telling the Board of Directors that he fixed the problems with TiltRac.
Questions/Comments?Please report any problems with this WEB page to
.