I have my coffee, I just watched Sarah Lane's on Tech Crunch's "Crunch Report" on the Apple Sept 9th event. Poor, and is it ok if I say "girl"? and I mean, poor girl had to read copy that just made the whole thing sound booooring. however, her reads are fun. (If you're offended that I worked in a gender label, ok.)
Here's the Thursday AM brain dump. So a funny thing happened on the way to Tech 3 the other day.
Actually it, the project, started about 3 months ago.
Quick backstory, we use storage for our video assets. Not much, about a 180tb. On top of the storage and playing administrator is a DAM software from the company DALET. (dah' let) The evolution that we have running was well thought out and is an amalgam of numerous workflows. Version 3.14 something.
We set it up back in 2012 and have changed very little. Due to legacy issues around the facility upgrades would have to happen in a silo somewhere else and the infrastructure would have to be completely new.
DALET runs well for us. I didn't say that it was love at first sight, it was more like...ah, an arranged marriage.
Standard processes run at this facility (transcoding, migration, archive and retrieval, tagging, editing, web based VOD with over 34,000 assets online, live streaming 24/7 for 12 networks, global integration, etc) as well as multiple scripts planted on virtual machines that assist in functions for the web. Metadata, XSL, PHP, etc.
Old facility, aging storage, legacy processes; Keep this in mind.
When it was announced by the new IT director that he had secured a new storage infrastructure for us we were all ready for the whiteboard meeting. You know the one, where everybody dumps what they know to be true by their experience. All participants either map out or call out their connections to the current storage set, the person with the pen scribbles a crazy view of the connections. Some color coding, lot's of boxes, etc. The pain and "choke" points are then given priority and then the work begins. Reverse engineering, <inset\commontechphrases> and then the work starts for the cutover to the new storage.
Somehow, the whiteboard meeting got cut, as in, wasn't planned. The new storage Customer Satisfaction guy figures if his tech says "no need" and "well do it right"...then...great! We did meet in a room with a FULL FORCE GROUP from <storage\company>, Their team all give us the NO PROBLEM speech and then poof! gone. I still hear the words from my mouth. I want you to connect with these people and be sure that we won't break any necessary connections..." as they walk out the door.
Sept 1, we cutover to the new storage. The fact that DALET works on a DNS and not an ip was great. Well thought out BTW. The upside, DALET engineering was on site. Another side note; Don't do something this big without having this type of back up. Engineers on site are gold. Just don't expect them to be happy that they are there.
The downside, the VM that runs the PHP script was pointing at a "linux speak" for DNS. The script stop running on the cutover. The web uploads and data exchange quit completely. I didn't note it until about an hour post storage swap. My bad. A test that I didn't get to test..oh, wait, NONE OF THE PROCESSES WERE WHITE BOARDED! That does not remove responsibility but it does widen the project responsibility and the scope of recovery now includes more people.
A couple days into the recovery I used social media to reach out to John Chennavism and Nate Flom. In the meantime, I heard what the mystery was about.
Our broadcast IT guy and the storage tech were tossing around a lot of vocabulary around the room. Another preface, I'm not a Linux guy, I'm a former editor turned DAM and PM here onsite. Attrition required that I take on the admin of straggler projects, manage day to day. blah blah blah. All that said to say, I don't always speak up because I don't think highly of my skills..but...actually, I do. Let's not split hairs, it's a humility thing. I you say that you aren't you might be and if you say that you are..well...are you?
Back three months, I clearly remember the new storage tech guy saying, "The last storage that you had set up was done completely wrong. I'm not sure why they did what they did but I've never seen anything done like this on storage before. I'm going to" ...while he was speaking, I faded out.
When he completed his "MILESTONE" speak, I said then and there, "I remember there was a lot of discussion and planning going on regarding all of the connections and connection types. It was well thought out and testing took about 6 months as we were building something that had not been tried before. Both the storage engineers who built the system and the software engineers come onsite for a couple of days to pound out the necessary steps. I would think that you might gather the info, create a cut over date with an opportunity to roll back and adjust."
Come to present day from three months ago, last week, there we were with the engineer who "set it up right" was going over the disconnected portion of our workflow. What they could not find was where the Linux box was pointing at to achieve the mount for the folders on the storage. mnt\dalet was an issue. (I'm not saying that his way is wrong, I'm noting that he assumed that the other set up was so far out of the wheelhouse that it was hard for him to retain the logic. It wasn't wrong, it was different.)
The current situation was an all hands on deck situation. Service tickets were pulled, phone calls were made and then the Tuesday after the holiday, an onsite was established with the storage tech, our IT personnel, and myself. Four people in a two person office, sitting in the office furniture not made for butts. A whiteboard is not fully and effectively being used. Not a single "I told you so". Just brain dumps. Two slashdots banging out on a keyboard. Now, my personal issues, here at work or when contracting, I'm a bit nervous and feel under-skilled sitting in a room of slash\backslash\dot\bracket types. What I did notice is that they, the slashdot guys where stabbing in the dark like I usually do, it was just being accomplished with a different vocabulary.
This is that exact moment when "You don't have to know what you're doing, you have to ask the right questions".