See Jim Jachetta, CTO of VidOvation, along with Bil Apker from TAG V.S., during a recent webinar for a look at IP-based multiviewer, diagnostics, and quality control that specific to live at-home production.
- How to meet low latency demands when using IP in live production
- Which diagnostics are critical and how to best monitor them for at-home productions
- New quality control approaches you can implement today
- Tales from the trenches, what we learned on a recent install
Jim Jachetta: Yeah, maybe. All right. It looks like we’ve got a good number of people logged in. Let’s get started. I’m recording. Good morning everyone, or good afternoon depending upon where you are. Jim Jachetta here. I’m the CTO of VidOvation. I’m with a good buddy of mine, Bil Apker, from TAG Video Systems. Let me give control over to Bil.
Bil Apker: Good morning everybody. I’m Bil Apker, vice president of sales for LATAM and North America. I’m glad to be here today.
Jim Jachetta: Yeah, you should have control, Bil. Let me see if you want to grab the screen.
Bil Apker: Sure.
Jim Jachetta: Wait, here presenter. Here you go. There you go. All right. You should have control now.
Bil Apker: Great. I’m just going to tell you a little bit about what we do at TAG. TAG is the world’s number one probing, monitoring and multiviewer software solution. 100% hot software, 100% IP, and 100% COTS. We manage both compressed and uncompressed in the same solution. We really focused over the last year on four specific broadcast and media applications. Live production, working with SMPTE ST 2110 and J2K, Playout and Master Control, which has a variety of flavors inside of that. Traditional delivery with H. 264, MPEG-TS. Then of course, OTT with a multitude of formats including the ABR stack. We do about 330 different probing capabilities inside the software. Anything from closed captioning, to PTP, audio video loss, [inaudible] 35. There’s a lot that we monitor and probe inside of there. We also support everything up to 12G uncompressed 4K on outputs and inputs. What that really means is that, you get more real estate for what you’re paying for.
Bil Apker: We’ve seen 100 pips on a nice, good size monitor look very, very good. In fact, you could see detail all the way down to lower thirds and things like that very easily. We have a completely open architecture as far as APIs are concerned. At least today, and we should change this slide, because we’re now monitoring or managing about 50,000 channels globally.
Jim Jachetta: Bil, I was introduced to your company about a year ago as a monitoring or a multiviewer company, IP based multiviewer. But your real roots are in this probing and the diagnostic, so you’re doing much more than multi viewing. Multi viewing kind of came along as a secondary function. Is that a correct statement or …
Bil Apker: Yeah, pretty much. I mean, if you look at what our probing capabilities are, they actually are extensive. When you’re looking at an ecosystem what it used to do, talk about the SDI world, where you had from the beginning to the end of the chain was all internal to a TV station. The production of the show, through master control, through the transmitter, that was all controlled by a television station. In the IP world, that is completely changed because the only real piece that anybody has internally is the processing basically. I mean if you’re still doing local production and smaller TV stations, that makes sense. But in larger organizations, we see a lot of the source cut material coming in from somewhere, whether it be AWS, whether it be another facility. Wherever it happens to be, it comes into a main hub and that hub processes it and adds all the stuff it needs to add, and then it pushes it out to numerous distribution networks.
Bil Apker: Whether it would be Hulu, whether it would be Disney+, whether it would be the upcoming Peacock, whatever it happens to be. If it’s live your doing it in multiple pieces. Then of course it’s leaving or it’s being distributed again out through the CDNs, and those CDNs are delivering it to your set top box in some way.
Jim Jachetta: You were telling me this earlier, you can probe at any point in the chain. We can elaborate on this more, but some of our customers, I see your technology being used by a transmission engineer. Some of you guys may have heard us talk about this before we actually got started. Where we work with a company by the name of AVIWEST where bonded cellular, doing at home production, bringing IP feeds into the master control. I could see TAG being used at the video engineer’s workstation. Now as workflows are changing, people are working from home. That IP multiviewer, that IP diagnostic workstation could be accessed from home. As long as the video engineer or the transmission engineer has an internet connection, they can monitor the entire system from anywhere. Then like you say, Bil, your probes, that’s just one probe point, the signals coming in. But then you can monitor the signals in the facility, going to each edit suite, going to your OTT provider, your CDN. You can probe at any of these points?
Bil Apker: Absolutely. This is a great example of what we’re talking about here. This is very generic, but we actually produce a slide from one of our customer’s conversations. What they needed to do is bring in content from these different places, either a live venue or a broadcast or in multiple ways. This could be coming from an IP source somewhere else. It could be coming from the cloud, it doesn’t matter to us. As long as it’s IP, it comes into the head end. Then at each one of these cogs, is kind of where we see that there’s some possibility of probing. These are typically IP addresses across the entire system, but we’re able to put a probe here and a probe here and a probe here and a probe in the output here and actually monitor the entire ABR stack. Actually monitor that entire ABR stack once it returns from the CDN. But it doesn’t matter to us because we take everything from a few kilobits all the way up to 12 gig uncompressed in the same system.
Bil Apker: Whether it is a MPEG-TS 50 here and an MPEG-TS 25 here and then some version of that in an ABR stack at whatever bit rate it is or whatever resolution it is, we’ll monitor it there. Then we’ll decrypt it and then bring it back in so you can actually see it. Imagine the broadcaster is this entire chain, they want to see it as they traditionally always have been able to see it throughout the entire process of making pictures on the output and sending it to a home.
Jim Jachetta: I don’t see it there. I forget how to turn on the little pointer here. Well, anyway, you see here where you have like … Yeah, go back to that slide you just had up, Bil. Or go forward one. Yeah. We had an issue where video was leaving the AVIWEST StreamHub in the cloud just fine. We weren’t dropping any packets. The audio was fine. We can look at the diagnostics within the AVIWEST instance, but we didn’t have this probe that you have here at CDN 1 and CDN 2. The CDN was claiming, “We’re getting the feed with garbled audio.” Back and forth, back and forth, it ended up that the CDN was having a problem, but there was no way for us to prove that point. If we implemented TAG Video Systems’ IP technology, we could probe within the CDN number one or CDN number two to figure out where the garbled audio was coming from. Is that correct?
Bil Apker: Absolutely. In fact, one of our recent customers that we actually built this slide around, is monitoring at several points from the packager, several renditions of the packager. They’re monitoring one rendition at four different CDNs. They’re bringing that all back into the facility so they can actually see it and then be able to report on it. Again, we’re monitoring at multiple points so that in the end we can go back and go, “Well listen, Akamai’s doing just fine. Why isn’t CentraLink working as well?” Not to throw anybody under the bus, but-
Jim Jachetta: It’s just an example, of course. Yes.
Bil Apker: Just an example.
Jim Jachetta: Yeah.
Bil Apker: But yeah, we’re monitoring it at numerous places and that obviously eases the mind of the broadcaster when he can sit there and go, “I know it’s clean going out of my facility. I know it’s clean at multiple CDN points. It’s not clean you, it’s got to be you.” It’s a very easy way to start pointing fingers.
Jim Jachetta: Well, and then the transmission engineers in master control see the problem before customers start calling and complaining. Minutes, hours could go by before a customer complains. This is advertising revenue. You were even telling me your probing can help predict problems. Maybe can you tell us more about that? How that works. That’s intriguing to me, that knowing a problem’s about to happen at some point in the future.
Bil Apker: Because we have a completely open API architecture and we work closely with a lot of NMS vendors. Whether it be DataMiner or MNC, that’s where really the power starts coming in. Because it’s not us that’s able to predict those failures. But it’s actually because we work so closely with the NMS folks, that they’re actually able to design algorithms or AI around what we’re doing to be able to predict failures in the entire channel chain. It’s a really great pairing of the two and it really takes the soccer approach to the next level. That’s really where the power comes in. I mean, we’re a probing monitoring and multiviewer company, so our focus is video. Their focus is capturing alarms and then doing something with them. That’s where we have a great relationship with a lot of them.
Jim Jachetta: Cool. Very cool.
Bil Apker: Yeah. We give lots of support and I can show you a … If we go to this slide. That’s not the one I want to show you. Where’s my other slide? Far too many PowerPoint presentations up right now.
Jim Jachetta: I’ll do some shadow puppets while you find the slide.
Bil Apker: Yeah. It’s here somewhere. What did I just do with that?
Jim Jachetta: Well, we were talking about this before, while you’re finding the slide. While we’ve done a lot of work with Turner Sports for the Ryder Cup. We’ve done some soccer championships with them where whole at home production. I think you and I were talking before the call that we have at home production where you’re producing at your home studio or at your home master control. But now we have talent working from home, talent, operators working from home. At VidOvation, we’ve been a champion of At-home production technology for quite some time. Some of the challenges from a transmission standpoint that we’ve solved is maintaining Gen-Lock, maintaining perfect Lip Sync with multiple cameras. Those are some of the infrastructure challenges we’ve overcome. In this scenario like our Live PD TV show, they want to remove bodies from the control room. They want the transmission engineer, the video engineer to work at home. They got 48 ISO camera feeds coming into master control, they dump it into an EVS system. How could TAG help? If we put probes on the IP streams coming into the master control, how can you help?
Jim Jachetta: How can you help monitor that in the studio? Or if an operator is working at home, how do they get those diagnostics at home?
Bil Apker: It’s interesting you mention that because it’s a twofold process when you start looking at that. Remote production has been talked about for what, five or six years now? Moving people around and having less walls in between everybody and less centralized people. Having people in the field, bringing back content to a central location is relatively straightforward. It’s what you guys do at Live PD all the time. That’s one aspect of it. In that aspect it’s very easy for us to gather that content whether it is in AWS, whether you’re bringing those IP signals back in house and then turning them onto glass so you can view them internally, is one side. The in home production that we’re all facing right now, is a totally different perspective. We do it very well because again, if those signals are all coming into a somewhat central location, because I think they still have to, most people are not doing at home production where they’re bringing those signals back into their house. They’re still bringing them into some central location.
Bil Apker: Well, the great thing is, is that because of what we do, we’re all IP in and out, we can produce mosaics for numerous different end users if you will. If a producer needs to see all these feeds coming in from different locations on their monitor, they can do that. But we can also produce a piece of that to go to a video wall, which we normally have inside of a control room. But even if that control room guy wasn’t sitting in a facility and he was sitting on his couch in his front room, we can also produce outputs to him as well. Again, it’s all IP based. Our mosaics output, we can either do MPEG-TS and uncompressed in studio, which is what normal people want to see, is an uncompressed on the screen. But that uncompressed signal actually has a copy. Every one of our outputs has a copy in HLS. That HLS copy can actually go to them at their house and then they can see exactly what you would see sitting in a control room at home.
Jim Jachetta: I have actually a very good question from Rajneesh. Sorry, Rajneesh. This is actually a great question. What if I don’t have my feeds already in IP? Now, I know Fox, you guys work with QVC, there’s many facilities now, ESPN. A lot of facilities are migrating if not already in all IP based master control. But what this gentleman is asking is, how do I convert HDMI, SDI to IP? Do I need a bunch of encoders? I guess the answer is, yes. That this technology makes sense if somebody is migrating to an IP workflow. They’d have to have a IP master control routing switcher. Maybe speak to that. Who is this technology ideally suited for and is it for everyone? If somebody is primarily in SDI workflow … Live PD as of a few weeks ago was an all SDI workflow. Now they’re scrambling to convert things to IP to have people working remote. Maybe speak to that. When is this appropriate, when does it work, when doesn’t it work? Maybe it’s not for everyone.
Bil Apker: Yeah. That really comes down to the fact that we acknowledge that we may be a little bit ahead of the curve. IP is still … In a lot of cases people think that its untested technology. We’ve been working with it for so long now, 12 years that we’ve been working in the IP side of things. But when you’re talking production, and specifically 2110 or 2022, people are adopting that as new facilities come on board or as they see the need for it. I see a lot more larger broadcasters really moving to it quickly because they can get signals from anywhere to anywhere very easily. More of them are pushing toward cloud based applications, which we work in very well. If you’re a TV station that has a bunch of SDI, you really have to weigh the pros and cons of adding exactly what you’re talking about, Jim, converters or gateways to be able to take an SDI signal into IP. But once you’re in IP, I think that’s where the power really comes in.
Bil Apker: Because now you’re using the full ability of the TAG solution and being able to deliver a multiviewer or probes from anywhere. It doesn’t matter where it’s from. That releases you from borders and walls and all this other great stuff that removing IP or removing SDI out of the equation really lends itself well to.
Jim Jachetta: Well, I think also there’s a lot more IP out there than people imagine. That if you are a user of just simply bonded cellular technology, that that is an IP based transport. The boxes you’re using now, the receiver box is probably spitting out SDI. If you’re using AVIWEST, VidOvation’s partner, it’s very easy to get licenses to spit out IP. I am sure other vendors are doing that as well, having IP outputs. Even if it’s just little islands of IP, if you’re not 100% all IP, TAG … I’m looking at bundling the TAG technology with the AVIWEST technology as a diagnostic tool. Here’s a common problem that we see. That the bonded cellular mounts on the camera. It goes between the battery and the camera. It’s on the camera. When the operator has to do a battery change, the operator will forget to turn the AVIWEST unit back on. The newer AVIWEST unit now has an auto transmit. When you reapply power, it’ll boot up automatically and start transmitting automatically.
Jim Jachetta: But if you have an older unit or if the operator turns it off accidentally … But a common problem will be, “Camera 37. Okay, it’s down.” They hear on the comms. Or camera 37 says, “Okay, I’m going to go change my battery.” On the comms. They’ll change it. Three minutes later, the camera’s not back up. “What’s going on?” “Oh sorry boss, I forgot to turn my camera on.” One of the features that I like about your multiviewer, is you have that penalty box that if a camera feed has something wrong with it or if it’s not there, that feed is moved to this area. A video engineer or some technical person, some transmission engineer could just be watching that penalty box. Then I guess, Bil, could we set a timer that if a feed stays in the penalty box more than two minutes to make something flash. That somebody needs to alert camera 37 that he didn’t turn the camera back on or, “Are you having problems getting another battery, 37? What’s up? Why are you not back on the air?” How would you help with that workflow problem?
Bil Apker: Yeah. It’s interesting because I’ve been in the multiviewer side of things for quite a few years. We’ve always had the big video walls with lots of pictures on it. Well, and especially when you’re getting into very large installations, penalty boxes really add a sense of urgency because it’s the one thing that flashes at people. It makes it perfect. You can set thresholds in our software for a certain time. Let’s say if it’s black for 30 seconds, don’t do anything. But if it’s black for over 30 seconds, start throwing stuff into what we call central air display or penalty box view. It really helps focus operator’s eyes over to that piece. For us, it’s not a new technology, in fact we’ve been doing it for many, many years. But it just gives that focus from the operator and takes his eyes from the big screen of pictures and moves it to something that’s more important in that moment. Yes, the central air display or penalty box views are crucial in any operation in my opinion, because there’s just too many things going on overall.
Jim Jachetta: Well, another problem I could see too is that, these at home productions, live sports, live reality TV. Early on, the Live PD folks were very clever. They were like, “What is live reality TV very much like?” It’s like live sports. They actually use EVS. They have quite a few EVS operators. They dump all the live video into an EVS instant replay system. If there is a problem with a camera, somebody unplugged some microphones, battery died on a Lav wireless mic, they may not see it for a few minutes or until it’s too late. It’s dumped. Now they don’t have audio on that video clip they want to use. Our system support four channels of audio. We could say, camera 37 has two audios, camera 38 has four audios. Knowing how many audios are supposed to be there, we could have alarms if certain audio is missing. Correct?
Bil Apker: Sure, absolutely.
Jim Jachetta: I could see that as a, “Hey, 37, what happened? Did you not plug the Lav in?” Or, “Can you please check the Lav on the talent or on the officer?” Is that stuff you could probe and look at?
Bil Apker: Absolutely. I mean, we have so many thresholds that we can set inside of it. I don’t know if you guys can see this picture.
Jim Jachetta: Yeah, that’s …
Bil Apker: But I can give you a little bit of a preview of kind of what we would see inside of the other software itself. It gives you a little bit of an idea as soon as it walks in. We have many thresholds. I mean, you can kind of see all the points that we cover in your audio [crosstalk 00:28:56]-
Jim Jachetta: Can you zoom in maybe a little bit? Let me … If I go full screen, I can kind of read it. That’s not bad.
Bil Apker: Can I make it bigger?
Jim Jachetta: I think you zoom your browser, hit the three little dots there on the side, I think … That’s not bad. Yeah.
Bil Apker: We have many thresholds in here about audio and video laws and we can set them to anything you really want. The great thing is, the notifications can be set as well. If you have a specific … Especially audio sometimes it can be critical depending on what channels you want to listen to. Mostly the stereo mix is probably channels one and two. You want those to be red. But you could also set channels three, four to be yellow if something failed in there. Or green or blue or pretty much any color you really want and kind of come down to that. Of course that will alarm. If it is set as something critical or the threshold that you determined it needs to be, again, that would throw it up in the penalty box view or central air display view as well.
Jim Jachetta: Well, see you in the case of Live PD too, one of these audio problems might not be catched during rehearsal. They do a rehearsal an hour or 90 minutes before they go live. All of this could be pre-programmed. I keep coming back to this fictitious camera number 37. But camera 37 comes up during rehearsal and three of the four audio channels that are supposed to be there are missing. During rehearsal is the time to sort this out, not while you’re on the live show. Again, it comes to that prediction. If a IP stream is starting to have problems or it’s falling apart, or if you’re … You mean our tools will show you packet loss. AVIWEST has a ARQ forward error correction. They use this Safe Streams Transport. But if, I don’t know, God forbid something was dropping frames, usually it’s something is misbehaving.
Jim Jachetta: Rebooting an encoder usually solves the problem. If something was stuttering, we were dropping frames, audio was garbled, you could set alarms for that as well? Then back to this camera 37, if 37 was dropping frames, the director could say, “Okay, 37, when you’re clear do a quick reboot.” You could probe those kinds of things?
Bil Apker: Absolutely, yeah. I mean, with 300 and almost 50 events that we monitor in any IP address, it’s pretty extensive. Including timing, including drop frames, we’ll look at that and be able to see what’s going on and alert the engineer in the end to maybe making a change.
Jim Jachetta: Okay. I don’t know if this is a good point to mention this. You guys kind of you’ve always had this philosophy, but you put a name to it, the Zer0 Friction thing. Not all companies operate the same. I’ll give you some examples with some other vendors. For example, AVIWEST, compared to some of their competitors, you buy a license, you own the license forever. You don’t get hit with renewing the license every year. Some vendors do that. That’s kind of a kind of an unethical revenue stream where they’re charging you every year for the licenses. Maybe speak to that, your Zer0 Friction philosophy. What’s that all about? I think you have a slide deck about that, right?
Bil Apker: Yeah, I have a little bit of a piece on it. About a nine months ago now, we took a look at the business and we started looking at what Tomer and Gal really had met when they started the business. They were two guys that started TAG and realized that IP video was going to be the future. In the beginning, they were very set because they were two guys and they didn’t want problems with the system itself. They didn’t have the support backing to do it. Now we have a lot more people for support these days because things are growing and things like that. But their whole philosophy was the fact that they wanted the ability to have a zero friction approach to television. That really is translated into what we call Zer0 Friction. The Zer0 Friction really is, multifunctional, multiapplication being compressed or uncompressed in the same solution, complete operational agility.
Bil Apker: I like to always say that it’s technology without borders or without walls anymore. A great example of this is, as we were talking earlier, let’s say you had a truck that was yacht set and is in its backs period. They’re testing out everything and the producer comes to them and says, “We need to monitor another 40 channels tomorrow.” In traditional ways, it’s another box, it’s another piece of hardware to be able to do that. But the great thing with TAG is, once you have what I like to call TAG currency, you can move your licenses from wherever they are in your facility. You have a truck set in Los Angeles and you have a truck on its way to Atlanta that won’t be set until Friday. Well, the perfect thing is, is you can take those licenses from that truck in Atlanta and move them to the truck in LA. Use them until Thursday when their shoot is over with and then move them back to the truck in Atlanta so that you can be ready for his next shoot.
Bil Apker: A completely flexible idea, a completely flexible for what the operation of that person is or that company is. It continues to be like that all the time. In fact, we move licenses for clients all the time, even on-prem when they want it from this … If it’s on-prem, they can move from here to there. It’s very easy for us to do that and it’s very easy for them to do that. A lot of our clients have always been doing that because they have different operational needs. Their setup changes on a relatively regular basis. It’s great to have that flexibility. What it also gives you is maximum utilization. Because a lot of people build for 100% threshold, they need this much stuff to do a little bit. But that this much stuff does once or twice a year. We’ve heard a lot of times that you only get about 30% utilization out of your system on any given day when it’s not all hands on deck. This really takes that from that 30% utilization and it makes it 100% utilization because now you’re moving licenses based upon your operational needs.
Jim Jachetta: Well, it’s interesting you say that because I see other analogies. Like you said, a truck is not being used, the studio is dark, a control room is dark, you can move the licenses around. Normally, a license is tied to the MAC address or the piece of hardware associated with it, if we’re talking about software. They’re not movable or you’d have to call to the vendor and unregister the license and then re-register it. It sounds like your process has a lot less friction, hence the Zer0 Friction. One of the things-
Bil Apker: A zero friction entity.
Jim Jachetta: Yeah. Well, one of the things I wrote about recently, I wrote a piece in Broadcast Engineering Extra before NAB, well before NAB was supposed to happen the pre-NAB issue. We talked about at home production, and it’s the same idea. That whether it’s people, personnel or a truck, if a truck is doing an afternoon hockey game in LA that truck probably cannot do any other events a few days before and a few days afterwards. It can do that one game on Saturday. If a truck is in New York doing an afternoon ball game in New York, that truck cannot be used for an evening ball game in LA. The truck physically can’t get there. If we do at home production where there is no truck or if we still use trucks, there’s an idea that NEP would just keep their trucks in a parking lot somewhere in a central location with a good internet connection. Or trucks would be parked somewhere regionally near different venues, and a truck can do more than one event a day. Wouldn’t that be more beneficial than NEP or Game Creek wouldn’t need as many trucks?
Bil Apker: Absolutely.
Jim Jachetta: Some of the truck manufacturers criticize me and say, “Hey Jim, aren’t you the guy with the no truck logo on your website?” I say, “Well, that’s just a marketing thing.” I think the truck will change. We’ll use smaller trucks. Now take it down to the personnel level, just like your software licenses, Bil, with the TAG software, you have your operators. Your highly skilled employee, your EVS operator has got to fly to LA to sit in a truck to do the hockey game. Wouldn’t it be nice if that EVS operator could stay at home or stay at a centralized master control and do a game in the morning, do a game in the afternoon, do a game in the evening? Where they could do two, three events in one day instead of two or three events in one week. To me, the most scary thing, I think, I’m not looking forward to getting in a plane anytime soon. I think that’s when catching a virus, getting sick in that recirculated air in that tin can.
Jim Jachetta: If I could work in a controlled environment where I sleep at home and I go to my regular workplace where there’s cleaning crews keeping everything clean, minimal exposure to other people … Or take it a step further, I’m working at home now. I think this all fits into the same ecosystem. I mean, do you want to add to that? It’s not only the license, it’s the truck, it’s the assets, it’s the software asset, the personnel asset.
Bil Apker: It’s interesting you bring that up because it’s one thing. You may recall this from recent years. I was explaining this to somebody the other day that, it’s funny that for broadcast television and television in general and remote production in general, that we still send trucks everywhere and we still send crews everywhere. In the movie professional business, the one thing that, I don’t know if you know this, that they did numerous years ago was they realized that colorists are expensive. Those colorists usually typically had to go to a facility and sit in a big room with a big screen and then do the colors where they were on that area. Well, I had heard a long time ago and I’ve worked with a lot of the studios over the year, both in LA and in San Francisco. One thing they do is, “Well, we have a colorist in Los Angeles and we can make that person stay in that seat all day long. He can color for a show here in LA or in the Valley or in San Francisco from his one location.”
Bil Apker: They basically already centralized the talent and brought the work to him so that he could do exactly what you’re talking about. I see that we’re going to move quickly toward that or we’re moving quickly toward that because now, I’ve been seeing it all over LinkedIn and all over the places that I monitor, that more and more of those people are moving audio to their homes. Or moving production studios to their home, or at least production control services to their home. Editing is really easy. I mean, I think computers are powerful, more powerful enough now that you can do editing from pretty much anywhere. But some of those big pieces, it makes a lot of sense. As a technical director, which I was one years and years and years ago, the one bad part about it is, I could never bring all those signals home. It’s very difficult to do that. Control service you can control remotely across an IP link. You don’t need to have those coming into the central control service because now controls services are flexible.
Bil Apker: But how do you get the video back? That’s where we really step in and to go, whether you’re sitting in front of a piece of glass in a dark room in a television station in front of a production switcher, or you’re sitting at home, we can deliver that same type of experience to you. Which eases those restrictions. That technical director that’s now sitting there doing four shows a day in the morning and then four shows a night for the other guy, they could do multiple shows all day long. If you’re centralizing everything, that means they’re just pulling feeds from different television stations. You can see exactly what you would see if you were sitting in the control room at home.
Jim Jachetta: Right. Well, what also makes … I mean, you guys help with the diagnostics, the multi viewing. VidOvation, a big part of what we do is the infrastructure. The bonded cellular technology I mentioned earlier, there’s usually about a second latency as kind of the sweet spot. Theoretically, we can go down to about a half a second, but then you need a more reliable connection. But in the film industry, we’ve done some projects where exactly like you’re talking about, that the director is at his house in Malibu. The colorist is at his facility in Hollywood, but the director wants to watch what the colorist is doing. I guess some directors are micromanagers, some are not. Or the director and the director of photography all need to both eavesdrop or give the colorist notes. We need extremely low latency but also high fidelity connections. We partner with another company called V-Nova, which they have technology that’s contrarian, the JPEG 2000. A lot of the common transport for the application you’re talking about, Bil, would be like a J2K, a high bit rate transport, lightly compressed.
Jim Jachetta: I mean, unless you had fiber optic connections between the director’s Malibu house and the colorist facility, doing uncompressed I don’t know if it’s practical. But these V-Nova guys, they just got a SMPTE, 2117 is the new standard. It’s near the quality of uncompressed, but it’s 25 to 75% more efficient than JPEG 2000. I see V-Nova integrating their transport with TAG. That’s why I think VidOvation partnering with TAG … In case, you folks can’t tell, Bil’s my new best friend now. That I could see bringing TAG and Bil, TAG Video Systems into many of our projects and integrating it with some of the IP transport we’ve got. But you agree that that low latency … You talk about that a lot too about latency, low latency applications. Maybe you want to speak to that?
Bil Apker: Sure. I mean, we do support JPEG 2000 both in and out MPEG transport stream, but those are definitely compressed formats. I think when we start talking, one of the big hurdles in uncompressed has always been latency. Typical broadcasters or typically production studios want the same experience they would experience with a SDI format. In multiviewers in SDI format, we were down to maybe a couple of frames, I think was what we typically were at. I’ve worked for other multiviewer companies in the past. We got it down in SDI through SDI video specific hardware to a couple of frames. We’ve just recently at TAG, number one through software, increased the capacity of the servers. If you had a server from a year ago that you were using 2110 with, we actually in software because of what we do, the magic they do behind the scenes … I don’t know what it all is because I try to stay away from it. We have incredible programmers and incredible guys that write stuff. But they were able to increase the density of the server that you had a year ago.
Bil Apker: Now your ROI is significant. I think we’re somewhere in the 40s. I can’t remember specifically where it is. But that was one of the things we did through the manipulation of drivers and operating systems and our own software. Then the one thing we did, we took our delayer or our processing delay through the server. Now this is a completely COTS server. There is no video specific hardware in it. It’s-
Jim Jachetta: There’s no GPU.
Bil Apker: No GPU, there’s no video specific car that’s processing video. This is strictly memory, CPU, power, a motherboard and a pretty metal box around it. And a big bandwidth network card that brings it all in, 100 gig network card. But we actually brought the processing delay down from two frames to one. Now that competes with every SDI multiviewer on the market as far as I can see.
Jim Jachetta: You’re saying a traditional SDI multiviewer, because of all the processing, the GPU has got to turn that into a visual representation to go up on the screen, it’s typical to have two or three frames compared to the one frame?
Bil Apker: One to two frames in SDI, yeah.
Jim Jachetta: One to two frames. I guess that’s a common question. If we go all IP, is that going to add the latency? You’re either on par or you’re better than the SDI solution. Is that what you’re saying?
Bil Apker: We’re at least on par, I wouldn’t say we’re better. But we definitely … I think we’re the only ones in the industry that can do that and in a mixed environment. 2110, 2110 out or 2022-7, 2022-7 out, or 2110 out, we have about a frame processing delay through the system. Plus, I believe we’re the only ones in the market that can do MPEG-TS and HLS and JPEG 2000 all in that same mosaic. I don’t think there is anybody else that does that today in peer software on a dongle the size of my thumb.
Jim Jachetta: Yeah. You’re founder there, Abe, likes to show a little … It looks like the little wireless keyboard, little USB, little nub you hook to your computer. He was like, “This is all you need. This is all you need. You need this little nub.”
Bil Apker: Yeah, this actually the software-
Jim Jachetta: Yeah, there it is.
Bil Apker: … [crosstalk] all right there. It’s software, it’s the OS, it’s all your licenses. It’s one of the cool things when people talk about redundancy with our solution. Let’s say for some reason the server failed. Now in a traditional way was you buy double of everything, you can do an easy switch over. Which we’ve done that as well. But I’m like, “If you have a server in standby next to this one, it’s on the same network. You pull this dongle out and you put it in the other server and you hit the power button, you’re up and going within five minutes.” It’s just that good.
Jim Jachetta: Wow. All the configurations travel with the dongle. You don’t have to take three hours to restore all your settings. Everything just migrates over.
Bil Apker: Everything migrates over very easily. In the year that I’ve been here, I’ve only heard of a one dongle having an issue out of the, I don’t know how many thousands of dongles we have out in the field today. But that was easily restored by our remote tech support team. They reflashed the dongle, they got it up and going, and they were very happy. It was a customer in Brazil. It’s very resilient. It’s easily backed up. We have lots of backups just in case you need to restore something. I know that my demo system in the back, I let a customer borrow my dongle because they needed it for some testing that they were doing. I said, “That’s fine.” When they brought it back, obviously all my configurations were gone, everything was [crosstalk 00:51:49]-
Jim Jachetta: Be careful what you lend people. They messed up all your settings?
Bil Apker: I know. Which is totally okay. I completely expected that to happen. But because I already had my image saved, I was able to restore it within five minutes and have it up and running just like you see here. It’s one of the easiest products that I’ve ever seen to support. Because there’re so many backups to it, there’s so much redundancy to it. It’s very simple when you talk about … Because it lives on a dongle on-prem, it makes it very easy to use. Because we-
Jim Jachetta: You mentioned COTS hardware. TAG and VidOvation will help the customer figure out how much processing power we need. If you have hundreds of IP feeds, you might need several servers. Then we deliver the video to an a 4K display using either optically or IP to a little set top box with 4K HDMI out. Do you have some topology? Maybe show us some of the COTS hardware we would need to set up a system. Do you have a slide that shows some of that stuff? We’re coming up on an hour. I mean, Bil and I can keep going all day here. Brevity is not usually my strong suit. But yeah, if you have a block diagram of what a system looks like.
Bil Apker: Yeah. This is a slide that we use a lot because it’s … I mean, one of the things is that this shows two things. Number one that, we have all these signals coming in from the left here. It doesn’t matter what they are, they’re coming into the central IP switch, and then they’re going out to the MCM. Which is taking them and processing them and putting them in pretty mosaics. The great thing is, if you have a small operation today, let’s say you have 10 channels and you just want to throw those up on a multiviewer and be able to probe them, that’s probably going to be a single server. Even if it’s maybe 50 channels, it could be a single server. But let’s say tomorrow you have a requirement that you need to add 1,000 channels or more to your system, well, that’s very simple for us. You just add a few more hardware pieces to it. You add a few more licenses and then you’re outputting all those signals into one.
Bil Apker: With this, we support SDI over IP uncompressed in items like we see here, the [inaudible 00:54:44], the AJA, [inaudible 00:54:46]. We also work with … You can also push their multicast through an STB. But again, because each one of these can be separate mosaics, you can also have a copy. Every one of these outputs is also an HLS copy so that you can view it on a mobile device, on your computer, somewhere else in your facility at home. That’s what makes it very flexible for what you’re doing when you’re operations.
Jim Jachetta: Very good. Someone’s asking kind of an obvious question, but I think it’s a good one. You mentioned COTS hardware, but does it work in the cloud? I mean, we kind of alluded to that, the transmission engineer working at home. I guess there could be physical servers in the master control feeding IP out to a computer at home or in a secondary location. How would we work in the cloud?
Bil Apker: The great thing is, is that over the last nine months that we’ve been heavily involved with AWS in fact, the super bowl this year was all brought in back through AWS to monitor in LA. We were pushing those mosaics out from AWS onto glass inside of a control room in Los Angeles. Again, it’s mostly compressed in AWS because there’s not really a pipe big enough right now to take uncompressed. But when you start talking about that the fact that we can live in AWS, it’s the same software. The software does not change across the board when you talk about putting it in an AWS or putting it on a VM or putting it on-prem. It’s exactly the same software across the board. You get the same probing capabilities, you get the same multiviewer output capabilities. It really is depending on what your operation is. Because what we have found is that, 24/7 AWS usage is very expensive. That’s where we start looking at the differences between, do you want to leave everything in your cloud and monitor it all in your cloud?
Bil Apker: Or do you want to bring it back on-prem and monitor it locally? Which can be a little bit more cost effective than what AWS have. Again-
Jim Jachetta: Well, you mentioned VM too. A lot (silence)-
Jim Jachetta: … the resources, storage resources. This can go in a physical appliance. Usually the physical appliance, your preferred brand is Dell. VidOvation can help you procure that hardware, the Dell hardware (silence)-
Jim Jachetta: … points. The Arista is (silence)-
Jim Jachetta: … that support that, set that all up. On-prem in a Dell in your virtual server, in your VM, you support that as well?
Bil Apker: Mm-hmm (affirmative).
Jim Jachetta: Now, we’re in a VM environment. How do you use the dongle or you handle the licenses a little different if it’s a VM situation?
Bil Apker: There’s still typically a dongle in the machine, but I mean it’s pretty much open because you can have multiple instance of it. It’s just actually pulling the licenses from the dongle at that point in time. But those licenses are completely flexible. Like we talked about before, if you had licenses from your on-prem servers, but now you want to light up some from the cloud, that’s very easy. You’re just moving those licenses out to an AMI that’s on the cloud and you can continue to use them. Again, completely flexible, borderless use of your solution.
Jim Jachetta: Okay. Someone’s asking a great question, Bil. What are the costs? I know you have some published costs that I think … Do you feel it’s appropriate to share your pricing model at this point?
Bil Apker: I can share generally what our pricing model is. I think that that’s something we should have a conversation with because it does change based upon how many licenses you buy. But we’ve taken a big step back with Zer0 Friction and actually leveled the playing field between 2110 and compressed. If you buy a 2110 or if you buy a compressed, it’s exactly the same price. That price goes down the more you buy. It’s kind of a great philosophy for us. We also have the ability to redo recordings and the encryption and use the ABR stack. Those are all different prices based upon the primary price as well. It’s very easy for us to design a system because the hardware has to go along with that. Or the AWS instead has to go along with that so you can have a very flexible way of working.
Jim Jachetta: Okay. Yeah. Actually, we encourage you at the end here … We’re coming up on the hour, Bil. Maybe this is a good point to kind of wind things down. Let me show you guys. Reach out to VidOvation. Let me get my screen back. Bum, bum, bum, bum, bum, bum. Let me see. Do I have to make myself the presenter again? I think I do. Make presenter. Come on. There we go. Yeah. You can see here, VidOvation. I’m hesitant … What they say in business, as we get older in age, Bil, we’re not supposed to reveal our age. But you can see I’ve skipped from going gray to just going white. I guess I still have my hair is a good thing. I’ve been doing this for the better part of 35 years. I started my career working for my dad at MultiDyne. My brother now runs MultiDyne, the fiber optic company. VidOvation is in its 10th year. We’ve done some really exciting projects in the 10 years time with the NHL, Madison Square Garden network, MSG network, NFL, Viacom. You can see some of our customers here.
Jim Jachetta: We turnkey the whole process. Upfront, we’re doing consulting, we’re helping you pick the right technology for your projects. We’re going to be plugging TAG technology now into a lot of our existing projects. We’re certainly going to bring it in to new projects that we’re working on. We’ll consult, we’ll help you design the system. We’ll help integrate the technology and then project management, warranty and support. We handle the whole project from end to end. VidOvation, we love wowing our customers with excellent support. 24/7 support for all of our customers. Whether it’s hockey, Live PD, IP TV, at Nickelodeon or Viacom or Paramount studios. We’re here to support you. Here’s some of the other projects we’ve done. But here’s our main contact information. You call the main number, (949) 777-5435. I think option two brings you to sales. So one of our sales engineers will help you. You have my contact information here, firstname.lastname@example.org. That’s my direct email. You can connect with us on social media.
Jim Jachetta: You’ll work with both companies obviously. VidOvation will start the process. We’ll bring Bil and his team in. We’ll work with the TAG engineering team to make the suggested recommendations. How many servers do you need? What horsepower server do you need? You can of course procure the server yourself. VidOvation, we feel it’s better to turnkey the whole design. Get the server, get the software, get the hardware. This way, if something doesn’t work right, it’s one phone call. The Dell server one morning doesn’t boot up, you call VidOvation because we sold it to you. We’ll support it, we’ll take care of it. If you folks have any questions … I think we’ve answered most of them. It looks like we got all of them, Bil. We are recording the session. We will put it up online in the next couple of days. Probably early next week you’ll see the recording up online. Bil also has a video, a new video they’re releasing about the Zer0 Friction.
Jim Jachetta: We can put some of that information up online. Reach out to VidOvation if you have a specific need or application, and we can arrange a demo and get you a price quote. Anything else you want to add, Bil?
Bil Apker: Yeah. Just the one last thing. I mean, we love the try before you buy scenario. If anybody wants to try the software in their facility, we’re happy to get a dongle out so you can start trying it or even a trial on AWS.
Jim Jachetta: Awesome. Well, thank you so much, Bil. Thanks for joining. Everyone out there, thank you for joining today. I hope you all stay safe and healthy, and look for more information online. Look for the recording online. Thank you folks. Thanks, Bil.
Bil Apker: Thanks so much.
Jim Jachetta: Take care.
Bil Apker: Bye-bye.