Select Page

5 Ways To Protect Your Live Productions Against Video Transport Issues [Webinar Recording]

Published on Nov 10, 2023 | Past Events, Featured, Webinars

Webinar Preview Video

Summary

Adi Rozenberg discusses the need for network observability in video over IP workflows. He explains that as the industry moves towards IP-based workflows, broadcast engineers face new challenges and responsibilities related to network management. However, current protocols like SRT, RIST, NDI, etc., are reactive and focus on individual streams rather than providing a holistic view of the network. Rozenberg emphasizes the importance of identifying and preventing network problems and treating IP networks as evolving entities that require constant monitoring and adjustment. He also mentions that network observability is not only beneficial for broadcasters but also for service providers who can proactively manage their networks and ensure the smooth delivery of services.

AlvaLinks Webinar hosted by VidOvation

Jim Jachetta of VidOvation speaks with Adi Rozenberg, CTO of AlvaLinks, makers of the first true network observability platform designed for media, about how you can gain peace of mind and eliminate video downtime with trusted, real-time, accurate network intelligence.

You’ll discover how to:

  • Ensure quality of service for your most intricate and complex traffic flows so you can gain trust in your network.
  • Monitor network events, latency, packet loss, and jitter for precise, real-time delivery with millisecond accuracy.
  • Enhance operational efficiency by tracking data as it moves to and from different sources, including cloud platforms, data centers, camera feeds, multicast services, and standard internet access points.
  • Fine-tune and optimize your network to increase accuracy and consistency throughout each phase of operation.
  • Use advanced methods to test and identify dynamic network behavior – before you go live and during live streaming.

Adi Rozenberg Bio

Adi Rozenberg is an innovator, visionary, and leader with over 27 years of experience in various technological industries such as broadcast video, networking, mobile, video walls, and more. Adi started a new company, AlvaLinks, specializing in sophisticated Video Transport to the cloud using AI to achieve the best performance and stability.

In 2010, Adi co-founded Video-Flow and wrote patents for delivering live video over the public internet. As CTO, Adi led the R&D team and support teams. He led technology development, patents, and the adaptation to a new video transport standard.

He actively wrote the TR-06 Reliable Internet Stream Transport (RIST) open specification and served as a director of the RIST forum. Adi’s expertise in broadcast video and networking positions gave him the option to form a unique R&D team that ties both video and networking experts into a united team.

Transcript:

Adi Rozenberg:

Just wondering, do you see?

Jim Jachetta:

There we go.

Adi Rozenberg:

Do you see my screen?

Jim Jachetta:

Yes. I really like the slide, Adi, where you show broadcast workflow cameras on the left, the venue, and then you got your circuit in between. There’s an amorphous cloud, whether that’s public internet or someone’s managed network, and then the studio on the other side. And then you overlay this disgusting mess of different paths and routes. I think everyone visualizes a relatively straight connection or repeat.

Adi Rozenberg:

That’s exactly what they do. Exactly right.

Jim Jachetta:

And then one of those segments that is highlight… Everything you’re doing you could uncover with a trace or the other tools that you mentioned, but you don’t get a graphical picture of what’s going on. That’s the big thing, right?

Adi Rozenberg:

No, the big thing is that you can. Data are tools that can give you that graphical representation, but then you will need a myriad of tools in order to make sense of the data. You’ll basically spend hours, days on different tools in order to synchronize the events and that is the maple. Basically, we’re saving you time. We are running all the tests all the time, collecting the data, putting them on a single pane so that you have the information, we can then correlate those events. Now, many people, what they will do is that they will have an alarm, like a problem happening on the multi-viewer, on the ETR 290 or bridge technologies or whatever, and then they will go and do the trace routes. Then they will start looking for the problem. Now, if it’s a transient problem, it’s already faded, it went away. Only if it’s a consistent continuous problem, then you will see it, but we need to know what happened before the alarm. So if the alarm hit at 2:00 AM, I want to see what happened 15 minutes before that.

I want to know what happened 30 minutes before that. Maybe something caused that. Maybe the network change before that, that increased my latency, increased my jitter, something happened. I need to be on top of that. We need to understand the causation of the problem that caused that alarm. Of course, we need to see the event that caused the alarm also, but what created that. You guys live in LA so you know what traffic jams are, right?

Jim Jachetta:

Yes.

Adi Rozenberg:

When people go and go on the highways in the lane in the morning, I know that they expect it to have a traffic jam, but that shouldn’t be the norm. The reason why there’s a traffic jam, because all the people jump almost at the same time go on the freeway to get to their destination. If there is a traffic accident, that will cause slowdown. You need to be aware of that. You need to be on top of those things. Specifically when you’re doing trafficking, video trafficking in that sense, delivery network, you need to be on top of that. If you have multiple diverse paths, you need to be on top of the latency of each path, whether it’s straight, is it changing? You need to be on top of that.

Nowadays, people don’t have that view. In fact, yesterday I spoke with the major service provider. It’s a well-known organization, I will not name them. And he said, look, I have diversity and if I have a problem and my alarms go wild, I’m picking up the phone and I’m calling my service provider, but what information do you give him? I just say, we have problems. How long does it take you to solve the problems? Say he tells me, oh, it’s not my problem, it’s his. But in fact I told him, but it’s your problem. You are providing a service. You are paid to deliver the service and you are either wasting time and money and you’re not providing the service. So we’re basically reliant on your service providers to solve the problem that you basically contract to do.

And then it hit him. It hit him that he needs to be on top of that so he can tell his service provider, “Hey guys, your latency is not as promised. You have more packages than the survey you provided. Maybe your bandwidth is not accurate, and that causes me to lose money and not to deliver the service that I was contracted to deliver.” That’s what hits him at that time. People don’t see that. Humans tend to think about, I have a problem, I will try to solve it.

Instead of getting a flu vaccination, I will wait. I will hope to ride it off and hopefully I will not get the flu. Okay, that’s a human nature, but we are in a business that we need to provide service and live services is the predominant money generator in our industry. That is why ESPN, NFL, UFC are paying our top dollar for those services. People want to watch those events. Those are premium commercials that we distribute. You pay more money to get live content. We cannot mess those services up. That’s why we have not one diverse path. Usually we’ll have four diverse path. We’ll have a redundant equipment and we need to be on top of that before the game, during the game, we need to be on top of that, so then if something happens we can recover. Sorry for preaching. I’m just-

Jim Jachetta:

No, no, no, this is good. Well, you’re going to repeat all that in a half an hour.

Adi Rozenberg:

If I remember this. The B12 deficiency, so…

Fallon:

Can I interrupt. I want to mention, I just wanted to mention we have a few attendees on and we’ll get started at the top of the hour.

Jim Jachetta:

Oh, okay. Okay. Should I go back into practice mode, Fallon?

Fallon:

If you’re able to. I think that’s a good idea.

Jim Jachetta:

So I just end it and then it’ll drop back to practice?

Fallon:

Yeah, you may need to rejoin links. It might kick you off. Let’s see what happens. But everyone that’s on right now, we’re starting at the top of the hour. We’re just doing a tech check run through for cameras and audio and all that good stuff.

Jim Jachetta:

Yeah, so maybe we better be careful what we say Adi, we don’t want to mention customer names. All right, here I’ll speak in generalities. So I talked to a customer and he says, I’m using service provider X and I say, you’re only using one provider. Do you have redundancy? No, just one provider. And I said, well, you must have perfect service. You never drop a packet. Everything must be perfect. We’ve been really happy with their service. It’s a 100% reliable. What would you say to someone like that? You’ll call me when it drops out or we were talking about that the other day. People don’t need a tool like this until they have a problem or they think they don’t need a tool like this until they have a problem. What would you say?

Adi Rozenberg:

I would say to them, you do need the tool because you need to understand your network. You need to be on top of your network because the network is a living thing. People think that the internet or they call it the dirty internet. Oh, I can tell you that I’m in touch with service providers that want to include our solution because they want to be on top of their services. As we see more and more traffic going under any type of circuit, more and more users, more and more application, that means collisions do happen. Even though you think you have on a private link, whether it’s NPLS or private circuits, you are still in contentions with other services on routers and other devices or even inside your own organization.

Try to remember the last time the IT people did a backup and that damaged your service. How long did it take you to understand that? Did you go out of service, try to remember the last time you had a connectivity problem? How long did it take you to find the root cause of that? Did you call the vendor? Was the vendor available to give you an answer in four hours or did he come back in four days? Did you lose money during that time? That is very simple. Many people that jumped on the IP wagon don’t see and understand that the IP is just like I would call it the big ocean and you need to navigate on that big ocean. And as many more and more ships go in that big ocean, then you need to have traffic management. You need to somehow manage that. You need to be on top of your game. We are not anymore in the old point to point ASI satellite, DS3-

Jim Jachetta:

You can go in a data center and you can go behind the rack and trace, trace the SDI cable A to B. Can’t do that anymore.

Adi Rozenberg:

Exactly. The IP people don’t understand. It’s based on packet switching, meaning that profit packets coming in and out. And I don’t want to scare people going against IP. I’m a big believer in IP and I think that IP is one of the biggest revolution, sorry, in our industry or the entire world, that might be even similar to the invention of flight or the wheel. Because as mediums, and now I’m putting my visionary hat, as we can see that more and more mediums will adopt IP, that means we’ll be able to use more and more traffic and we can provide more and more information. IOT will come to age autonomous cars, a lot of services. Connectivity will be everywhere, but that means a lot of traffic. And that traffic, we need to be on top of that so that there will be tools and means to safeguard the delivery of the high B trade or premium content that we want to deliver.

Jim Jachetta:

I’m glad we’re recording. It’s funny, one time I did a webinar and we had this 30 minutes of banter. Some listeners that logged in, I edited this part out. They went back to the recording, the part before the webinar was the most interesting part than the actual webinar. I say, oh, I edited that part out, so I’m going to keep this part. So I’ll tell you two things. A funny story. So we’re rolling out private 5G for venues, public venues, big spaces. It’s more reliable than wifi. You own a piece of spectrum to solve some of these problems. But I’ll tell you a story how I’ve done this several times. So it’s our own fault.

Our lab, we QC and test streaming equipment, bonded cellular before we ship it. And if there’s a big event, we might have 50 even a hundred units on at the same time in our lab. And not to use a lot of cellular data, we hook them to the network to test them. Well, on a busy workday and the lab is not in its own VLAN, we lit up like a hundred units at five megabits per second each all trying to go out to the public internet. Everyone says, “Hey, the phones all stopped working. What’s happening?”

So I’m like, what are the guys doing in the lab right now? And it like, okay, I think the guys in the lab either need their own internet connection or they definitely need to be on their own VLAN or need to set some quality of service that they don’t take the whole internet pipe. But it’s stupid things like that, right?

Adi Rozenberg:

Exactly. The problem is that those things will happen. It’s a norm. That’s why we don’t want IT people to sit down, have a cup of Joe and be happy. We need them to be on their toes, make sure that everything works smooth. It’s not only the broadcast people that needs to worry about pixels, everybody needs to worry about the workflow. I know that we want everybody to be happy, but at the end of the day, the organizations that we are servicing are there to make money, and money comes from a smooth production and people enjoying their NFL game or their soccer or whatever the Messi goal or whatever they want to see. So we need to give them that service and not to be content to say, oh, I have diversity and I will zap between one to the other because every zap creates a disturbance in the force in that sense. We don’t want that.

But going back to your question, people that already made the switch, they understand that from time to time they have problems. They do understand that every time they bring a new service, that service might impact other services running in that organization. Sometimes they need to configure the service, whether it’s with their Q based protocols like SRT, Zig Z or even NDI today. They need to know to be on top of what is the best configuration to do that. Now the network might change, so they need to adapt their configuration to the network changes. And that can change. Every time you bring in a new equipment to change your router, you activate your redundancy. You need to be on top of that. I’m not here to solve the problem. Not yet.

Jim Jachetta:

You’re working on that.

Adi Rozenberg:

I’m working with the company. Let’s say that working on something that will make it much more robust, but it’s still a work in progress. But the idea is that that equilibrium, that with network, with IT, with trafficking equilibrium will change every time you add a new, just adding a new car, additional car to a highway, you get to a point that suddenly there’s a traffic jam.

Jim Jachetta:

So we’ll kind of repeat what we’re doing again. I’ll ask you these questions.

Adi Rozenberg:

Sure.

Jim Jachetta:

No, I think it’s human nature too. We’re not able to correct problems in ourselves or in anything when we’re not aware. So personally, you grow by self-examination. Your tool could just simply be used to evaluate vendor A and vendor B for your IP circuit. If you see repeated dropouts every couple of hours on circuit A, you need to help them troubleshoot. If they’re not willing to fix it, then you don’t use vendor A anymore. You use vendor B and go find a vendor C that the… I know you’re a firm believer of redundancy and transmission. We’ll get into that too. Well, you’ll introduce yourself, your little bio. You talk about Wrist. So which one of you three guys, it’s a collaborative process. It was you, Cyro and Ackerman’s that invented Wrist or I think a lot of the patents, the intellectual property actually came from you and VideoFlow, correct?

Adi Rozenberg:

Yes. To be very modest, most of the Wrist specification are based on the work that I carried out at VideoFlow. The offshoot is the Wrist advanced profile, which has no previous IP behind it, but I was the one that wrote the initial specification as we call it in the activity group bucket list. But otherwise, most of the technology capabilities features are coming from the work and field deployments that I did in my days in VideoFlow. And most of the patented work or the IPR as we call it, is signed to VideoFlow. VideoFlow was generous to provide that as a round Z free of use for the members that do need to register their usage with VideoFlow. Many forget to do that or don’t understand, that they might be subject to a lawsuit of not doing that.

Jim Jachetta:

You just simply identify that you’re using the technology that-

Adi Rozenberg:

All you need to do is just send an email to VideoFlow and say, “I am using Wrist. I just want your approval, and VideoFlow is obliged to give you that right.

Jim Jachetta:

Even if VideoFlow thinks vendor X is a competitor to VideoFlow, it’s-

Adi Rozenberg:

No problem.

Jim Jachetta:

… no problem.

Adi Rozenberg:

That was done with Zig. One of the things that came from the work with Wrist was that everybody were frenemies. Zig Z, High Vision, Cypradius or DVO, that had their own patents, Net Insight. Everybody came together to find the best protocol that suits them as the organization based to their customer, and we shared knowledge. That’s the idea.

Jim Jachetta:

Has High Vision rolled out Wrist? They seem pretty set on SRT or they’re working on it. They’re going to eventually do a Wrist solution?

Adi Rozenberg:

I cannot talk on behalf of-

Jim Jachetta:

Well, I shouldn’t either. We’re a High Vision partner, but yeah.

Adi Rozenberg:

But I’m in touch with the high vision folks.

Jim Jachetta:

Yeah, so Wrist is a sym ratified standard. SRT is what we call in the industry a defacto standard, right? A vendor came up with a protocol and decided to open source it like a standard, but it hasn’t been ratified by a third party organization. Same thing with NDI. NDI would be classified the same, right? It’s not a standards body approved standard. It’s a defacto vendor standard that has been open sourced.

Adi Rozenberg:

But I also just want to let you know, I was the one that called for SRT 2.0. I went on record saying that we need to work on SRT 2.0. I think it was in 2018 already when I said that we need to change some of the basis of the SRT to make it much more usable. There are many features that are now available in SRT that were missing that I was a very big talker pointing out what is missing. People still don’t understand what they need to do with SRT and how they can best use SRT. And in fact, I approach a high vision folks, we’ll not name them because we are on record, I don’t want to expose them without them being here. Is that I asked them, let’s go back to the VSF. Let’s create an activity group to see what we can do with the SRT and how to push it forward in the industry.

What do we need to change in the SRT to make it much more… Although it is acceptable, I think there are many limitations that people don’t look at and that basically might harm their operation. Many of our clients are using SRT, but they don’t know the limitation, what they need to configure, what they need to look when they’re configuring SRT. All they think is that oh, SRT is open source, just like open SSH, it’ll work for me. And we need to have technical recommendations by the VSF to basically say to people, okay, these are the scenarios, these are the deployments, and this is what we recommend as a technical recommendation of what you need to do so that you will have a continuous service. One of my last presentation of the VSF talked about the combination of Wrist and SRT, so that we’ll enhance the capability of SRT that is missing today, for example, to do bonding using Wrist and SRT.

Jim Jachetta:

I believe two NABs ago they came out with, I don’t know if it’s a bonding of SRT, it’s a failover. I think they have a failover feature now.

Adi Rozenberg:

They came with a feature that you can do either failover or you can do a seamless switching. You basically create a group and but you have to do send two copies or multiple copies. It’s not-

Jim Jachetta:

And you pick the one that’s working at the moment. Hot switch, like a routing switch. So you have two of the same camera feeds coming in. One drops your hot switch to the good one. So you use twice as much bandwidth. That gives you a 100% redundancy.

Adi Rozenberg:

But my idea is that to do it like bonding, like the seller bonding is that we will take 50% or 30% on one link, 70% on the other link, or we can dynamically change that. And we do that by using Wrist, that was one of the features that came with Wrist since 2018. The dynamic nature of Wrist that was missing in the underlying protocol of the SRT. Okay, so that is something that is still missing today. That is a feature that many folks in the industry don’t understand why do they need that. Because they think one dimension, I have one link, I will send my… I have a uncast, I’ll use only one. Or maybe I will have a backup, but they don’t understand that they will have a glitch switching. So those things that are available in Wrist are not available in SRT, and I think we need to bring that to SRT as well. Now, I don’t have any dog in the race, as they say. I’m not affiliated with the protocol company anymore, so I can be visionary as much as I want.

Jim Jachetta:

Yeah, your IP you’re codec and transport agnostic.

Adi Rozenberg:

Exactly.

Jim Jachetta:

Now, just out of curiosity, would High Vision lose some of its autonomy if they partnered with the video services forum to help? I see the video services forum, it’s a precursor to building an actual standard, a certified standard. Or not necessarily video services forum, it was just an investigative panel would be put together to make the protocol better without-

Adi Rozenberg:

You don’t have to do that. Basically, what the BSF does is basically issues technical recommendations. Some of the technical recommendations are in fact specifications that you should follow in order to have interoperability or have a better service.

Jim Jachetta:

Yeah, so when they’re done, they would issue like a TR 101, SRT recommendation or here’s how to use SRT to its fullest potential.

Adi Rozenberg:

Exactly.

Jim Jachetta:

And it would be a user guide, a roadmap of how to have success. So it would be win-win. High Vision would get all this free feedback from the community. It’s a great way to solicit feedback from the engineering community.

Adi Rozenberg:

Yeah, exactly. You get immediate feedback from expert in the industry that wants to basically have better products, better services, everything. Now, just for you to know from the history, when the Wrist, actually the Risk Activity group was started, iVision was one of the founding members, and they proposed to use SRT. I don’t want to go why it was rejected, this is not the place to explain that. But many vendors did come forward to propose their own solution. The Risk Activity Group selected to go with Standard IATF RFCs based on RTP, which was the norm at the time with encoders, decoders that were able to do RTP. And they saw that it’s easier to migrate those devices or continue to use the RTP and not to change the code entirely, that was the reasoning behind it. And by miracle, my protocol and my suggestion were fully based on RTT and so on standard.

Jim Jachetta:

So it worked out.

Adi Rozenberg:

Yeah.

Jim Jachetta:

Yeah. I do believe Wrist is on the roadmap for High Vision. So High Vision, it’s almost 18 months, maybe almost two years now that High Vision bought the Avie West French Bonded Cellular company, and the product was a great product. Before the acquisition, Avie West had done an amazing SRT integration. I think that’s how the engineering groups got to know each other. So the SRT group in Canada work with the engineers implementing SRT in Rennes, France, and they’re like, “Hey, we both sort of speak French. The French-French, Canadian-French, it’s funny. It’s the opposite of what I would’ve thought.

The French speakers from France say the Canadian speak too fast. They’re like New York French people. You would think that at the source, at the motherland, they would speak more fluently faster. So they’re always slow down, slow down. I don’t understand your Canadian French. Or Jeremy and some of the guys at High Vision said that the French will actually use a larger word, a more Latin-based word for something. Whereas the French-French, the French from France use a slang or a shorter word or a more modern word. I guess the Canadian French is older.

So I think they just like working together. Avie West did an amazing NDI integration. Somebody’s got NDI cameras coming in or going out, or you want to hook to your production twister or TriCaster. Sym 120 10 was announced recently so they put an IO card in there to do that. So I think it’s going to be a matter of necessity if they want to. Their stream hub, their receiver, we call it a vendor. It works with third party encoders. So it can take an SRT feed in an RTMP, HLS in transcode to a different profile of SRT, HLS, MPLS. Here’s a common workflow problem. I capture content in the field. I wanted to use H 265, lower my bit rate, so I use less cellular, maintain grade quality. But then I want to go to Facebook or YouTube or CDN. No one supports HEVC 265 because of licensing. They don’t want to touch it.

The licensing has expired or has already been paid off for 264, so they want to use 264. So some of our competitors will say, well, we don’t have transcoding in the middle in our receiver, so you’ll have to do 264 at the camera side. So you sacrifice quality. Now maybe I’m doing two workflows. I want to come out SDI and I want to go IP to Facebook. Now my SDI is not going to look as good. So we transcode in the box. If we use a powerful enough server or in the cloud, we can do 3, 4, 8 transcodes to optimize for a Facebook profile, send an optimized. They want constant bit rate, a certain bit rate. So we do all that. So I think Wrist is just a natural progression. I think if enough customers will ask for it, they could easily do it. It’s a Linux box. They can-

Adi Rozenberg:

Basically, visually people ask me, do you have feelings about Wrist? I say, look, my feelings is about the customer. It’s all about the customer. I’m trying to cater to what the customer wants. He wants to use SRT, I will give you… Well, you know what, in my previous company, we basically were the first one to roll out a multi-protocol gateway that said basically you want to do SRT, we’ll give you SRT. You want to do Wrist, we’ll give you Wrist. We’ll give you the best Wrist possible. You want to use Zig Z, we’ll give you Zig Z. You want to do RTP, we’ll give you… You want to transcode or trans protocol. You want to use our propriety protocol, which is the best in the world. So wait, you are here to choose. The price is exactly the same thing. You should not care about what protocol do we use.

The only thing that we need to worry about or you need to worry about is that you will have a continuous successful service. Now, if you demand us to stay on the SRT or Zig Z, because your end device is like that, so be it. I really, I don’t want to care. All I want is a peace of mind that you’re happy and you’re enjoying the product. As you see, it’s not like, oh, I didn’t invent the SRT so I don’t want to do that. No.

Jim Jachetta:

It’s what the customer needs. Yeah, exactly. We run into the same thing. Just speaking in bonded, our competitors have a much larger footprint. So bonding. I think us bonded cellular vendors need to get together and come up with a standards protocol. Maybe we hold back some of our secret sauce, but it would be nice if there was a bonded cellular interoperable standard that Brand L could send to brand H, send to brand T, send to brand D. I won’t name the names, I’m just using their initials, but there’s ways around that too. So if you have a-

Adi Rozenberg:

On the record, I have very hard time to answer this, but all I can say is that the big problem is that everybody went to a closed garden-

PART 1 OF 4 ENDS [00:35:04]

Adi Rozenberg:

The big problem is that everybody went to a closed garden type of solution, and that closed that. There are some vendors that do use a common transport protocol, and that is the way that vendors can interoperate. The question is whether it will make sense for them or to their customers to say, “Hey, we need to have a way to communicate with others. To branch out and not to go through a SDI transcoding type of operation.” It’s too costly. It needs to be market demand. And vendors won’t be pushed to that by themselves.

Jim Jachetta:

Yeah. What happens a lot with our customers, because we do have a smaller footprint, so a workaround where I don’t have a Haivision StreamHub receiver. So we would send the mobile encoder, the bonded cellular encoder, either to the cloud or to master control, and then just come out IP, SRT. And then whoever you’re sending a feed to, they’re going to have an IRD or a decoder with SRT ready.

Or what we do a lot is we work with AWS Elemental, the MediaConnect guys. MediaConnect likes SRT or RIST. Those are their two favorite players.

Adi Rozenberg:

And Zixi.

Jim Jachetta:

And Zixi. Correct. And Zixi. That’s correct. So we’ll send them an invite to the stream or they’ll make a request for a connection. So from the cloud we’ll connect via MediaConnect. So there’s a way to connect to anyone, and all of our customers probably have an AWS MediaConnect account. But then it comes down to latency. We got to go from the mobile encoder to the StreamHub in the cloud or receiver, then up to MediaConnect. Then MediaConnect from LA to New York or Portland to New York.

But see, these are the things you can snoop out. So if you’re buying a connection, you can say, “Why are we doing all these weird routes to get my signal?” It’s funny. When I’m doing a webinar, too, I have the bad habit of mentioning something now that’s eight slides from now. I spoil my own thing. But we had a customer doing a golf tournament, and they bought a circuit from a reputable vendor. Vendor we all know. But they only bought one circuit. That made me nervous. I didn’t sleep well that weekend of the tournament. It was a major tournament. I was like, “Don’t you guys want to backup up circuit?” Saying, “Well, you said your Haivision Aviwest encoders are rock solid.” Well, not of the connection is cut. I can’t make it… and they didn’t want to do cellular as a backup. They were using the rack-mounted bonded cellular technology.

So they do have two land ports, so we could’ve had Service Provider A and B. They only chose A. And everything was fine. There was a brief packet loss during rehearsal, but the three, four days of the event, it was zero. Not a single packet loss. But why did we get that dropout during rehearsal? Thank God it happened during rehearsal and not the live. Now, with Forward Error Correction, ARQ, even during rehearsal we were able to smooth that out. The payload was not affected.

But still, if the dropout was long enough and bad enough, ARQ Forward Error Correction can only do so much. So during the event or a couple of days… it took them a couple of days to figure this out. They figured out that the circuit they bought was being routed… the event was in Europe. It was in Paris. They were being routed through the Middle East, through Asia, through the Pacific across the US to get to the east coast. Now, why would a carrier do that? It’s cheaper to go that way. That’s less… and they’re like, “Why is the latency so high?” They were going the wrong way around the world. But you see this customer could’ve, within 10 seconds, found out that problem and solved this.

Adi Rozenberg:

One of the things, going back to SRT and RIST, what is missing. One of the things that RIST can do is that you may have one primary circuit, MPLS or fiber, but you can carry the ARQ or recovery over the internet, which basically doesn’t need to carry the entire bulk of your service, only the ARQ. This is something that RIST can do. SRT is missing that capability because it does not support the bonding option, like bonded cellular will do.

So you can basically take a circuit, send only the data without further correction, only your data, would just leave headroom and bandwidth, and all the ARQ for the correction, use it over the internet. And that should basically be your backup plan for packet loss recovery. Or also, if the entire circuit goes down, RIST will allow you to basically push the data over the internet while your circuit is down. Just as an example. And that is why I’m saying that we need to move these capabilities that are right now specific protocol dependent, push it to the entire industry. So it won’t be a protocol against protocol. I’m fed up explaining why the RIST is better, but I say, “I want you to have the best SRT possible and I don’t want me as a user, as a viewer, to have the hiccups because you-“

Jim Jachetta:

To suffer. Yeah. Haivision opened the protocol for a reason, for its interop. They have starch competitors using their protocol, but they can interrupt. And you could argue that it helped these competitors who maybe were paying Zixi for licensing and now, “Ooh, I can get SRT for free.”

Adi Rozenberg:

Yeah.

Jim Jachetta:

So kudos to Haivision for doing that. So I feel like we’re preaching. We’re evangelizing.

Adi Rozenberg:

I’m not trying to preach. I’m just saying I-

Jim Jachetta:

No, no, but in a good way. We’re preaching for the betterment of the technology, not for a particular brand or protocol.

Adi Rozenberg:

My basic take is that you need to select your partners. You need to select your service providers. It’s fine to select Zixi if you are okay with Zixi and you like what this Zixi can provide you with the ZEN Master and all the capabilities.

Jim Jachetta:

The broadcaster, yeah.

Adi Rozenberg:

Yeah. And you do need a partner that will help you, that when you are in trouble, somebody will be there to pick up the phone and help you. All of the problems that many vendors don’t understand. That when I select SRT and I’m doing my own gig from the open source, support falls on me. As a vendor, I need to provide the adequate support for any problems that happening. And it won’t be Haivision to provide the support. That’s the thing.

Jim Jachetta:

Well, yeah. You’ll have to call the vendor who’s implementing the SRT. And maybe-

Adi Rozenberg:

No. In the early days you could’ve called Haivision. But Haivision, if you are a vendor of a Harmonic and the Harmonic is your SRT, if the Harmonic SRT is bad, you can’t have any grievances against Haivision.

Jim Jachetta:

Right. Well, you’ve got to talk to Harmonic and Harmonic’s got to know their implementation. Even though it’s an interop standard, not everyone’s implementation of SRT is of the same quality. Is there some truth to that?

Adi Rozenberg:

I would say that the Haivision implementation is the best when we buy the-

Jim Jachetta:

They invented it. Yeah.

Adi Rozenberg:

They invented that. They have better control. They have better feature sets than the open source version. I know of at least three other technology vendors that implemented their own SRT code. They have the own SRT stack that was tested against the SRT. That thought that the SRT is not as good as they wanted. They improved that, and they used their own stack.

But my thing is that when you select a vendor, make sure that they have the right people to answer phone calls in the middle of the night when you have a problem. And then that might be that, you know what? Maybe I will select Haivision because they have encoders, decoders over a Harmonic or somebody else. This is something that people don’t put enough emphasis about people that-

Jim Jachetta:

Yes, yes, yes.

Adi Rozenberg:

Same goes for integration and so on.

Jim Jachetta:

Yeah. We were the primary, if not the only reseller integrator of Aviwest before the Haivision acquisition, so it was actually written into our contract that Vidovation was first tier, second tier support. And then if we had a deeper problem, we would escalate a third tier to the factory. So me and my team would get those calls at 3:00 in the morning. We do a lot of fishing tournaments, bass fishing tournaments. They begin at dawn, 5:00 AM.

Now, if it’s in the Eastern time zone, what time is 5:00 AM Eastern in California? 2:00 AM. So the way I have our support system work, it goes to our senior tech first. It rings his cell phone. Then the guy below him. Then another guy, then another guy, then another guy. Then it rings me. When they say the buck stops with the boss… so I have a really obnoxious ringtone when support calls. And my wife’s hand will come over, smack me in the face, “It’s for you.” You know what ringtone I have? You know the one that’s like the nuclear… [inaudible 00:46:25]. The atomic bomb warning. The klaxon sound.

So my wife will smack me, “It’s for you.” And I’m like, “Oh yeah, it’s just… hi Bob, how are you doing? How’s the weather over there?” So we pride ourselves on good support, but yeah, now that Haivision purchased Aviwest, it’s all under the Haivision umbrella. We have 400 employees in North America backing the product now. And we do a lot of projects with Haivision. It’s a really great relationship. Love. I took French in junior high and high school. I don’t remember a word of it, so it would be helpful if I could speak French, but in another life. I think that RAM has been written over with other things. It’s been purged.

So I think we’re getting close to our start time. We’ve got quite a few people already logged in. Hey, thanks, everyone. Picking today for a webinar may not have been the best idea. The day before Thanksgiving. I was telling Adi earlier that I called a couple of key customers yesterday to invite them to the webinar. And one guy’s, “Oh, I hear background noise. Am I calling you at a bad time?” “No, I’m just picking up the turkey.” He’s in the grocery store shopping.

One customer said, “I’m doing a lot of prep tomorrow.” East coast because, it’s 1:00 now at east coast. “I’m starting my prep around noon tomorrow. I’ll be brining the Turkey, doing things. So I’ll turn the webinar on on my iPad. I’ll watch you on the iPad while I prep for Thanksgiving dinner.” And we are recording this. So we’ll give it another couple of minutes. It’s 9:59.

I was telling Fallon that you and I both like to chat, so we were like… some of my webinars have gone 90 minutes. So look, I may edit some pieces of it, but our whole little pre-show discussion, I’ll definitely use that. It’ll be like the bonus. Do they do that in Israel? In a TV show, like a talk show, they’ll be like, “The bonus round.” The banter after the cameras were shut off. They record that.

Adi Rozenberg:

They do have that. But the problem with me is that my friends or coworkers don’t like me to talk too much. They say that if they just open a mic to me, I can talk and talk and talk. So everybody that meets me at Vitrance NAB and he wants a piece of my mind or my vision, I can just spew out my thoughts.

Jim Jachetta:

You can go.

Adi Rozenberg:

I can go. I can give trade secrets, everything, as long as I’m talking.

Jim Jachetta:

When I was a kid I was very shy, introverted. I really didn’t come out of my shell till later in high school, college. So people that have known me my whole life, “He was such a quiet, polite little kid. Now he never shuts up.” All right, we’ll give it one more minute. I think we’re good.

Adi Rozenberg:

As soon as we lose the fear of speaking, webinars, presentation, panels, everything becomes like people want me to do a video and people can… like when we are doing the RIST phone videos, everybody comes prepared with papers and [inaudible 00:50:32], and they’re just making prepared. I’m just coming, “Okay. Tell me what I need to talk about. Let think a minute.” And then I just-

Jim Jachetta:

Wing it.

Adi Rozenberg:

I’m just talking. Yeah, wing it, and that’s about it. One shot. Okay, I’m good. Thank you.

Jim Jachetta:

So I’ll introduce you. You can do your own little bio, who you are. I’m sure a lot of people know who you are. And then I have a little poll. It asks, when you’re using video over IP transport, do you have occasional dropouts? “Yes.” “No, my circuit is perfect.” Or thirdly, “I don’t use video over IP. So we’ll poll that and then I can post the results. You and I can vote as well. We’re users too, so feel free to vote. I know how I’m going to vote, but I won’t tell you.

Adi Rozenberg:

Okay.

Jim Jachetta:

So why don’t we get started? So let’s do a little pause in case I want to edit.

Good morning, everyone. I’m Jim Jachetta, CTO and Co-founder of Vidovation. Today, I’m very excited to have Adi Rozenberg from Alvalinks on our webinar, and he’s going to teach us about network observability or how we can observe what’s happening in this cloud or black box of video transport. So take it away, Adi. Lay some knowledge on us.

Adi Rozenberg:

Thank you. Thank you, Jim, for hosting me. Let me just share my screen. I’m going to try to do that. Let me know when this screen is available.

Adi Rozenberg:

We haven’t started it yet.

Adi Rozenberg:

Are we okay?

Jim Jachetta:

No, no. We see the… not the actual output. Now it’s black.

Adi Rozenberg:

Don’t see?

Jim Jachetta:

I swear we tested this a half hour ago.

Adi Rozenberg:

Do you see my name?

Jim Jachetta:

I see. It’s blanked out.

Fallon:

I would go in and out of sharing again, Adi. And that works. Because I saw it briefly but then it went away.

Jim Jachetta:

Yeah.

Fallon:

I see it now.

Jim Jachetta:

No, no. But it’s-

Adi Rozenberg:

I stopped sharing. I’m going to do it again.

Jim Jachetta:

It’s not the output of the deck.

Adi Rozenberg:

Hold on one second. Let me start. Do a share screen. Going to do PowerPoint. Put share. We see it now?

Jim Jachetta:

We see it, but it’s… okay, go ahead. Yeah, so launch it. Launch it. It’s not showing us the presenter output. It’s showing us the… you know what I’m saying? You have two screens.

Adi Rozenberg:

Yep.

Jim Jachetta:

That screen’s not working. Maybe switch screens.

Adi Rozenberg:

Fine. This is the… now?

Jim Jachetta:

No.

Adi Rozenberg:

No. How do I kill the presenter mode?

Adi Rozenberg:

I have the presentation.

Adi Rozenberg:

So you know what? Let’s do it like this and let’s-

Adi Rozenberg:

No. Hey, hey, hey. I got a solution here.

Adi Rozenberg:

You see the less graphics, but…

Jim Jachetta:

Yeah, go ahead.

Adi Rozenberg:

Just once again. Last try. Now?

Jim Jachetta:

No.

Adi Rozenberg:

Fine.

Fallon:

Jim, can you go ahead and share it and then

Jim Jachetta:

Yeah, yeah. I’m trying to remember where I put it.

Fallon:

All right, cool.

Jim Jachetta:

Here it is.

Adi Rozenberg:

I apologize for the difficulties.

Jim Jachetta:

Here, I got it. So here. Stop. Stop-

Adi Rozenberg:

Are we in host mode or we’re in-

Jim Jachetta:

Yeah. Stop, stop. Stop sharing, Adi.

Adi Rozenberg:

Okay.

Jim Jachetta:

Thank you.

Adi Rozenberg:

Well, sometimes we cannot-

Jim Jachetta:

Sorry, guys. It’s a holiday. We started drinking the apple cider, the spiked cider too early. So here we go. Here we go. I got it. I got it. Here we go, Adi. Okay. Do you see it, Fallon?

Fallon:

Yes. Looks great.

Adi Rozenberg:

I see it. Yeah.

Jim Jachetta:

It’s the version without the… it’s fine. It doesn’t have all… that’s fine. Here we go.

Adi Rozenberg:

So can you go to the next? Thank you very much. My name is Adi Rozenberg and I am going to talk about a new solution that we are providing. I’m coming from a company called Alvalinks. I will give you a short bio of myself. I will talk. I’m a visionary and a technology leader with rich experience in developing state of the art broadcast and network technologies in the field of broadcast video and networking. I am the author of eight patents on live video delivery over public internet and adaptivity in video delivery.

In 2018, I was a recipient of the Technology and Engineering Emmy Award for my pioneering a reliable transmission method of a live contribution and distribution TV links. Me and many other people with me. I’m actively promoting the secure and reliable delivery of IP networks worldwide, and I’m a contributor to the RIST Activity Group for the development of the Reliable Internet Streaming Transport protocol code, RIST. But I also know all other protocols in the field. SRT, NDI, 2110, and even Zixi. Next slide, please.

Jim Jachetta:

Yeah, hold on one second. That was the older deck I found the most recent one. Here we go.

Adi Rozenberg:

Okay.

Jim Jachetta:

There we go.

Adi Rozenberg:

It’s the same content.

Jim Jachetta:

Beautiful. Do you see the next slide?

Adi Rozenberg:

I don’t see any screen-share.

Jim Jachetta:

Oh, let me see. Let me… stand by. There we go. Third time’s a charm. There we go.

Adi Rozenberg:

So let’s talk about video over IP. It’s becoming a norm. In the last six years it becomes a norm as we have the spread of cloud, IP workflows coming, new protocols are coming. Open source protocols like SRT, RIST, NDI are coming, and the industry moves to IP-based workflows. We are moving away from the traditional way we used to do things using dedicated fiber, MPLS towards internet and cloud workflows. I’m not saying they are bad. Actually I’m saying those should do that, and do a combination of those as you see fit for your services. Cloud production and remote editing is challenging for everybody. The delivery network is something that many people don’t look at. They think it’s there, it will provide everything, but in fact it’s becoming more and more problem-prone than before.

Jim Jachetta:

They think it’s magic. It just always works.

Adi Rozenberg:

Exactly. Like, “I just need to call the service provider, the local service provider, and everything will work. I can consume Netflix, Amazon Prime, Apple TV, and it’ll always work. It will never fail on me.” So let’s try to understand what is missing. Why is there a need for something here to make some logic or some sense in the issue? The problem is that more and more responsibilities are falling on the broadcast teams to master. Broadcast engineers that are trained in video production, taking the camera feed, taking the NFL game, suddenly need to understand things like ping, routing, cloud, CICD, gateways, redundancy, availabilities of VPCs. Things that they did not learn.

Jim Jachetta:

Jitter. Jitter is another one, right?

Adi Rozenberg:

Jitter, yeah.

Jim Jachetta:

Latency.

Adi Rozenberg:

Exactly. Those are new things. Network availability. These are things that pose a problem for them to learn, to master. And the technology is ever-changing, all the time. Every year or two years we have a new protocol, a new workflow that we need to master.

Jim Jachetta:

We were talking before we started Adi, I had mentioned that I hear this all the time, a broadcast engineer who’s proficient at IP would say, “I miss the days where I could trace an SDI cable from the camera to the switcher to the router to the production switcher to the transmission link to the microwave link. There was a path you could trace out. It was easier to troubleshoot signal continuity.” It goes into an IP switch, even in your own network, it’s a void, it’s a cloud, it’s a black box. Once it’s packetized and it goes into my facility… facility is one thing. At least your IT department has control over that. Once it hits the public internet, or even a service provider, their managed cloud, their managed services, it’s all a big unknown black box, right? And that’s…

Adi Rozenberg:

Exactly,

Jim Jachetta:

You help us shed light into that black box.

Adi Rozenberg:

Exactly. So just think of it that you take your ethernet cable, plug it into the switch, and then it goes to the cloud. What is the route how it goes through? Where does it hit? But moreover, when I have a problem and it is outside my organization, whom do I call when it hits the cloud? What is the name of that? IT manager of AWS? Azure?

Jim Jachetta:

You call Jeff Bezos. I got Jeff on my phone.

Adi Rozenberg:

So in fact what we see is that more and more network-related problems identification, resolution are taking a lot of time and resources for many organizations. And we want to get rid of that. We want to save time. We want to be on top of that thing that we call the network. We can’t allow it to stay like a beast, like that black box. We need to have more control. We need to have more visibility so we can prepare ourself, get ready to be on top of problems so that, if something is brewing or happening, there is a trend, I can react to that. I can activate redundancy, I can select maybe a different availability zone, I can select a different circuit or service provider. But today everything is done reactive. Something breaks down and then I make the decision. But while I’m making the decision, I have service problem. I have black frames. I am suffering. [inaudible 01:02:50].

Jim Jachetta:

Use our automobile as an example. Some people drive around and they don’t think about car maintenance, routine maintenance, rotating the tires, changing the oil until the red light comes on and there’s a problem. We want to prevent these problems.

Adi Rozenberg:

And then you pay a hefty total for fixing that.

Jim Jachetta:

Because you burned out your motor now because you didn’t change your oil.

Adi Rozenberg:

Exactly. Exactly. Something simple that you could’ve done could have saved you a lot of dollars. So many people will ask, “Well, I have reliable delivery.” Like you, I said that I’m working with the RIST. Why are they not enough? Why RIST, SRT NDI, QUIC, or even WebRTC are not enough? And the fact is that they are reactive to events. They basically react to packet loss, they react to jitter, they react to reordering. They are trying to shield the receiver against any network artifact, but they are focusing on one single stream. They don’t care about the other streams that might be running alongside with them.

They are not seeing that maybe there is a brewing problem from IT backup or IT operations or somebody else using the same medium like them that’s causing them the problem that they need to fix. They need to be vigilant and see what is going on so that they can react. They can go to their service provider, ask for better quality of service rules. They can ask for more bandwidth. They can talk with their IT people. Or maybe their own services to get more peace of mind. The problem today is that they don’t get that. They just have basic information about how many packets were sent, how many packets were received, how many packets may have been lost, how many packets were received. What caused the problem? Or what are the latency changes, jitter? Usually that information is missing.

Jim Jachetta:

It happens too often. And luckily your RIST or the bonded cellular SST is robust enough to overcome momentary losses. But there’ll be a hit, some unexplained hit, and nine times out of 10 it’s some bottleneck on the public internet. Or it’s not necessarily, the first hop of the cellular link. It’s the backhaul from the cell site to the Sonnet network and then the public internet. But you don’t know where this problem is happening. So it would be nice to know. Like, “We took a hit and we don’t know why.” To know that, “Yes, it was the internet ISP at the broadcast facility.” “No, it was the cellular provider.” “No, it was the cellular provider’s Sonnet link,” or their backhaul link, et cetera. You can dissect all of that. Correct?

Adi Rozenberg:

Exactly. Exactly. Basically, you raise a very critical point. You want to know to whom to call, whether it’s the vendor, the service provider. As you have the information, it will pinpoint the problem faster and you can resolve the problem faster. You might not be able to fix it immediately, but pinpointing the problem is the first rule where I need to go and address it next time so that it will not repeat itself.

Jim Jachetta:

Yes. Yeah. We were talking before we started that any form of correction, whether it’s self-correction, I need to improve something in myself. The first step is being aware that you have a problem. And that’s personally. And it applies to technology. Adi is a big proponent of running tests before the event, run tests while you’re evaluating service providers, and then continuously probe and test during the event, and you can avoid serious problems on game day or during your event.

Adi Rozenberg:

Exactly. So let’s hit the next button.

Jim Jachetta:

Okay.

Adi Rozenberg:

So the question is, how do we identify network problems and how do we prevent them? We need to get to the root cause of the problem to ensure that we tune our delivery solution and use the right network solution for our intended purpose. If you have a five megabit pipe, you will not try to push 20 megabit stream through it, right?

Jim Jachetta:

Right.

Adi Rozenberg:

And if the pipe has high latency, you will not try to push a low latency type of service across it.

Jim Jachetta:

Yeah, like Constant Bit Rate low latency or JPEG 2000 or something like that, right? Yeah.

Adi Rozenberg:

Exactly. So IP networks, what you need to think about coming out of this presentation is that IP networks should be treated like an evolving entity that is constantly changing and evolving, meaning that what you tested today might change tomorrow. What you tested at 9:00 AM might change at 11:00 AM. You need to be on top of that. And of course, in two weeks time, that might change. And in six months time it will change. You need to be on top of that. Otherwise you’re constantly reactive. And that reactiveness will take time, materials, support fees, contractors. That will be a pain in your bottom line.

Jim Jachetta:

Well, yeah. Lost advertising dollars, lost revenue.

Adi Rozenberg:

Exactly.

Jim Jachetta:

Sponsors don’t pay for TV shows or sporting events that are black or jittery or dropouts.

Adi Rozenberg:

And you might lose your next account because you fail to deliver. So what is missing from what we do now? With the traditional workflows, broadcast solution will use multi-viewers to see how the data was encoding or encoded or received. We are using ETR 290 compliance to see how the transport was delivered. We have alarm monitoring. Those focus on content compression, transport packaging, and delivery issues after the fact. Okay. Slide the next, please.

Jim Jachetta:

Oh, you know something? I forgot to do our little poll. So let me see here. We can… let’s ask our audience. Have you ever experienced… regarding video over IP…

PART 2 OF 4 ENDS [01:10:04]

Jim Jachetta:

… or experienced regarding video over IP. Have you ever seen an unexplained packet loss? A dropout? All the things Adi discussed, jitter, black screen, freezing, even minor things. We were talking about a customer of mine that I actually spoke to yesterday said that I use vendor A for my video over IP transport. I never have a problem, never have a single problem. Maybe Adi’s overconfidence is something to consider that if you’re overly confident and you think your solution is infallible, that’s the guy who needs to watch out, right? So you see that? Yeah, everyone says we’re all in the same boat. We know an IP link is not perfect. So sorry to interrupt you Adi, but-

Adi Rozenberg:

That’s okay.

Jim Jachetta:

… we made the poll. I wanted to use it. Fallon made it for us.

Adi Rozenberg:

I can tell you that before going to the next topic is that I’ve been in touch with tier one service providers using IPs organization that I did not even imagine that might need my services because they run a very tight ship and they have either their own circuits, their own fibers, and in fact they were the first one to acknowledge that they do have problems. They do suffer. They invest a lot of money and the manpower in order to make sure that things like that do not happen and they do. So they need to be on top of that and they are always looking for better ways to improve their operation.

Jim Jachetta:

Yeah, we were talking about that too, that this is not just a tool for the customer, the broadcast or the sports league. The service provider could have this tool as part of their arsenal to be proactive so they have control over how the signal is routed. So if there is a congestion at their POP or their data center in Chicago at the moment, they can intelligently route around that to get to New York from LA, et cetera.

Adi Rozenberg:

That is correct. And that brings me to the next segment. Basically IT solutions today. No, no. The IT solutions today basically use commercial observability tools. Here is a small list, ThousandEyes, Zabbix, SolarWind NPM, Datadog NPM. Those tools are used by broadcasters, circuit providers. But these tools focus on one thing, network availability, cloud resources, application users, how many users, and they do a limited sampling of the network. They do not focus on the video delivery. They cannot in their current installation detect transient events like jitter, packet loss. Things that happen for the duration of 10 milliseconds or 100 milliseconds are not sampled because usually they will sample the network once a second or even more than that. Now when you test the network once a second with a packet, an event that happens for the duration of 10 or 100 millisecond will not be registered with that one packet unless that event is prolonging for a long time and this is a constant event of packets. Transient congestion events, transient latency changes, and I will show you that in a real environment cannot and will not be detected by these solutions. So even-

Jim Jachetta:

So, the granularity of these tools is not fine enough to capture those 5, 10, 15, 20 millisecond events.

Adi Rozenberg:

Exactly. Just as an example to people that in the audience, if I’m doing a ping test, I would send a ICMP packet every one second. I can make it even faster. I can send it every 100 millisecond. That means I am sending 10 packets every second. Imagine if I’m running how many packets I’m not seeing, how many loss events I’m not feeling. I need to have better granularity, I need to have better accuracy. I need to have something which is at least one millisecond or below that to sample truly what is the latency changes on a packet by packet? What are the routing changes? What are the jitter repair impacts and so on.

Next, we already addressed that the reliable delivery solution, I addressed that in the previous slide. RIST, SRT, ZIXI and NDI, they focused on the jitter recovery, but they have narrow viewpoint. They focus on a stream by stream basis and that is a problem because they don’t see who is causing the problem. The causation, they don’t see what caused that packet loss and whether they recovered that or not. I am saying we need to be on top of that. So now let’s talk about what we do and what is different.

Alvalinks CloudRider, which is the name of our solution, provides a proactive network observability that is a patented technology. We basically run multiple tests between the source and destination gathering more than 20 KPI tests in parallel synchronizing time. So then now we can synchronize events and see them on one single pane. So we can now see the correlation between events as they happen. Each test is focused on dedicated packet arrival, whether it is sent, arrived, and we can compare that on a packet by packet incoming or outgoing between source and destination. We take that metadata, process that and send that to a cloud database, timely cloud database for further analytics and visualization. In the future, we’ll add machine learning and AI to give tool tips and other capabilities that are still in the design. User can see real-time data and watch historical data, which is very important.

I like previous solutions that you have an alarm and you look at what caused the alarm. Now we can see the trend, we can see that the latency has changed. Maybe the jitter is rising, maybe the network is doing so many reroutes, those reroutes cause packet loss, latency changes and I have some slides and then a realtime network to show you that later on.

Jim Jachetta:

You were telling me, Adi, actually when we first talked, you, me and your CEO, you showed me live, that a certain cloud provider does some sort of a resynchronization or a warm reboot or refresh of all their data centers every couple of hours. Now that works fine for traditional enterprise data storage, email, web traffic, et cetera. But if you’re streaming a video which comes through the internet like a fire hose, even if it’s a couple of hundred millisecond little hiccup, if that happens every two hours during a eight-hour live sporting event, that’s bad. So you might determine, I don’t want to use that cloud provider for this type of workflow. It’s not meant for video over IP.

Adi Rozenberg:

That is 100% correct. What we found without mentioning-

Jim Jachetta:

Naming names, yeah, yeah.

Adi Rozenberg:

What we found is that we had the customer that he thought that he had multiple diverse path with the cloud provider and in fact those diverse path basically went through the same routing cord that was causing it to change every few hours and create a blip of packets loss and reroute latency change every few hours. And that was basically the diversity that was the idea is that if one path is suffering, I can just jump. So in fact, all the paths suffered exactly the same problem at the same time. No diversity and they made different decisions accordingly.

So now for the audience, I want you to remember the traditional workflow that you have when you’re doing a sport event. Let’s do left to right. We have a stadium, we have cameras in the field where they are with cameramen or they are now self powered and they’re remotely controlled. They will usually go locally to local encoders and we will watch, monitor locally the encoding quality. We will then push that to the cloud to a destination for delivery. And we’ll then want to monitor that delivery. We’ll put a multi-viewer, let’s say TAG as an example or any other multi-viewer of your choice. And why do we do that? Because we want to see the delivery. We want to make sure that the delivery was okay, but in fact, what we are doing, we are checking whether the network was performing because we in fact know that the source was pretested and there was no problem, no alarm came from the source. So in fact, we are trying to guesstimate what the network does.

So let’s visualize that. What do we want to be on top of? Let’s do some graphics animation. So let’s hit the button. Basically we’ll start with, let’s detect the paths. We think that we have a unicast path, and this is example for one of our customers that was streaming from Japan to Florida. He thought that he has two distinct diverse paths, but our test showed him that in fact they have a lot of two points that are joined together. Basically the path goes to two points that are appearing on both routes and actually his routes are intertwined so that he has no diversity. After learning about this, he was able to change the service provider so that he had true diversity. Moreover, with our testing, we also found out that the latency configuration that he had in his RQ protocol was misconfigured and he had to put some more milliseconds in order to maintain and deliver. That hits another button.

Jim Jachetta:

Here’s another story that popped into my head, Adi. So a sports network, a customer of ours went and rented two different circuits, SLAs, from two different vendors and they found out after a failure that there was a commonality between those two circuits that they… There’s nepotism, we’re all sharing different data centers and a lot of it we have no control over it and maybe even the service provider doesn’t have control over some of these things. They just let it flow as it may. So this example, it was the same provider. You bought redundant circuits from the same provider. So my point is just because you got two different vendors, you could still have this same problem, right, Adi?

Adi Rozenberg:

Exactly. And I was telling you that we found something like that just today, six different circuits that I will show you. Six different circuits, all six circuits which are diverse geographically. One is in Europe, one is in the US. We did not even think they have any commonality other than some sources. And in fact, we found that they had a singular point, one point that was common. And I will show you that as we will go to that point.

Jim Jachetta:

In any engineering project, whether you’re launching a rocket or television, good engineering, you don’t have a single point of failure.

Adi Rozenberg:

Single point of failure. Exactly. And in fact, we have a correlation. We found exactly when we had the problem, we immediately could trace that and see that specific point, that specific hub caused the problem at that specific time, which was mind-blowing.

So going back to the workflow, so what we need to monitor, we want to monitor how the data leaves our organization, how that stadium, we want to focus on the streams. In this example, we have like ATS or SRT streams in this capture that we monitor leaving the source. Now we want to do the opposite on the other side. We want to check how the leak, how the network behaves and what are the pitfalls. So this is a live chart. I will show you many others that basically shows the behavior of the network. If one has a keen eye, on top you can see packet loss at the magnitude of 25%, you can see problems that arose from the network delivery. You need to be on top of that. You need to see what happened, what caused that, and we’ll see that in later slide. We’ll see things like that, that explain that more in depth.

Jim Jachetta:

Those are pretty long events of 25% packet loss on a circuit that you have in SLA that you paid for.

Adi Rozenberg:

In this specific circuit, this is an MPLS that the QS was not set properly. Nobody knew that and the customer in this case was suffering from black frames, freezes. The broadcast people said constantly the network is not working perfectly, but the IT people, they were clueless about it and they said, “We bought a circuit from a known MPLS provider and we have the five nine SLA, so everything should be in fact, okay.” Once we came over and installed our OutRider source and destination, we immediately saw this problem and then we issued a report a day after the circuit provider basically admitted that they did not set the QS properly for the service. And after a day, that circuit went from 25% packet loss to 0.2% packet loss, which is not the greatest, but still SRT can overcome that.

Jim Jachetta:

SRT or anything with forward error correction or ARQ can overcome that. So it was an honest mistake. Something was not provisioned right. So these live events, there’s always a setup and rehearsal. Adi and I encourage anyone doing a live transport over IP, public internet or something, MPLS or whatever. This could have been discovered before the event to prevent dropouts during the event. This could have been ironed out during rehearsal and set up instead of live at the event.

Adi Rozenberg:

In this example for, sorry for interrupting, this example, the customer was unable to go to air, live to air to launch new services for more than six weeks. That means six weeks of time to money lost.

Jim Jachetta:

Oh, my goodness.

Adi Rozenberg:

Okay.

Jim Jachetta:

Oh my goodness.

Adi Rozenberg:

Just imagine how much power and money lost, revenue lost because of that.

Jim Jachetta:

And we won’t talk pricing on a webinar, but your services, it’s a service, it’s not that expensive. You’re paying, it’s a fraction of the cost of what you’re paying for this circuit. Why not have a tool to evaluate that circuit you’re buying every week, every month.

Adi Rozenberg:

Before jumping to the slide, I would just tell you a testimony of a customer of ours. One of our customers with the tools that we have yet to basically launch a service from the Far East to Europe. He basically installed our, we provided docker container, installed it, one in each premises, one in the destination, and he was able to pretest the network instead of traveling from the far east to Europe. And he wrote to us, “Guys, we saved more than $15,000 in travel expenses sending two engineers spending the week finding out what is the problem. We did everything from our home. We understood the problem. That justified paying for the system. It was far less expensive. Our operation are now very happy because this has saved us some carbons because they are very carbon-free, carbon emission-“

Jim Jachetta:

Save time, money, carbon footprint. Yep.

Adi Rozenberg:

Yeah, exactly. So this is a testimony that we’ve got.

Jim Jachetta:

And we’re concentrating on transport, but if you have a large facility or a multi-location facility connected with your own private WAN, you really need this tool inside the facility as well for troubleshooting.

Adi Rozenberg:

Correct.

Jim Jachetta:

It’s not just for transmission.

Adi Rozenberg:

Correct. So let’s understand what we do. We do many functions. Let’s start with link evaluation before going live. We have an event. We come a day before, maybe a few hours before. Most organizations come at least a day before. We want to evaluate the circuits that we have. Usually we will lease the service. We are not in control even though we may been here two weeks before, somebody else was in between, somebody else changed the configuration. We have to validate and test everything. So we need to do the following function. We need to do path discovery. We need to understand what dynamic path changes might have happened since the last time we used the event. What are the packet loss of each route? And this means that we might have multiple routes. The system is not tuned to one specific route. We’ll test all, we will discover and test all available routes so we can focus and see which is better. Or maybe we want to use bonding.

We will want to discover service provider discovery. Are we using the right service provider year on end. We want to test path RTT, path latency of jitter. We want to discover the path ops that are truly diverse [inaudible 01:30:11]. Many people do that by pushing up reel encoders, but they just check whether the encoder hits the destination. They don’t have the visibility about the network and how it behaves during that transmission.

Now sometimes we want to test the SLA. We want to do a step-by-step type of testing. So we want to test what is the impact of every step that we will take to discover the packet loss of each step. What is the jitter when we introduce a step, what is the RTT? Because sometimes the network is reactive. I’ll show you an example where the network basically reacted to the changes and basically clamped the signal completely and the customer didn’t know about it. He was not aware of that feature in the network that he had to go and reload.

Jim Jachetta:

This is a good point, and I know from my bonded cellular experience that a cellular connection typically, with 5G it’s improving. So it’s more with 4G, LTE, it’s 40 to 60 milliseconds is a typical connection, but there’ll be these unexplained spikes, 1200 milliseconds.

Adi Rozenberg:

Exactly.

Jim Jachetta:

But it only occurs for a split second. So you see these spikes. Now if you’re a bonded cellular provider like Highvision, they actually monitor latency first, then look at throughput second. So they’re monitoring those huge spikes and what they’ll do is if the latency spikes for the moment that until it restores itself, they won’t send any traffic on that path.

Adi Rozenberg:

They basically halt the transmission.

Jim Jachetta:

Yeah, yeah. In order for bonded cellular to work, the bonded industry had to figure this out. But if you don’t have bonding capability, you’ve got no way to traverse that. So this gives you visibility of that.

Adi Rozenberg:

Exactly. So now with Highvision, with [inaudible 01:32:18], this is contained within the technology, so you don’t have visibility to that. Now what happens if you don’t have the solution or you want to have more visibility so you can better tune it? That is what we are bringing forward. So maybe you want to use SRT and have throttling because with SRT, you can configure it to do throttling. Same goes with RIST, same goes with Zixi. There are ways to do that, but you need to know the landscape. You need to understand what is going on. You need to understand the history when it happens in reaction to what so then you can fine tune your sender and receiver to overcome these problems so you have a fluid service without interruption.

Jim Jachetta:

Many customers don’t want to use a variable bit rate. “No, no, no. I have a 5 mg. I want my show to run at 10 mgs.” So they want to do constant bit rate at 10 mgs. Now doing that over cellular, yes, if you have a very good connection, we can get away with that. But if the pipe collapses below 10, I mean it’s 10 plus overhead, so your transport stream may actually be 12 mgs. If that pipe collapses and can’t support the 12 mg, you’re going to lose frame. So you need the variable bit rate, but as we all know, constant bit rate always looks better if you can push it through.

Adi Rozenberg:

Exactly.

Jim Jachetta:

So these are the things we can probe for, test for.

Adi Rozenberg:

So here I’m giving you some slides from our, what we call a stream test, and this is two examples. On the left you can see a fixed bit rate that we run, and on the right is the step-by-step. Now one of the things that we added is that we added circuit breakers. One of the things that we are worried about coming from the industry is that we don’t want to cause any harm. So we need to have some circuit breakers so that if by any chance we hit a level of packet loss or latency that might harm other services, my services that they are running alongside with me, I don’t want them to inject too much packet loss. So the circuit breakers are there to stop me midway so that I will not cause harm. I can’t have a human in between to make the decision. I need the machine to run a circuit breaker to stop that test, that evaluation. When it happens and by any chance, it might cause harm and something like that might cause harm if you’re running services and you are not aware. So this is important and it is in the technology.

Jim Jachetta:

Yeah. Let me see if I understand Adi. So in the example, I have a 10 megabit per second stream going through my circuit. Alongside that, you’re running a 1.1 megabit per second test stream. Is that what you’re doing?

Adi Rozenberg:

No. So let’s do it like that. Let’s say I have a pipe which is 40 megabits and I’m running my service. I’m running two services. One is 10 megabits, another one is 10 megabits. Now I want to test the network and I want to introduce another 30 megabit sprint, which if we do the math 50 is more than the 5 capacity of 40. The idea is that if I’m not aware of the available bandwidth, I’m engineered that I did not read the contract that I have in fact 40 megabits and I’m pushing that 30, it will over saturate the pipe and it will basically cause damage to the two services that are running. I need to stop that ASAP, I can’t wait a minute or for a human to come by and stop me. I need to stop that. Even if I’m doing step-by-step. So I would start, let’s say by at one megabit or two megabit, but there will be a point that the introduction of more and more bandwidth will cause harm to the link. I have to stop before it causes harm to those two services that are running. I don’t want to cause any problem. Of course, if there are no services running, I can do whatever I want. I can over saturate.

Jim Jachetta:

Right. You’re doing a test before the event, but while you’re live, you don’t want the testing probe to interfere with a live show and then back. So it’s instantaneous, you see a bottleneck and you shut off for a split fraction for a second. Very cool.

Adi Rozenberg:

Usually when we are doing running our CloudRider technology on a continuous service, we usually will consume about one megabit. We will send about 100 packets, give or take, plus some more for pathway discovery, which is about 1.1 megabit in total. So that is running over the headroom that you originally have and we are running back and forth to test the link ingress and egress.

So let’s see. Next slide is from a real location. This is from a real stadium that I will not name, but in this case on the left, can you see my mouse? In this case, the customer who had seven cameras installed, he wanted to introduce the eighth camera and every time he introduced the eighth camera, the entire circuit went down. There was a clamp that basically brought the entire service down anytime he introduced that eighth camera, which was about 20 megabits.

So in fact, he had our agent there for testing the network and we told him, let’s try to do step by step. So he said, “Okay, let’s try to move 10 megabits.” And that’s what you see on the left side with the mouse. Basically as soon as we started to increase the beat rates, suddenly the clamp came and you can see immediately the impact. Packet loss error rate went to 25%. And when we tried to push further, it went to 50%. Look at the RTT jumping sky-high from almost 0 to more than 200 millisecond. Buffer formulation is a feature that we have that basically calculates, takes into account all the packet loss, latency, RTT, and basically gives the user a number that we believe if his buffer is set correctly, we’ll be able to overcome packet loss. And here we are jumping from 0 to almost 5,000 millisecond this case.

So basically the system stopped itself. You see here a sharp dive in the test because the system stopped itself and said we are causing harm. Basically the system was said that more than 25%, shut it down. So then we said, let’s do that gradually, let’s find where that limit comes. And we stayed the 2 megabit by 2 megabit increase and we found out that here at 8 megabit there is a clamp, meaning that the system was rigged, the network, the routing was rigged so that when they cross a certain boundary, it will clamp the network completely.

Jim Jachetta:

That sounds dangerous.

Adi Rozenberg:

But it’s normal because somebody said-

Jim Jachetta:

You paid for that bandwidth, so they’re giving you extra headroom that you didn’t pay for. So you need to manage that. So is this the event here, Adi? I got the red pointer. This is the event happening right here. The eighth camera was added. It even almost goes above 50% RTT, it looks like 300 milliseconds buffering, and then your analysis shut off. Is that what we see here?

Adi Rozenberg:

Yeah, that is the circuit breaker coming on.

Jim Jachetta:

Circuit breaker hit and then we backed off and then here you’re slowly probing again to try to-

Adi Rozenberg:

Find that limit.

Jim Jachetta:

Here, maybe the customer turned the eighth camera on again. That’s why we see this big spike here?

Adi Rozenberg:

Yeah.

Jim Jachetta:

Okay. And then here you are probing. You’re probing.

Adi Rozenberg:

Exactly.

Jim Jachetta:

Very cool. Very cool.

Adi Rozenberg:

Basically, this is a real testament of what is happening. You need to be aware. People come to an event, they are not in control of the circuits. Sometimes they even own the circuits, but they don’t know who was there before. You need to test. Production teams know that they need to test, but they don’t have the adequate tools to do that. Let’s go for the next slide.

So continuous path exploration. What we need to do is that when we deploy Alvalinks agent, it’ll start to discover the path. It will look for all the available options to reach from point A to point B, all the available mediums, whether it’s LAN one, satellite settling, whatever it is available, we would try to discover that and continuously try to discover that. People might know trace route. This is like running trace route, but on steroids.

Now the problem with trace route is that trace route is based on ICP ping and ping, when they’re running ping over a network, it will propagate on a different way than a video stream will. So we need to do something which is basically using UDP session based. We basically listen to the video session and try to run as if we were in fact a video packet running on the same session trying to detect the hubs that the video might take. Now what that gives us is that when the network is going over load balances cloud, we now can discover not devices, load balances and continuous changing of the network. One thing that we were able to establish and find is that once we are talking about cloud, the cloud ever-changing all the time, it’s very diverse, but the same can be true on data centers data when you have BGP changes and so on.

Jim Jachetta:

So your test payload or your test stream mimics a video stream by being UDP, just like a video transfer.

Adi Rozenberg:

Not only UDP but UDP using the same session parameters as your video stream, but without confusing the receiver.

Jim Jachetta:

Gotcha. You have your test payload, not your video payload.

Adi Rozenberg:

Exactly. So what we do is that we provide, at the end of the day, which UI with correlated time-based charts for doing event. Zoom in and out, you can also do, you can see real time data and historical data. We focus on what matters for video delivery, what are the problems that you need to be aware of.

Now, we don’t talk in a video lingo. We try to talk in IT lingo because at the end of the day, IT problems will be solved by the IT people, the IT teams and the service providers and the cloud providers. They understand IT talk, but we need to give them the information, accurate information so that they will understand. We can’t tell them we have black frames. They don’t understand what is a black frame, what it’s caused from, what is the causation. We need to talk about in TTL RTT hops, the same lingo that an IT team member will understand because that is how he was trained.

This is an example that I showed before about the diverse path that we discovered. Basically the system running discovered multiple paths and that they were using exact two end points and you can see the difference between the top and the bottom. You can see that they look different, meaning that the path from Japan to Florida is not exactly the same as the path back. This is another thing that I will show you later on.

Jim Jachetta:

Well, yeah.

PART 3 OF 4 ENDS [01:45:04]

Adi Rozenberg:

This is another thing that I’ll show you later on.

Jim Jachetta:

Well, yeah. Most providers have returned video of some sort, program video going back to the event, teleprompter going back. And we were talking about this before we started, I think in our mind we visualize a straight line, it goes very cleanly. Japan, a data center in Honolulu, hits California, hits Texas and goes to Atlanta. It’s not a straight line, it’s not even on a managed circuit, right? You would think, “Well, I have a managed connection.” The public internet, I would expect this not a managed connection.

Adi Rozenberg:

I will give you a crazy example. I was talking, it was at NAB this year, NAB Vegas this year. And I was showing the system to a customer, to potential customer, and we had that flight line, which was like 22 hops, which is standard, which is okay. Suddenly the system which is ever running, it’s continuously running, bringing information, updating. Suddenly the screen goes black, meaning that suddenly we see 200 hops. We see crazy center, we call it an elephant trap. It’s like a mesh of points for five minutes. And I’m starting, oh my God, there’s a software bug, something that can’t be, no way.

Jim Jachetta:

It’s not possible. It’s not possible.

Adi Rozenberg:

It’s not possible.

Jim Jachetta:

You thought the tool had a bug. It’s not possible-

Adi Rozenberg:

The tool has a bug, yeah.

Jim Jachetta:

Okay, okay.

Adi Rozenberg:

But we did draw the information from somewhere, right? We can’t just push bogus information. 45 minutes later I’m getting an email from my data center saying to me, “Oh, we had a network problem that caused us to go to different location. And that slowed down your connection.”

I said, “I saw that. I literally saw that happening as I was demonstrating-“

Jim Jachetta:

In real time you saw it happening.

Adi Rozenberg:

Yeah. And it’s like you need to be a believer to see that. Sometimes you see problems that you say, “It’s unimaginable that I will have from 22 to jump to like 200 hops. That’s unimaginable.” But it did happen. I wish [inaudible 01:47:37].

Jim Jachetta:

Or even on a smaller scale, I’m expecting a circuit, a direct circuit from Paris to Atlanta, and instead it’s going the other way around the globe with 22 hops. So you assume that the obvious is that it’s a simple, simple connection.

Adi Rozenberg:

So let’s jump next. So here’s an example. For instance, we have here two paths. Same basically going left to right. Top, we see a straight line. The return has load balances. You can see in green there is a change, meaning that the path will divert itself, and then it hits another load balancer, meaning that it goes from one service provider and jumps to another one, and they have another load balancer. And what does that mean? That means that whether you are return path, or maybe you are doing an SRT ZIGZAG, your message back about a problem will be stall down. It’ll slow down. So you might need a bigger buffer to accommodate, because the network is not at the same latency or distance as you might think.

Jim Jachetta:

Right. On the return leg, I talked about the example of video coming on the return leg, but ARQ is coming back, or the ARQ requests are going the opposite direction. And if the ARQ request is delayed, as you said, Adi, we got to increase our buffer. And-

Adi Rozenberg:

It’s not only the ARQ, but let’s say you have a remote control, maybe you are running a remote NDI camera and you need to control it with the TCP. While we are measuring that in UDP, any TCP circuit and we can trace TCP circuit-

Jim Jachetta:

Or I’m shading the camera and now this would cause a little bit of delay. My camera control would stutter. I’m moving left, it doesn’t move left right away, or I’m changing iris. So very cool, very cool. And while this is happening on the return leg, the outbound leg was fine. It’s a different path.

Adi Rozenberg:

Exactly.

Jim Jachetta:

It’s like it’s a different circuit.

Adi Rozenberg:

Exactly. So let’s go to the next slide. So then we come to what we’re doing, which is a continuous stream delivery performance. We are testing the link. In the first previous slides, we talk about evaluating the link, we’re going to a new location, bringing a new service. But we also need to be on top of the circuit service as we are running on our 24 by seven operation, we need to be on top of packet of jitter variation, route changes, all the same problems. We need to be on top of that. But not only that, we need to be vigilant and detect any other IP flow that either we generate, or somebody else is generating, or using the line that we can detect that may cause problems. We had cases that IT did the backup, and I had slide deck that I showed that a year back at the NABIP showcase, that we had the problem that was caused by an IT back up operation that consumed more than 200 megabyte of my video traffic, and that caused packet loss and latency changes.

But then, we may want to focus on our video. We want to monitor the video, how the video behaves. So we can then select specific flows and we can tell the software, “We want you to focus on those flows, either at the source or the destination, or both. Please monitor them on a packet by packet basis.” Now we don’t touch, we don’t touch the content, we don’t do it 90, we don’t do video decoding. We look at the IP, we look how the IP behaves. In fact, one of our customers is using a multicast workflow and he wanted to make sure that his source multicast and the free destination lookalike. So we did that exactly and he was very surprised. We put a flow monitor on the source and we were installed on the destination. We put a flow monitor on the destination, and we could compare and show him how the flow, the multicast flow on the source, how it is received on the other end, what is the differentiation, can trace the latency, the different arrival time of the multicast.

And in fact, what we found that was pleasant to see is that the original jitter that he was outputting, his multicast was even lower at the receiver. Meaning that the network basically smooth and improved his jitter performance as the multicast was received on the destination. So meaning that we can now focus on SRT streams. Everything has an IP, what we call in the industry, five tap characteristics that we can press and follow. Of course, we can generate alarms for the events. So we can then send emails notification for third party tools, put that on the web UI, and so on. Any question before we go forward?

Jim Jachetta:

I think we’re good.

Adi Rozenberg:

Okay. So let’s see some examples. And this is a very important example. This is a view of real-time capture with historical data. In this case, the customer came to me and said, “Look, we have the major problem that we have an alarm that happened a few hours back, and we need to see what happened to the network. Now we don’t know what happened. We tested, we run standard IT tools and we couldn’t find the problem.” So he said to me, he dialed me the exact hour. I went to our database, I dialed in the range to see what is happening. And here we could see that 7:00, or 4:20, that was in the local time, the network has changed. You can see here the RTT and the buffer emulation, and there is also the latency that we don’t see, has changed.

And that change caused packet loss that was beyond the protection level that we had, that caused ETR to 90 events to happen. But the network change also, the latency has changed completely on that network. Meaning that his buffer, he was using an SRT in this case, was inadequate to fulfill full-blown packet recovery, because he was trying to run a very low latency buffer, and meaning that he basically failed. He tried to run 120 millisecond buffer and as the network changed, basically it started to get hits. And you can see here on the graph itself, you can see those small spike hitting that beyond the 100 level on the buffer [inaudible 01:55:06]-

Jim Jachetta:

Right here, yeah. And then there’s a hit right there. It all kind of correlates. Yep, you can see it.

Adi Rozenberg:

Exactly. Once they basically cross over the 120 millisecond, he got hit on his SRT. So we were able to correlate those events and basically tell him, “Look, all you need to do is increase your latency buffer. Can you increase that for 120 to 200?” And in fact, eventually even change that to 700 millisecond, because we were able to find even bigger events that he didn’t even expect. But the idea is that this is the causation, this is the built it of the network to identify network related problems. He constantly thought that the problem that he had was to do with the vendor and the SRT protocol.

Jim Jachetta:

With the bonded cellular products, it is a little bit of trial and error. If we see a packet loss in the payload, in the video and the audio, we increase the latency and it’s kind of, you need a human being to do that. We can’t run-

Adi Rozenberg:

[inaudible 01:56:21].

Jim Jachetta:

Yeah, our modern products can go down to 250 millisecond buffering even over cellular, but that’s pretty aggressive. 5G connections can handle that. So it’s kind of like trial and error. You need to test it. So this is a much more scientific way to, you can actually see that the buffer is full and figure out why, and make an educated decision on the setup of your transport stream.

Adi Rozenberg:

Let’s go to the next slide. This is another example. The same customer. I will issue a paper about that and we will circulate that. We had four diverse paths that we thought that truly diverse path, not only that the customer used four different data centers and the circuit providers, but everything was designed for redundancy, and redundancy, and redundancy. So if there is an event, at least one link will survive and will not see the problem, or it’ll be time difference. But in fact, what we could see, and you can see the difference. This is exactly the same time, actually I have four of those, but it did not fit the slide deck. And you can see the difference. It’s exactly the same event happening, causing problem, exactly at the same time. Slightly different impact on every line, but it’s happening exactly at the same-

Jim Jachetta:

It’s similar. It’s similar, but yet different. And these are three independent circuits, and is this the connection that you found, the commonality, the single point of failure?

Adi Rozenberg:

This is, I cannot go into details, but there is a-

Jim Jachetta:

You’re under NDA, okay, okay.

Adi Rozenberg:

That is, I’m under NDA. There was a commonality here that we could not basically, but the thing that I want the audience to see is that the ability to detect and correlate is, it’s not only that you say, “Oh, I have a problem. I found a problem. I have maybe alarm here and alarm here.” I need to have the tools and the evidence to see that, I need to see what has changed, or what is correlate link A to link B to see what happened. Not just to say I had alarm on A, alarm on B, I need to see the difference. What if they are time synchronized? Then something is wrong in the setup. Now I have the tools and the means to go and figure out what do I need to do? Before the correlation, this specific customer worked for more than eight weeks to figure out what is going on. He didn’t have that smoking gun. This is the smoking gun. And that event happened time after time. Look at the magnitude of the event. It was like more than 80% packet loss.

Jim Jachetta:

Yeah. And it was very frustrating. You have no visibility into this black hole. I’m doing everything right. I have four redundant circuits. This problem shouldn’t be happening. And you were able to shed light, observe this in real time to shed light on the problem.

Adi Rozenberg:

Exactly. And how do you basically get, let’s dive into the information, let’s dive into the crops, and understand how do we see that? In the middle you see the RTT, you see there is a hit on the RTT, meaning that something slowed down that measurement of the RTT, meaning that the network has changed significantly that the RTT gets a hit. So there’s a deep depression in the RTT. You can see on the lower chart, you can see the cloud driver packet, which is the test stream that we’re running, basically shows that what arrived-

Jim Jachetta:

It collapsed. The pipe collapsed.

Adi Rozenberg:

Exactly. In this case the network ground, BGP reconstruction, bandwidth reallocation, and that slowed down our stream that we were constantly pushing, even did that twice. And it also did that on the RTT, which is not in our control, meaning that the network latency has changed. In this case, it’s changed for the better, because you see that it is lower, meaning that suddenly the network was faster, lower RTT means the network is faster.

Jim Jachetta:

Yeah, it seems counterintuitive, right? That if-

Adi Rozenberg:

Yeah, but here-

Jim Jachetta:

… the circuit collapsed, yeah, yeah.

Adi Rozenberg:

But it also blocked many of the packets to go through. And that is the problem. So here we can correlate events.

Jim Jachetta:

Very cool. Very cool.

Adi Rozenberg:

Let’s go forward. Let’s see if we have more.

Jim Jachetta:

Yeah, I think that’s it. So how does VidOvation, or how does a customer deploy this? What are the different… We discussed this, I know the answer to this question, but tell our audience how do we deploy? You mentioned a docker is one-

Adi Rozenberg:

We provide a Docker container that you basically can download from Docker.hub, and then you will contact Jim to get a certificate that you download the instruction. Basically, you can download that from the web. You download that once you have a valid license, or agreement with Jim and [inaudible 02:01:59], and you can install that. We then empower deploy cloud fronted and backend, and database in the AWS environment.

Jim Jachetta:

In the cloud, yeah.

Adi Rozenberg:

In the cloud, so you can have web UI to get that and you can run with it. As I said, Docker container. If your devices do not support Docker containers, which many all devices do not, you can put that on a Windows PC, run it on a virtual machine that can run a Ubuntu, then it’s Docker friendly. Or you can take a low cost [inaudible 02:02:36], put Ubuntu, it costs you $0 to install it. [inaudible 02:02:43] And then you put a Docker container. We provide support to install that, connect that to a local switch. You are up and running.

Jim Jachetta:

So the Ubuntu physical machine or the Docker all communicate to a cloud instance of Availink software, and that’s where the data is stored, that’s where the analysis happens. So a customer could have hundreds or thousands of these probes, as you call them, all streaming, all sending data to the cloud in real time.

Adi Rozenberg:

Exactly.

Jim Jachetta:

Yeah. I think this is something we’ve all needed. Does anyone have any questions? I know it’s the day before Thanksgiving. I don’t see any questions, so I will… Probably won’t happen until next week, but next week we’ll publish and transcribe this whole session. This is some really great information, Adi. I’m hoping to book some discovery calls, some demo meetings with some of our customers. We haven’t officially announced the Availink solution other than this webinar, but the customers I’ve talked to are like, “Oh man, I need this, I need this. I hope it’s not too expensive.” And it’s a great value and I think it’s one of those necessary tools that we all need in our toolbox, correct? Very good. Very good.

Adi Rozenberg:

Do we want to do a live demo? Let’s connect to a system.

Jim Jachetta:

Yes. Yes. Oh yeah, yeah. Yes. Let me give you back camera control. Hold on, let me stop. Stop my share now. You should have share back.

Adi Rozenberg:

Let me try to share. Maybe it would work this time. Hold on here. Let me know if you see the screen.

Jim Jachetta:

Perfect.

Adi Rozenberg:

You see the screen?

Jim Jachetta:

Yes, yes.

Adi Rozenberg:

Excellent. So this, just take the mouse here. So this is demo system that we’ve been asked to run on three different cloud, public cloud environment. We have AWS, we have Google services, GCP, and we have Azure. We also have two data center location, one in France, one in Strasbourg, which is near the French German border, and one in St. Louis, the gateway to the west. And let’s take just… Here this, I’m not generating traffic. So in this case, just let’s look at St. Louis to Azure. We can see on top that the network already has load balancer.

The reason why the lower connection has no port is because Azure obscures its make the path. But we will still see what is happening. So let’s jump into the link and let’s load the data. Oh, that’s nice. So what we can see is that, if you can see the data, we see that a constant error rate of almost 8% happening right now, between St. Louis data center running on cogent circuits, to Azure. You can see that the hops are changing. You can see that the hops changing basically to, just zoom in slightly so we can have better view. You can see the correlation between the hop changes and the RTT, meaning that every time the hop change, the network rearrange it itself, meaning that it’ll perform better or worse, meaning that the RTT and the latency will change for the better or worse with those changes. So that probably creates that high level of packet loss that we see.

We can see the latency here, the latency changes and spikes, meaning that this is below, you can see the constant average latency, but we can also see average changes, which are almost 100% in magnitudes than the standard latency. Which once again if we are jumping that creates packet loss that we will need to overcome. So if we were to use SRT, I would recommend to use a big buffer and multiple retransmissions as an example. Let’s just check another one. Let’s do AWS to AWS just for kicks. Okay, let’s see. AWS, Virginia going to California.

Jim Jachetta:

So what you’re doing there is you’re just clicking on that segment. Bam. Now I’m looking at that magnet from the-

Adi Rozenberg:

Exactly.

Jim Jachetta:

… graphical high level. Boom, I can go right in and see.

Adi Rozenberg:

Exactly. The system collects the data all the time. And just going through the web UI, seeing real-time data, you might see that we will come here and the network will change, because we discovered the new product. And look at this phenomenon. One agent is in North Virginia, one is in North Carolina. Look at the difference in the number of hops. The make of the top one is shorter than the reverse path. Look at that.

Jim Jachetta:

Interesting, Interesting.

Adi Rozenberg:

Okay. And look at-

Jim Jachetta:

We have no control over that. AWS may not even have control over it.

Adi Rozenberg:

Correct. But we need to be aware of that, because if you want to use such a circuit, we need to test it to be on top of that. So we will know-

Jim Jachetta:

Set our buffering intelligently, and our…

Adi Rozenberg:

Without bad mouthing anybody, today if you want to do a stream to AWS circuits, whether it’s SRT or ZIGZAG, you are on your own. You have no tools available to know what to do. You have rule of thumb rules. You need to be on top of that. You need to make your service successful. In this case, for instance, our advice is that testing the network, you need to have a buffer, which is at least 300 milliseconds. So that-

Jim Jachetta:

Right, but you want a little headroom, you don’t want to send it exactly to 300. So maybe you want to send it 10%, 25% higher than that, or 400.

Adi Rozenberg:

Exactly. Now if you want to validate whether we are bullshitting you or not, remember the old for [inaudible 02:09:33] correction technical recommendation, that said for [inaudible 02:09:37] correction to work, you need at least 120 milliseconds of buffer based on east coast to west coast buffer sites.

Yeah, this is that exact number. You can see that, 121. That is us testing the network and saying that based on the network conditions, that is the buffer that you need to do. That is what we are saying. That when there is not that much packet loss, but you definitely need to be on top of that, because you have events that the network is chomping and you need to be more than that. But that is just to show you.

Jim Jachetta:

Yeah. And then if you put your mouse right on that peak, you get the exact number of what that latency is.

Adi Rozenberg:

Exactly.

Jim Jachetta:

You move your cursor to… It’s-

Adi Rozenberg:

You can move your cursor-

Jim Jachetta:

… 239, yeah, yeah, yeah.

Adi Rozenberg:

You can do a full page. You can watch it like this. If you-

Jim Jachetta:

Get really, really fine resolution, yeah.

Adi Rozenberg:

You can extract that as a CSV, as a PDF, all the standard things that one might expect.

Jim Jachetta:

Very cool.

Adi Rozenberg:

Let’s look at just for the kicks. I’m just having fun. I don’t know what this is a live system, I could not prepare it, so I’m just having fun.

Jim Jachetta:

Let me see my-

Adi Rozenberg:

Look at the opposite, the path back. Actually better than the one going-

Jim Jachetta:

Yeah, yeah.

Adi Rozenberg:

So basically you need to be on top of things to validate your network. I’m just trying to find a curious link. Let’s take France to Google. Let’s go from France to Google. Let’s see that one. Oh, that is, this is more…

Jim Jachetta:

Oh, look at that. So one path is clean, but then there’s a multi hop path.

Adi Rozenberg:

Yeah, exactly. So you can see here spikes happening. I’m just playing-

Jim Jachetta:

And we’re not saying anything good or bad about Google or AWS. It’s just telling you how to set your parameters, your buffer size. You go in, instead of guessing and changing that buffer depth by trial and error, we can do it intelligently.

Adi Rozenberg:

Exactly. This is not a system. If one would want to connect from France to Google, it would try to connect to the nearest Google availability zone. All I want to show is that, look at this long distance, look at the problem that you can see. That is why we’re demonstrating that. So now I’m looking at Google to AWS, and once again, we see problems. We see buffer emulation changing. All the things that you can discover, you can look at, in this case, we are looking one hour back. We can look at, let’s say-

Jim Jachetta:

But that circuit is usable if we increase the buffering, right?

Adi Rozenberg:

Exactly.

Jim Jachetta:

Or if it’s bad enough, then we can determine we don’t want to use that-

Adi Rozenberg:

Exactly.

Jim Jachetta:

… that provider, or that segment.

Adi Rozenberg:

So right now, I’m giving you the way to look at six hours. Let’s see what happened in the last six hours, how good it was. Immediately you can see, well, the jitter has changed. Look at how the network changed. During those six hours, we had some kind of a major event happening. And that event was because the network has changed. You can see that definitely the jitter went from a high value to lower value, meaning that the network between Google to Azure was optimized by somebody. It caused an event. But now different conditions can go bad. Of course, we don’t know, we need to be on top of that. That is the message.

Jim Jachetta:

And then like you said, historically, so it’s the day before the event, some rehearsal, we run your test 24 hours, and historically we can see where the peak of the RTT hit, where the peak of the jitter hits, or where the peak of the buffering occurred. And then we know, I mean, it’s not a crystal ball, right? If the event is seven days from now, things could be drastically there. But we got a pretty good probability of a historical snapshot of how to set up our circuit.

Adi Rozenberg:

Exactly. Basically people have the tendency to test once and hope for the best. So what I tested now might change. We need to be on top of that. If you tested the network now and it’s an 18 millisecond, it might change, jump to 100 millisecond in an hour. If you are not aware of that, you will fail. Your service will fail.

Jim Jachetta:

Right, right.

Adi Rozenberg:

That’s the message. You need to be on top of things, because they are definitely changing.

Jim Jachetta:

Here’s the thing I thought of too. I’m a sports league, a production truck, and I’m going to the same venues several times throughout the year. I can look at my historical data from last event last year and make an intelligent decision of what’s the worst I’ve seen in the last year, two years, whatever. If you have that length of historical data, and then if some anomaly does crop up, that data is stored for the next time. So we can make sure it doesn’t happen.

Adi Rozenberg:

Exactly. You can compare, you can make decisions, everything. And you can just-

Jim Jachetta:

So I was telling Fallon, Fallon’s in our marketing department, she set up the webinar, it’s like, “Adi only has 14 slides. It’ll be about a half hour, 45 minutes. I swear it won’t be one of Jim’s usual 90-minute webinars.” But I’m a geek for this stuff, but we could do this all day, Adi. But this is great stuff. Let me bring back up the, if you could stop sharing, let me just bring up the… I can take it back. Let me bring up the closing slide here. I need to drag you over here. So if any of you folks need to contact us. Oh, it went to the beginning. Sorry about that. Let me do this differently. Sorry, my producing skills are a little off today. There we go. And let me share, I’m not seeing the right screen as an option.

Well, so you contact VidOvation. We’ve just started our partnership with Adi and his team. We’re real excited. I think this is a very important topic. Certainly reach out to me. There’s my email address, our phone number. I can be reached at jimj@vidovation.com. You can call 949-777-5435, or visit our website. Adi, thank you so much. This was a lot of fun. I definitely want to have you back again. Maybe we can do a deeper dive. I’m sure we’ll be doing demos in the coming weeks. This is a real, real, real hot topic. Thank you so much and everyone have a good Thanksgiving. Thank you, Adi. Thank you, Fallon. Have a good holiday.

Adi Rozenberg:

Thank you Fallon, and have great weekend with family and friends.

Fallon:

Likewise. Good job, guys. Chat later.

Jim Jachetta:

Thanks, Adi. Thanks, Fallon. Bye-bye.

Adi Rozenberg:

Take care. Signing off.

Jim Jachetta:

Thanks everyone.

PART 4 OF 4 ENDS [02:17:50]

 

Continue Reading