What are MJPEG, H.264, and H.265? How do They Relate to Watching Security Cameras in a Browser?

How a patent war changed how you can view your security cameras.

There's three ways that professional security cameras record video (and one way that Google does ).

If you ever wondered why a cheap low resolution camera can work in any browser but every professional camera system only works in Internet Explorer and Safari, then this guide is for you.

MJPEG - Moving Pictures

MJPEG is a slow series of individually compressed pictures; It's not exactly video, in the way that we normally think of video.

MJPEG has one clear advantage to everything else: It works in any browser! The problem is that it takes about 5-20x the hard drive space as H.264, which means that long term data retention is absurdly expensive. It also take 5-20x the processing power of H.264, which is why it is usually limited to the lowest resolution cameras or HD cameras that only can record when they detect motion.

Because MJPEG files are so inefficient, MJPEG is really only used on cameras where you don't have to worry about storing lots of data. For example, if you have a consumer-grade doorbell camera, it is recording, on average, about 3 minutes of video - a day. People who have consumer-grade doorbell cameras, also tend to only watch a single camera at a time, so it doesn't really matter if it takes your computer 20x the resources to display that video. If you have a commercial-grade camera, watching say an office or restaurant, you are typically recording 9-20 hours a day, while cameras watching machinery or oil refineries might be recording 24 hours a day. Commercial grade cameras also need to be able to be viewed in a myriad of different grids and views. The difference in how much you have to record and how many videos you need to watch at once, creates a very big difference in the required video compression for devices that might seem to do exactly the same thing: record video.

Block Oriented Compression Example.

H.264 - The Golden Standard

H.264, H.265 and VP8/VP9/AV1/WebM are video in the way that we normally think of video. H.264/5 are even the type of files you watch when you are watching movies at the movie theater.

H.264, H.265 and VP8/VP9/AV1/WebM all have what are called "Golden Frames" which are 100% true images, and then use block oriented compression to define the differences from Frame A to Frame B. If part of Frame B differs from the Golden Frame then it is updated, if not it just uses the Golden Frame's info. This saves massive amounts of storage space without really losing anything of value. It also focuses nearly all of the processing power of the camera on the areas of the screen with activity, so you can record much higher resolution video in H.264/5 than MJPEG. Block Oriented Compression is great, because it reduces storage costs, allows you to stream higher quality, crisp clear video without taking too much data, and is what allows you to be able to watch things like Netflix, Youtube, or your surveillance video in HD. MJPEG video would never be able to load fast enough.

H.264 is used in 90% of surveillance cameras and is considered the industry standard.

H.264 is required for ONVIF support (ONVIF is the Open Video Network Interface Format), which is what different manufacturers devices to talk to each other. If you buy another brand camera and want to record it with a SCW NVR, you would need both devices to have ONVIF compliance for the cameras to talk to each other.

ONVIF compliance requires surveillance cameras to support H.264.

The Google / Mozilla Situation

Both H.264 and H.265 were the result of collaboration by many parties including Apple, Microsoft, CIsco, Dolby, and the Moving Picture Experts Group. Most of the movie and tech industry was involved with the creation of the patents which went into H.264, including Google and Mozilla (the maker of Firefox).

H.264 and H.265 require license fees to use them, and this makes adopting a technology difficult and expensive. Google argued that H.264 and H.265 should be open source, so that anyone can use them. At the time, the Moving Picture Experts Group wasn't enforcing the patent fees much anyways, since everyone was just saying "I'll let you use my patents royalty free, if you let me use your patents royalty free." H.264 was kinda free (at least if you had part of the patent contributions). This seemed like a waste to Google - and just to be clear, we think it was a waste. Paying lawyers and requiring complex negotiations to result in, what was usually, just the free trading of patents is a waste of money. Google and a couple other patent providers, including Mozilla, agreed to open source their patents to make it truly free. They thought everyone else would do so as well.

And they were wrong.

And then, Google lost their power. Not only did the other providers not open source H.264, they did the opposite - they raised the rates of licensing. The equilibrium had shifted. Before Google open sourced their patents, there were no major parties to collect fees from. Now, the other companies now had a major player to charge for the use of their patents. Google could no longer say "I'll let you use my patents royalty free, if you let me use your patents royalty free," since anyone could use their open source patents. Google and Firefox could no longer include H.264 support without paying fees. So, Google and Mozilla (the maker of the Firefox web browser) started working on a different block oriented compression called VP8 (and whcih would evolve into VP9, AV1, and WebM). Then Google blocked H.264 in it's web-browser Chrome to try to force all the other parties to open source their patents or use VP8/VP9/AV1/WebM.

This also didn't work.

Google largely failed to convince most companies to switch to VP8 compression because it took too long to develop (for example, a full decade later, there is still no camera hardware that can record in VP8/VP9/AV1/WebM natively, on the hardware level), and because it faced massive legal challenges. Yes, VP8/VP9/AV1/WebM all ended up being free and open source, but it wasn't free and open source for some time, and it was very unclear as to whether a brand who adopted it would get their pants sued off for patent infringement. Google has now admitted that VP8/VP9/AV1/WebM infringed on at least some of Apple's H.264 patents. For a while, it was also quite substandard in terms of compression - nowadays, VP9 is in the ballpark of H.265 (it is about 20% worse*), but ONVIF had already settled on H.264/H.265 as the standard.

Google backtracks and allows some use of H.264

Google and The Motion Picture Group reached an agreement to be able to use H.264 video in Youtube. The MP4 container format with the H.264 video codec is now supported by Chrome and Firefox again in some, but not all cases (although Chromium and Opera do not support the format).

(We say "supported in some cases" because in each Chrome build, Google frequently changes what h.264 profile(s) that its browser supports. This guide isn't meant to be super technical, but even inside h.264 compression you have different greater or lower levels of compression called "profiles." Most ONVIF cameras use either a manufacturer specific profile or the "advanced" h.264 profile. Lately (since Chrome build 88) Google only wants to support "base" h.264 compression). As of 9/12/2019 for the newer Admiral/Imperial systems, partial support for Chrome, Firefox, and updated Safari browsers has been added.

Sadly, the overlay for motion grids, line crossing, text overlays, etc are not supported by this agreement. As almost all security cameras have configurable text overlays telling you the time and date and name of the camera/location, this means that nearly all security cameras still cannot use Chrome/Firefox. The programming language that the entire security industry uses for those overlays is called NAPI. Chrome and Firefox once supported this plugin language, but don't support it anymore.

There's a War Going on, and the Security Industry is Stuck in the Middle

The security industry is caught in the middle of this, and this is why you'll notice that Mozilla Firefox / Google Chrome / Safari support is largely missing from most security camera systems and Microsoft Internet Explorer browser support is included. Because ONVIF rejected VP8/VP9/AV1/WebM, if you make a VP8/VP9/AV1/WebM camera, it can't talk to any other devices and people want the ability to use whatever device meets their needs.

There should be an open standard for video compression. This standard should be availible to use in every browser. We agree with Google on this very strongly and we would love for surveillance video to work in every browser, but there must be a legal open standard before anyone will adopt it. The way Google went about infringing patents made it impossible for anyone else to adopt VP8/VP9/AV1/WebM without significant legal risk, so no hardware manufacturers made hardware that adopted it.

Before any surveillance manufacturer will adopt another video compression standard, ONVIF must adopt it and hardware must be created that can create the file format directly (not just convert to it). There's zero chance that ONVIF will adopt a standard that infringed on their current partner's patents. There's zero chance that a security camera manufacturer will create a camera that converts the video to VP8/VP9/AV1/WebM when processor manufacturers will only create chips that natively records h.264/5.

But Google still wants to force everyone to use VP8/VP9/AV1/WebM and the Motion Picture Group still wants Google to pay royalties for Chrome to use each codex / profile.

So how do surveillance companies make it work? We write or lease our own software that has licensed the patents in H.264/5.

H.265 - The Future Standard

SCW uses H.265 in most of its new products. H.265 is the successor to H.264. It requires a plugin in all cases. Sadly, it still faces the same patent issues, and still doesn't have support in Firefox or Chrome. We're currently working on a piece of companion software that runs in the cloud, allows you to store video offsite, converts the video to whatever your browser supports, and performs more sophisticated video analytics than are possible without supercomputers. This software will work in Chrome / Firefox. It will have a subscription fee, however.

Why did SCW choose H.265?

The biggest and most important reason that we picked H.265 over VP8/VP9/AV1/WebM is that there is no camera chipset that can create VP8/VP9/AV1/WebM video! Google keeps trying to force video creators to store the video in the royalty free VP8/VP9/AV1/WebM formats, which have clear advantages of being open source and powerful, but no embedded chipset manufacturer can record in this format directly. You can use a program like Final Cut Pro to convert video into one of these open source standards, but there's not yet a way to record directly - at least not using a hardware accelerated method required for a low-power device like a security camera.

You should be able to record 20-40% more video with H.265 than H.264 and about 10x more than MJPEG.

In comparison to H.264, H.265 offers about double the data compression ratio at the same level of video quality, which means both substantially improved video quality at the same bitrate and lower storage requirements. H.265 saves hard drive space and increases the quality of an image by splitting the screen into even more adjustable grids than H.264. This focuses nearly all of the processing power of the camera on the areas of the screen with activity with a much higher accuracy than H264.

The processing power required to create H.265 is about half of what is required to make H.264 video, enabling 4K recording at 30 FPS instead of 15FPS.

The image quality differences of H.265 is most noticeable in situations such as attempting to watch your camera remotely if your internet upload speed is slow.

H.265 video loads better over cellular data speeds.

Our cameras are backwards compatible. If you are using our RTSP feeds to bring SCW video into independent commercial applications, switching to H.264 video encoding may be required. All SCW products can use H.264, or H.265. H.265 is approved by ONVIF, but is not required for ONVIF compliance.

*H.265 is still about 20% More Efficient than VP9.