DeathMark

Programmatically scan videos for points of interest for video streaming.

Problem

In 2020 like most people our company transition to more remote work and I started to invest in my home office to prepare for the coming onslaught of zoom meetings. Once I had a comfortable setup, I had everything I needed to start streaming outside of work as well because why not right? I started playing around with twitch and cutting my videos down for youtube.
Most of my streams were 2+ hours and while playing 2+ hours of games with your friends is an investment in relaxation and friendship. Watching back 2+ hours wasn't a time investment I wanted to make.

Conception

My game of choice is Valorant at the moment but this is true of most games; In the game there is a visual queue to show you have scored a point.

Because they are UI elements they are usually visually bright, consistently placed, and distinct from the background view of the game.

The score indicator is kind of like a flashing light, if I had something like a Light-Dependent Resistor, I could record when it flashed.

Application

I have a concept "check for increases in white in the video" MDN has a great example of how to check each frame of a video using HTML canvas. The example is set up like this:

ctx1 = Canvas 1 context ctx2 = Canvas 2 context

We are going to borrow their function and focusing on the computeFrame section. You can see below the RGB values for each pixel in each frame.

processor.computeFrame = function computeFrame() { //drawing the full frame to canvas this.ctx1.drawImage(this.video, 0, 0, this.width, this.height); //get the frame from canvas at 0 x and 0 y let frame = this.ctx1.getImageData(0, 0, this.width, this.height); let l = frame.data.length / 4; for (let i = 0; i < l; i++) { let r = frame.data[i * 4 + 0]; let g = frame.data[i * 4 + 1]; let b = frame.data[i * 4 + 2]; if (g > 100 && r > 100 && b < 43) frame.data[i * 4 + 3] = 0; } this.ctx2.putImageData(frame, 0, 0); return; }

In the example, above it's checking this range of color and if it falls within the threshold it will make it alpha instead thus creating a green screen or in this case yellow screen.

Simple enough, i will just check for white pixel in the area.

if (g > 240 && r > 240 && b < 240) { // 255,255,255 is white so 240 -> 255 is mostly white // is white pixel }

But the game has complex visuals and many elements would trigger just "white"

Every picture is made up of an almost unique amount of colors and shades so all I needed to do is get as close as possible to that unique number.

`` let skullFound = [] let white = [] let green = [] let red = [] for (let i = 0; i < l; i++) { let r = frame.data[i * 4 + 0]; let g = frame.data[i * 4 + 1]; let b = frame.data[i * 4 + 2]; if (isWhite(r,g,b)) white.push({r,g,b}) } if (isGreen(r,g,b)) green.push({r,g,b}) } if (isRed(r,g,b)) red.push({r,g,b}) } }

if(whiteThreshold(white.length) && greenThreshold(green.length) && redThreshold(red.length)) {
skullFound.push(video.currentTime)
white = []
green = []
red = []
}

``

After 30ish minutes of trial and error, I was able to get about a 99% accuracy rate at 2x speed with the videos tested with 1 main exception being, if the Character Sage is within the cropped section when she is shot with a sniper rifle... which is kinda rare.

Conclusion

While the current system is not perfect, It's a simplistic solution to the problem I was facing that can be built upon later.

I believe methods like the one above can be applied to many game videos. I look forward to finding more fun techniques in this area in the future.