The rise of video call applications in many domains also comes with some privacy considerations. A typical scenario worth looking at is screen sharing.
How do we prevent mistakenly sharing sensitive material during screen sharing in Telehealth, E-learning, and many other use cases?
Zoom provides an advanced screen-sharing functionality that doesn’t come with the regular browser screen-share dialog.
I will walk you through how to implement similar functionality which can be later integrated into your main Video Call application or solution.
Click here to see the full source code.
Selecting the Screen to Share
We use the browser media devices API to bring up the screen selection dialog. After this, we handle the selection of a portion of interest using CropperJs.
async function startCapture() {
try {
captureStream = await navigator.mediaDevices.getDisplayMedia();
const imageUrl = await getImageFromVideoStream(captureStream);
const imgElem = document.getElementById("previewImage");
imgElem.style.display = "block";
imgElem.src = imageUrl;
const videoDisplay = document.getElementById("screenshareDisplay");
videoDisplay.style.display = "block";
new Cropper(imgElem, {
zoomable: false,
restore: false,
crop(event) {
coordinates.x = event.detail.x;
coordinates.y = event.detail.y;
coordinates.height = event.detail.height;
coordinates.width = event.detail.width;
const generatedStream = processVideoTrack(captureStream);
videoDisplay.srcObject = generatedStream;
},
});
} catch (err) {
console.error(`Error: ${err}`);
}
}
Getting the Coordinates of the Area of Interest
To get the portion we want to share, we grab an image from the screen we selected and pass it to the Cropper Library. This way we can get the coordinates that reflect the selected screen.
async function getImageFromVideoStream(stream) {
const canvas = document.createElement("canvas");
if ("ImageCapture" in window) {
const videoTrack = stream.getVideoTracks()[0];
const imageCapture = new window.ImageCapture(videoTrack);
const bitmap = await imageCapture.grabFrame();
canvas.height = bitmap.height;
canvas.width = bitmap.width;
canvas.getContext("2d").drawImage(bitmap, 0, 0);
return canvas.toDataURL();
}
const video = document.createElement("video");
video.srcObject = stream;
return new Promise((resolve, reject) => {
video.addEventListener("loadeddata", async () => {
const { videoWidth, videoHeight } = video;
canvas.width = videoWidth;
canvas.height = videoHeight;
try {
await video.play();
canvas
.getContext("2d")
.drawImage(video, 0, 0, videoWidth, videoHeight);
return resolve(canvas.toDataURL());
} catch (error) {
return reject(error);
}
});
});
}
The image is then passed to the Cropper instance. At this point, we display the box for cropping the area of interest on the image. Dragging the box updates the coordinates of the selected area which we use to crop the video stream from the screen share, frame by frame.
new Cropper(imgElem, {
zoomable: false,
restore: false,
crop(event) {
coordinates.x = event.detail.x;
coordinates.y = event.detail.y;
coordinates.height = event.detail.height;
coordinates.width = event.detail.width;
const generatedStream = processVideoTrack(captureStream);
videoDisplay.srcObject = generatedStream;
},
});
Cropping the Video track from the Screenshare
This is where we see the insertable stream at play. We modify each frame by cropping to the selected screen portion coordinates we get from the cropper.
// Cropping each frame
function cropVideoFramesWithCoordinates(frame, controller) {
const newFrame = new window.VideoFrame(frame, {
visibleRect: {
x: getSampleAlignedCoordinates(Math.round(coordinates.x)),
y: getSampleAlignedCoordinates(Math.round(coordinates.y)),
width: getSampleAlignedCoordinates(Math.round(coordinates.width)),
height: getSampleAlignedCoordinates(Math.round(coordinates.height)),
},
});
controller.enqueue(newFrame);
frame.close();
}
// Insertable stream logic
function processVideoTrack(track) {
const mainTrack = track.getVideoTracks()[0] ?? track;
const generator = new window.MediaStreamTrackGenerator({
kind: "video",
});
const generatedStream = new window.MediaStream([generator]);
const processor = new window.MediaStreamTrackProcessor({
track: mainTrack,
});
processor.readable
.pipeThrough(
new window.TransformStream({
transform: cropVideoFramesWithCoordinates,
})
)
.pipeTo(generator.writable)
.catch((err) => {
// TODO: Figure out how to prevent this error
console.log("pipe error: ", { err });
});
return generatedStream;
}
Demo
Conclusion
This shows you the general approach to achieving partial screen share as we have in Zoom. You can always improve the UX to suit your use case.
In subsequent articles, I’ll demonstrate how this can be integrated into existing video call platforms.
Top comments (0)