Supercharge Dyte Calls using Media Add-Ons & Middlewares

In one of our previous blogs (https://dyte.io/blog/streams-blog/), we talked about video and audio middlewares using Streams API as well as the process of creating and using them with Dyte's SDK.

We have been able to provide customization and integration with third-party services such as Google Speech to Text and Symbl.ai transcriptions thanks to these middlewares. They have also made it easier to use the following features:

• Blur/change the background
• White-label meeting participant feeds
• Save a photo of a participant for bookkeeping purposes

Let's take a look at some of the media add-ons and middlewares we've created on top of the Streams API.

Dyte Video Background Manager

The background manager allows changing the background of the video feed of meeting participants. These backgrounds are useful when you don't want to reveal or show the actual setting.

Integration code:

// import/require the package
var DyteVideoBackgroundTransformer = require("@dytesdk/video-background-transformer")

// Initialize the video transformer
const videoBackgroundTransformer = await DyteVideoBackgroundTransformer.init({
  meeting: meeting,
  segmentationConfig: {
    pipeline: 'canvas2dCpu', // 'webgl2' | 'canvas2dCpu'
  }
 });

const imageUrl = '<image-url>';
 
// Add the middleware to the meeting object
meeting.self.addVideoMiddleware(
    await videoBackgroundTransformer.createStaticBackgroundVideoMiddleware(imageUrl)
);

/*
* NOTE: passed imageUrl must allow CORS to avoid tainting the canvas.
* You can find such images on <https://unsplash.com/> & <https://imgur.com>.
/* 

You can find the NPM package at https://www.npmjs.com/package/@dytesdk/video-background-transformer.

Audio Transcription & Translation (Google STT)

People speak hundreds of thousands of languages around the world. Many times, we attend a seminar or conference where the speaker speaks a language that is not the common language of everyone in attendance. You don't have to worry about this when using Dyte.

Dyte provides an add-on that integrates Google Speech to Text with Dyte. This allows the speaker to speak in a language of his or her choice while translating it to others in their native languages.

Integration code:

The integration looks as simple as the following snippet.

import GoogleSpeechRecognition from '@dytesdk/google-transcription';

const speech = new GoogleSpeechRecognition({
    meeting,
    target: 'th',
    source: 'en-US',
    baseUrl: <backend-url>,
});

speech.on('transcription', async (data) => {
    // ... do something with transcription
});

speech.transcribe();

Google STT requires a subscription and a BE endpoint for security. You can find the frontend integration guide at https://github.com/dyte-in/google-transcription/tree/main/client & sever setup using NodeJS at https://github.com/dyte-in/google-transcription/tree/main/server.

Audio Transcription (Symbl.ai)

Symbl.ai allows you to generate not just the transcriptions but also offers conversation analytics, conversation topics, contextual insights, customer tracker, and summarization.

We have created a wrapper plugin that integrates Dyte with Symbl.ai thus avoiding the hassle for our clients.

Integration code:

import { activateTranscriptions } from '@dytesdk/symbl-transcription';
activateTranscriptions({
    meeting: meeting, // From DyteClient.init
    symblAccessToken: 'ACCESS_TOKEN_FROM_SYMBL_AI',
    noOfTranscriptionsToCache: 200,
    transcriptionsCallback: (allFormattedTranscriptions) => { 
    console.log(allFormattedTranscriptions); },
});

Note: You can get the symblAccessToken by referring to https://github.com/dyte-in/symbl-transcription#how-to-get-symblaccesstoken.

You can see the full guide at https://github.com/dyte-in/symbl-transcription.

Branding on the Participant feed

In the multi-company conferences and Dev conferences, the usual branding on the top left side of the meeting UI doesn’t make much sense. You might want to show branding per participant to indicate the company that they are representing. Dyte provides very simple media add-ons now to support such a use case.

Adding your company branding is now just a few lines of code.

Integration code:

async function CompanyBranding() {
    const loadLogo = (url) => {
            return new Promise((resolve) => {
                const image = new Image();
                image.src = url;
                image.onload = () => {
                    resolve(image);
                }
                image.crossOrigin = true;
            });
    }

/*
* Logo URL must allow cross origin retrival otherwise the video will get stuck.
* Tip: A logo with a transparent background looks much better.
* Issue: <https://developer.mozilla.org/en-US/docs/Web/HTML/CORS_enabled_image#security_and_tainted_canvases>
* Solution for S3: <https://aws.amazon.com/premiumsupport/knowledge-center/s3-configure-cors/>
*/
    const logo = await loadLogo('<img-url>');

    return (canvas, ctx) => {
        ctx.drawImage(logo, canvas.width - 200, 100, 100, 100)
    };
}

// Somewhere in your code, once you have the meeting object handy.
meeting.self.addVideoMiddleware(CompanyBranding);

You can find the middleware to add branding at https://github.com/dyte-in/streams-middleware-samples/blob/main/video/company-branding.js. We also provide a comprehensive guide on how to add it to your codebase at https://dyte.io/blog/streams-blog/.

Countdown Timer

Doctor-patient appointments, one-on-one horoscope readings, and paid sessions are time sensitive and are typically charged per minute or per hour. In all of these cases, it makes sense for the participants to see the remaining time in order to make the most of their meeting. We understand the importance of having a timer for the feed. Dyte provides a middleware to add such a timer directly to the video feed to address this.

Just a few lines and voila!! you can get the countdown started.

Integration code:

function CountdownTimer() {
    
    function convertSecondsToHHMMSS(sec) {
        let hours   = Math.floor(sec / 3600); // get hours
        let minutes = Math.floor((sec - (hours * 3600)) / 60); // get minutes
        let seconds = sec - (hours * 3600) - (minutes * 60); //  get seconds
        
        // add 0 if value < 10; Example: 2 => 02
        if (hours   < 10) {hours   = "0"+hours;}
        if (minutes < 10) {minutes = "0"+minutes;}
        if (seconds < 10) {seconds = "0"+seconds;}

        return hours+':'+minutes+':'+seconds; // Result in HH : MM : SS format
    }

    const meetingTill = new Date().getTime() + 1800000; // current + 30 mins

    return (canvas, ctx) => {
        let remainingSeconds = Math.floor((meetingTill - new Date().getTime()) / 1000);
        remainingSeconds = remainingSeconds > 0 ? remainingSeconds : 0; // fail-safe
    
        const timeRemainingInHHMMSS = convertSecondsToHHMMSS(remainingSeconds);
        ctx.font = '30px serif';
        ctx.fillText(timeRemainingInHHMMSS, canvas.width * 4 / 5, canvas.height / 5);
    };
}

meeting.self.addVideoMiddleware(CountdownTimer);

You can find the middleware to add a countdown timer at https://github.com/dyte-in/streams-middleware-samples/blob/main/video/meeting-countdown-timer.js. We also provide a comprehensive guide on how to add it to your codebase at https://dyte.io/blog/streams-blog/.

That was a quick set of 5 media add-ons that we provide out of many. We always encourage the Dyte community to build & share such add-ons to spread the love.

If you haven’t heard about Dyte yet, head over to https://dyte.io to learn how we are revolutionizing live video calling through our SDKs and libraries and how you can get started quickly on your 10,000 free minutes which renew every month. If you have any questions, you can reach us at support@dyte.io or ask our developer community.