Speech Recognition and Storyline

Yes, you can!

The Web Speech API enables you to incorporate voice data into web apps. The Web Speech API has two parts: SpeechSynthesis (Text-to-Speech), and SpeechRecognition (Asynchronous Speech Recognition.). Today, we will look on how to implement Speech Recognition functionality in Articulate Storyline 360.

Why would you want to do it? Well, maybe it is a language course or even a compliance training and you are creating a phone conversation role play.

Something to note: Web Speech API is an experimental technology and at the moment is only fully supported by Google Chrome. So make sure to change your browser, if you haven't done so already.

Enough words. Let's play!

Download Storyline 360 project file

So how to incorporate Speech Recognition scripts with Storyline? This all is really a credit to an amazing Web Speech API repo on GitHub containing demos to illustrate speech recognition capabilities.

To begin, create a new Storyline project, where all slides advance "By User".

1. First, let's create a slide, where we will check if user is using Chrome as this will prevent users try to use functionality on unsuported browsers.

2. Create a new number variable - isChrome with a default value "0".

3. Add a trigger to execute the following Javascript when timeline starts:

if (!('webkitSpeechRecognition' in window)) {
 console.log('User is not using Chrome');
} else {
    console.log('Proceed');
    var player = GetPlayer();
    player.SetVar("isChrome",'1');  
}

4. Add another trigger to jump to the next slide if isChrome changes with a condition isChrome to be equal to 1. The script above evaluates if user's browser supports Speech Recognition. If yes, it changes a variable, which in it's turn triggers Storyline to jump to the next screen. If this doesn't happen, user will be stuck on the slide where you can advice to use Chrome

5. Create a second slide, where all magic is going to happen!

6. Just create a text variable SpeechReceived and insert it on the slide.

7. Add a new button that will trigger speech recognition with the following script:

  var SpeechRecognition = SpeechRecognition || webkitSpeechRecognition;
  var SpeechGrammarList = SpeechGrammarList || webkitSpeechGrammarList;
  var SpeechRecognitionEvent = SpeechRecognitionEvent || webkitSpeechRecognitionEvent;

  var recognition = new SpeechRecognition();
  var speechRecognitionList = new SpeechGrammarList();
  recognition.grammars = speechRecognitionList;
  recognition.lang = 'en-GB';
  recognition.interimResults = false;
  recognition.maxAlternatives = 1;

  recognition.start();

  recognition.onresult = function(event) {

    var speechResult = event.results[0][0].transcript;
 //return speech and change storyline variable with a result
    var player = GetPlayer();
    player.SetVar("SpeechReceived",speechResult);  
    console.log('Confidence: ' + event.results[0][0].confidence);
  }

  recognition.onspeechend = function() {
    recognition.stop();
    console.log('Speech ended!');
  }

  recognition.onerror = function(event) {
 console.log('Error occurred in recognition: ' + event.error);
  }
  
  recognition.onaudiostart = function(event) {
      //Fired when the user agent has started to capture audio.
      console.log('SpeechRecognition.onaudiostart');
  }
  
  recognition.onaudioend = function(event) {
      //Fired when the user agent has finished capturing audio.
      console.log('SpeechRecognition.onaudioend');
  }
  
  recognition.onend = function(event) {
      //Fired when the speech recognition service has disconnected.
      console.log('SpeechRecognition.onend');
  }
  
  recognition.onnomatch = function(event) {
      //Fired when the speech recognition can't recognise speech
      console.log('SpeechRecognition.onnomatch');
  }
  
  recognition.onsoundstart = function(event) {
      //Fired when any sound — recognisable speech or not — has been detected.
      console.log('SpeechRecognition.onsoundstart');
  }
  
  recognition.onsoundend = function(event) {
      //Fired when no sound present
      console.log('SpeechRecognition.onsoundend');
  }
  
  recognition.onspeechstart = function (event) {
      //Fired when speech starts
      console.log('SpeechRecognition.onspeechstart');
  }
  recognition.onstart = function(event) {
      //Fired when the speech recognition service has begun listening
      console.log('SpeechRecognition.onstart');
  }

All done! You can now publish your project, the script above, when triggered will launch voice recognition. Value received will change SpeechReceived variable. There are multiple additional events in the script, that you can also explore.