Receive DTMF/Speech Input Using Java

Overview

You can use speech input or dual-tone multi-frequency (DTMF) tones (a.k.a. Touch-Tone) to route callers or otherwise change call flows for applications such as interactive voice response (IVR), virtual assistants, and mobile surveys.

Prerequisites

To get started, you need a Plivo account — sign up with your work email address if you don’t have one already. You must have a voice-enabled Plivo phone number to receive incoming calls; you can rent numbers from the Numbers page of the Plivo console, or by using the Numbers API. If this is your first time using Plivo APIs, follow our instructions to set up a Java development environment and a web server and safely expose that server to the internet.

You must set up and install Java(Java 1.8 or higher) and Plivo’s Java SDK to handle incoming calls and callbacks. Here’s how.

How it works

Receive DTMF

This example shows a multilevel IVR phone application that uses digit press input captured using the GetInput XML element. A virtual assistant answers incoming calls and offers the caller three choices: “Press 1 for your account balance. Press 2 for your account status. Press 3 to speak to a representative.” If the caller enters 1 or 2, the application will retrieve the requested information and play the caller a text-to-speech message. If the caller presses 3, the application will redirect the caller to the second branch, which offers two new choices: “Press 1 for sales. Press 2 for support.” The application then connects the caller with the requested department.

Create a Spring application to detect DTMF input

Edit the PlivoVoiceApplication.java file in the src/main/java/com.example.demo/ folder and paste into it this code.

Note: Here, the demo application name is PlivoVoiceApplication.java because the friendly name we provided in the Spring Initializr was “Plivo Voice.”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
package com.example.Plivo;

import com.plivo.api.exceptions.PlivoXmlException;
import com.plivo.api.xml.*;
import com.plivo.api.xml.Number;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.*;

@SpringBootApplication
@RestController
public class PlivoVoiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(PlivoVoiceApplication.class, args);
    }

    // Welcome message, first branch
    String WelcomeMessage = "Welcome to the demo. Press 1 for your account balance. Press 2 for your account status. Press 3 to speak to a representative";
    // Message for second branch
    String RepresentativeBranch = "Press 1 for sales. Press 2 for support";
    // Message that Plivo reads when the caller does nothing
    String NoInput = "Sorry, I didn't catch that. Please hang up and try again";
    // Message that Plivo reads when the caller presses a wrong digit
    String WrongInput = "Sorry, that's not a valid input";
    
    @GetMapping(value = "/multilevelivr/", produces = { "application/xml" })
    
    public Response getInput() throws PlivoXmlException {
        Response response = new Response().children(
                new GetInput().action("https://<ngrok_identifier>.ngrok.io/multilevelivr/firstbranch/").method("POST")
                        .inputType("dtmf").digitEndTimeout(5).redirect(true).children(new Speak(WelcomeMessage)))
                .children(new Speak(NoInput));
        System.out.println(response.toXmlString());
        return response;
    }
    
    @RequestMapping(value = "/multilevelivr/firstbranch/", method = RequestMethod.POST, produces = {
            "application/xml" })
    public Response speak(@RequestParam("Digits") String digit) throws PlivoXmlException {
        System.out.println("Digit pressed:" + digit);
        Response response = new Response();
        if (digit.equals("1")) {
            response.children(new Speak("Your account balance is $20"));
        } else if (digit.equals("2")) {
            response.children(new Speak("Your account status is active"));
        } else if (digit.equals("3")) {
            response.children(new GetInput().action("https://<ngrok_identifier>.ngrok.io/multilevelivr/secondbranch/")
                    .method("POST").inputType("dtmf").digitEndTimeout(5).redirect(true)
                    .children(new Speak(RepresentativeBranch))).children(new Speak(NoInput));
        } else {
            response.children(new Speak(WrongInput));
        }
        System.out.println(response.toXmlString());
        return response;
    }
    
    @RequestMapping(value = "/multilevelivr/second/", produces = { "application/xml" }, method = RequestMethod.POST)
    public Response callforward(@RequestParam("Digits") String digit, @RequestParam("From") String from_number)
            throws PlivoXmlException {
        System.out.println("Digit pressed:" + digit);
        Response response = new Response();
        if (digit.equals("1")) {
            response.children(new Dial().action("https://<ngrok_identifier>.ngrok.io/multilevelivr/action/").method("POST")
                    .redirect(false).children(new Number("<number_1>")));
        } else if (digit.equals("2")) {
            response.children(new Dial().action("https://<ngrok_identifier>.ngrok.io/multilevelivr/action/").method("POST")
                    .redirect(false).children(new Number("<number_2>")));
        } else {
            response.children(new Speak(WrongInput));
        }
        System.out.println(response.toXmlString());
        return response;
    }
}

Control the gathering of DTMF inputs

You can improve DTMF collection by using attributes available for the GetInput XML element, such as digitEndTimeout, numDigit, finishOnKey, and executionTimeout.

digitEndTimeout sets the maximum time interval between successive digit inputs. The default value is auto and other allowed values are 2 to 10 seconds. If the user provides no new digits within the digitEndTimeout period, the digits entered to that point will be processed.

numDigits sets the maximum number of digits the user can provide on the current call. The default value is 32 and the allowed values are 1 to 32.

If the user provides more digits than the value of numDigits, Plivo will send only the number of digits specified as numDigits to the action URL; additional digit inputs will be ignored. For example, if numDigits is specified as “4” and the user enters five digits, the last digit will be ignored.

finishOnKey defines a key that users can press to submit the digits they entered. The default value is # and additional allowed values are 0-9, *, <empty string>, and ”none.” When you set the value to <empty string> or “none,” DTMF input collection ends depending on the digitEndTimeout or the numDigits attribute.

Note: These three attributes apply to input types dtmf and dtmf speech and do not apply to input type speech. If all three of these attributes are specified, the priority is for finishOnKey.

executionTimeout sets the maximum time during which Plivo detects input. You can use this timeout to tell the application to process the next element in the XML response when a user doesn‘t provide input during the call. The default value is 15 seconds, and allowed values are 5 to 60 seconds.

Detect speech input

The GetInput XML element can also capture speech input.

How it works

Receive speech

This example shows how to implement a simple IVR phone tree. A virtual assistant answers the call and offers the caller two choices: “Say sales to talk to a sales representative. Say support to talk to a support representative.”

If the caller says “sales,” the caller will be connected to a sales representative; if the caller says “support,” they will be connected to a support representative.

Code

Edit the PlivoVoiceApplication.java file in the src/main/java/com.example.demo/ folder and paste into it this code.

Note: Again, the demo application name is PlivoVoiceApplication.java because the friendly name we provided in the Spring Initializr was “Plivo Voice.”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
package com.example.Plivo;

import com.plivo.api.exceptions.PlivoXmlException;
import com.plivo.api.xml.*;
import com.plivo.api.xml.Number;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.*;

@SpringBootApplication
@RestController
public class PlivoVoiceApplication {
    public static void main(final String[] args) {
        SpringApplication.run(PlivoVoiceApplication.class, args);
    }

    // Welcome message, first branch
    String welcomeMessage = "Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative";
    // Message that Plivo reads when the caller does nothing
    String noInput = "Sorry, I didn't catch that. Please hang up and try again";
    // Message that Plivo reads when the caller speaks something unrecognized
    String wrongInput = "Sorry, that's not a valid input";
    
    @GetMapping(value = "/ivrspeech/", produces = { "application/xml" })
    
    public Response getInput() throws PlivoXmlException {
        final Response response = new Response().children(
                new GetInput().action("https://<ngrok_identifier>.ngrok.io/ivrspeech/firstbranch/").method("POST")
                        .interimSpeechResultsCallback("https://<ngrok_identifier>.ngrok.io/ivrspeech/firstbranch/")
                        .interimSpeechResultsCallbackMethod("POST").inputType("speech").redirect(true)
                        .children(new Speak(welcomeMessage)))
                .children(new Speak(noInput));
        System.out.println(response.toXmlString());
        return response;
    }
    
    @RequestMapping(value = "/ivrspeech/firstbranch/", produces = {
            "application/xml" }, method = RequestMethod.POST)
    public Response callforward(@RequestParam("Speech") final String speech,
            @RequestParam("From") final String fromNumber) throws PlivoXmlException {
        System.out.println("Speech Input is:" + speech);
        final Response response = new Response();
        if (speech.equals("sales")) {
            response.children(
                    new Dial().callerId(fromNumber).action("https://<ngrok_identifier>.ngrok.io/ivrspeech/action/")
                            .method("POST").redirect(false).children(new Number("<number_1>")));
        } else if (speech.equals("support")) {
            response.children(
                    new Dial().callerId(fromNumber).action("https://<ngrok_identifier>.ngrok.io/ivrspeech/action/")
                            .method("POST").redirect(false).children(new Number("<number_2>")));
        } else {
            response.children(new Speak(wrongInput));
        }
        System.out.println(response.toXmlString());
        return response;
    }
}

Speech recognition attributes

Speech models

Different applications may benefit from different automatic speech recognition (ASR) models, which you can specify using the the GetInput XML element‘s speechModel attribute. By default, it has a value of default, which is suitable for long-form audio, such as dictation, but you can also try command_and_search for shorter audio clips, such as when you expect callers to use voice commands or voice search, or phone_call, if you want to transcribe audio from a phone call. Explore the models and see which works best for your use case.

Example XML:

<Response>
<GetInput action="https://<yourdomain>.com/action/" method="POST" inputType="speech" speechModel="command_and_search" redirect="true">
<Speak>Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative</Speak>
</GetInput>
<Speak>Sorry, I didn't catch that. Please hang up and try again later.</Speak>
</Response>

Hints

You can use the hints attribute to potentially improve speech transcription results by defining words and phrases that are common in your use case. For example, a call center where callers use voice commands to connect to various departments can use the names of the departments as hints.

Allowed values: a non-empty string of comma-separated phrases
Limitations are:
- Phrases per request: 500
- Characters per phrase: 100
- Characters per request: 10,000

Example XML:

<Response>
<GetInput action="https://<yourdomain>.com/action/" method="POST" inputType="speech" hints="sales,support" redirect="true">
<Speak>Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative</Speak>
</GetInput>
<Speak>Sorry, I didn't catch that. Please hang up and try again later.</Speak>
</Response>

Controlling the gathering of speech input

You can improve the functionality of speech input collection by using GetInput XML attributes such as speechEndTimeout, language, profanityFilter, and executionTimeout.

speechEndTimeout sets the time that Plivo waits for more speech input after silence is detected. The default value is auto; other allowed values are 2 to 10 seconds. If the user doesn‘t provide new speech input within the speechEndTimeout period, the speech collected to that point will be processed.

language specifies the language and national/regional dialect of the audio to be recognized on calls. The default language for speech detection is en-US. You can choose your preferred language from the list of supported languages.

profanityFilter: If a user speaks any profane words, Plivo can filter them out during transcription if you set this attribute to true. The profanity filter applies only to single words — it doesn‘t work for a combination of words. The default value is false.

Note: These three attributes apply to input types speech and dtmf speech and do not apply to input type dtmf.

Example XML:

<Response>
<GetInput action="https://<yourdomain>.com/action/" method="POST" inputType="speech" speechEndTimeout="5" language="en-IN" profanityFilter="true" executionTimeout="25" redirect="true">
<Speak>Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative</Speak>
</GetInput>
<Speak>Sorry, I didn't catch that. Please hang up and try again later.</Speak>
</Response>

Real-time speech recognition

You can use the interimSpeechResultsCallback attribute to perform real-time speech recognition. If you specify a URL for your application server to this attribute, you can receive real-time callbacks of the user’s recognized speech while the user is still speaking on the call. Plivo sends the transcribed result to your server URL with attributes such as UnstableSpeech, Stability, StableSpeech, and SequenceNumber.

UnstableSpeech holds the interim transcribed result of the user’s speech, which may be refined when more speech is collected from the user.
Stability is an estimate of the likelihood that the recognizer will not change its guess about the interim UnstableSpeech result. Values range from 0.0 (completely unstable) to 1.0 (completely stable).
StableSpeech holds the stable transcribed result of the user’s speech.
SequenceNumber holds the sequence number of the interim speech callback, which helps you order incoming callback requests.

Example XML:

<Response>
<GetInput action="https://<yourdomain>.com/action/" method="POST" interimSpeechResultsCallback="https://<yourdomain>.com/interimcallback/" interimSpeechResultsCallbackMethod="POST" inputType="speech" redirect="true">
<Speak>Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative</Speak>
</GetInput>
<Speak>Sorry, I didn't catch that. Please hang up and try again later.</Speak>
</Response>

Data logging preferences

You can use the GetInput XML element’s log attribute to manage input logging preferences. It defaults to true, but if you define it to false, logging will be disabled and Plivo will not log digit and speech input.

Rate this page

🥳 Thank you! It means a lot to us!