Protocol Documentation

Standard audio format

Sample rate in Hertz of the audio data This field is optional for `FLAC` and `WAV` audio files and required for all other audio formats.

Id of the audio requested (Note that this could be session_id to request the inbound audio resource)

For multi-channel audio, this is the channel number being referenced. Range is from 0 to N. Default channel 0 will be used if not specified

Number of milliseconds from the beginning of the audio to return. Default is from the beginning

Maximum number of milliseconds to return. A zero value returns all available audio (from requested start point). Default is all audio, from start point

Binary audio data that was requested

For multi-channel audio, this is the channel number being referenced.

In case of large audio, data will be split and there will be multiple AudioPullResponse messages. final_data_chunk field is set to true for the last message

Binary audio data to be added to the audio resource

Streamed binary audio data to be added to the session audio resource

Returns a block of audio data from an audio resource.

ASR interaction to associate this dtmf_key with

DTMF key press to be added to interaction stream for processing. Valid keys are 0-9, A-F, *, #

Event returned form Vad (AudioManager)

Session Events used to report errors to the API user

A grammar URL to be loaded

A string containing the raw grammar text

Reference to a previously defined "global" grammar Note: label must consist of letters, digits, hyphens, underscores only

Reference to a previously defined "session" grammar Note: label must consist of letters, digits, hyphens, underscores only

Reference to a "builtin" voice grammar

Reference to a "builtin" DTMF grammar

Optional label assigned to grammar, used for error reporting Note: label must consist of letters, digits, hyphens, underscores only

Log Event Timestamp (UTC)

can be either a VadEvent or a SessionEvent

The label of a previously defined global phrase list

Optional interaction object being referenced

String containing event information

The interaction object being referenced

The type of event this message represents

The offset in milliseconds from the beginning of the audio resource that this event occurred

String containing event information

Used to specify the type of settings to request

The language selector the specified grammar e.g.: "en-US", "de-DE" or dialect independent "en", "de", etc.)

Reference label for global grammar Note: label must consist of letters, digits, hyphens, underscores only

A grammar URL to be loaded

A string containing the raw grammar text

Optional grammar settings applied to this request

The status of the grammar load

The mode of the loaded grammar

The label for the loaded grammar

A list of strings containing word and phrase "hints" so that the transcriber recognition is more likely to recognize them. This can be used to improve the accuracy for specific words and phrases, for example, if specific commands are typically spoken by the user. This can also be used to add additional words or phrases to the transcriber's vocabulary.

A label that can be used to reference this list within a transcription request

The language selector describing which ASR resource will process request e.g.: "en-US", "de-DE" or dialect independent "en", "de", etc. Note that phrase lists are inherently language-independent, so this field is only used to direct which language-dependent resource will process the phrase load request

Optional settings specifying boost options for phrases

The status of the phrase list load.

The label for the phrase list.

Optional unique reference per request message. A UUID value will be auto generated if not supplied by client

Valid deployment identifier (UUID) to associate the request with

UUID related to the operator (entity or person making request)

Load a globally defined grammar

Load a globally defined phrase list

Get specified global default settings

Default session settings

Default interaction settings

Default grammar settings

Default recognition settings

Default normalization settings

Default VAD settings

Default CPA settings

Default tone detection settings

Default audio consume settings

Default logging settings

Optional settings specifying boost options for phrases

Will reset all of the settings to default

Reference to corresponding request correlation_id

Global event notification (typically errors)

Global default settings (which were requested)

Response to a global load grammar request

Response to a global load phrase list request

Default session settings

Default interaction settings

Default grammar settings

Default recognition settings

Default normalization settings

Default VAD settings

Default CPA settings

Default tone detection settings

Default audio consume settings

Default logging settings

Optional settings specifying boost options for phrases

The interaction object being referenced

The interaction object being referenced

The interaction object being referenced

Status of request

The interaction object being referenced

The interaction object being referenced

Status of request

Parameters for this interaction

Optional settings specifying audio to process for interaction

Optional settings related to voice activity detection

Optional settings related to all interactions

Interaction ID (uuid) that can be used during subsequent AMD processing

The language selector the specified grammars e.g.: "en-US", "de-DE" or dialect independent "en", "de", etc.

List of grammars to use, one for each root grammar to activate

Optional grammar settings to apply to this interaction

Optional recognition settings for this interaction

Optional settings related to voice activity detection

Optional settings specifying audio to process for interaction

Optional settings related to all interactions

Interaction ID (uuid) that can be used during subsequent ASR processing

Parameters for this interaction

Optional settings specifying audio to process for interaction

Optional settings related to voice activity detection

Optional settings related to all interactions

Interaction ID (uuid) that can be used during subsequent CPA processing

The language selector the specified grammars e.g.: "en-US", "de-DE" or dialect independent "en", "de", etc.

List of grammars to use, one for each root grammar to activate

Optional grammar settings to apply to this interaction

Input text to be parsed against specified grammar[s]

Maximum milliseconds to allow for a grammar parse. If this is exceeded, a timeout error will be raised. Range 0-10000000 (~166 minutes) Default: 10000 (10 seconds)

Optional settings related to all interactions

The interaction object being referenced by the request

Language to use for normalization (e.g. en-us)

All words in single string.

Optional settings specifying whether text normalization step should be performed on output of this interaction.

Optional settings related to all interactions

Interaction ID (UUID) that can be used during subsequent Normalize Text processing

Transcription language selector this request. e.g.: "en-US", "de-DE" or dialect independent "en", "de", etc.

Optional phrase lists for interaction

If `true`, transcription will perform continuous recognition (continuing to wait for and process audio even if the user pauses speaking) until the client closes the input stream (gRPC API). This may return multiple FinalResult callback messages. If `false`, the recognizer will detect a single spoken utterance. When it detects that the user has paused or stopped speaking, it will return an FinalResult callback and cease recognition. It will return no more than one FinalResult. Default: false

Optional recognition settings for this interaction

Optional settings related to voice activity detection

Optional settings specifying audio to process for interaction

Optional settings specifying whether text normalization step should be performed on output of this interaction.

Optional settings specifiying boost options for phrases

Optional settings related to all interactions

Optional list of grammars to use during transcription when a grammar matches during transcription, the semantic results of the grammar will also be returned

Optional grammar settings for embedded grammars

Optional name of a language model (decoder) to use when processing transcription. Default is to not specify this, allowing engine to use default language decoder

Optional name of a acoustic model (encoder) to use when processing transcription. Default is to not specify this, allowing engine to use default language encoder

Optional custom postprocessing to enhance decoder functionality. Default is to not specify this, allowing engine to use default postprocessing

Interaction ID (uuid) that can be used during subsequent ASR processing

Synthesis language for this request (e.g.: "en-US", "de-DE", etc.)

SSML type request and parameters

Inline TTS definition (text and optional parameters)

Audio format to be generated by TTS Synthesis Note: this is not configurable at Session or Global level, since it is explicitly required for each interaction request.

Optional timeout to limit the maximum time allowed for a synthesis Default: 5000 milliseconds

Optional settings related to all interactions

Text to synthesize, can simple text, or SSML

Optional settings for voice synthesis.

URL from which to fetch synthesis request SSML

Enables or disables the verification of a peer's certificate using a local certificate authority file upon HTTPS requests. Set to false (disabled) to skip verification for trusted sites. Default: true

Interaction ID (uuid) that can be used during subsequent TTS processing

The interaction object being referenced

Create AMD interaction request

Create ASR interaction request

Create CPA interaction request

Create transcription interaction request

Create TTS interaction request

Create a grammar parse request

Interaction begin processing

Interaction finalize processing

Interaction request results

Create a normalize text request

Interaction cancel

Explicit request to close interaction

The interaction object being referenced

The interaction object being referenced

Requested results

Optional list of strings containing words and phrases "hints" so that the transcriber recognition is more likely to recognize them. This can be used to improve the accuracy for specific words and phrases, for example, if specific commands are typically spoken by the user. This can also be used to add additional words or phrases to the transcriber's vocabulary.

Optional reference to previously defined global phrase list(s)

Optional reference to previously defined session phrase list(s)

The bool value.

The bytes value.

The double value.

The float value.

The int32 value.

The int64 value.

The string value.

The uint32 value.

The uint64 value.

AMD result in the form of an ASR-type message.

Raw ASR output used to produce semantic interpretations

List of all possible semantic interpretations for given transcript.

List of the N best possible matches provided via ASR.

The modality of the input, for example, speech, dtmf, etc.

Language defined when creating the interaction.

All words in Phrase so far.

All words in single string.

Time in milliseconds since beginning of audio stream where recognition starts.

Length of transcript in milliseconds.

Overall confidence of the entire transcript.

CPA result in the form of an ASR-type message.

The interaction object being referenced

Final result for the specified interaction. Null if status error > 0

Final status of the interaction

Status code produced. Returns 0 on success. this is comming form the 'internal' result message and should be passed to the caller shall we include the error here ??? or better send it as a SessionEvent ??

Input string used during grammar parse

List of all possible semantic interpretations for given text.

The modality of the input, for example, speech, dtmf, etc.

Language defined when creating the interaction.

Set to true if more input on input text is valid of interaction grammars.

Type of token.

All data in token

Input word used to create segment.

Index to words in original input.

Output after Inverse Text normalization.

Token information used in Inverse Text normalization.

Data add for redaction.

Final output for segment.

Input string used for the text normalization request

Normalized result message

All segments in result.

Output after Inverse Text normalization.

Output after Inverse Text normalization and redacted.

Final output after Inverse Text normalization and punctuation and capitalization_normalization

Final output after Inverse Text normalization, punctuation and capitalization_normalization, and redaction

The interaction object being referenced

Partial result for the specified interaction

Redacted Personal Identifiable Information.

Type of redaction

Redaction Score

Results for an ASR interaction

Results for a transcription interaction

Results for a grammar parse interaction

Results for a TTS interaction

Result for a Normalize Text interaction

Result for an AMD interaction

Result for a CPA interaction

Structure containing Semantic Interpretation.

Json string containing Semantic interpretation.

The label of the grammar used to generate this Semantic Interpretation.

Value 0 to 1000 of how confident the ASR is that result is correct match

Tag Format of in grammar used to generate this Semantic Interpretation.

Raw input text for the interpretation

Name of the artifact being referenced

Offset in milliseconds to the named artifact

String containing warning message returned from synthesizer

Optional line indicating where the issue was detected

List of the N best possible matches provided via ASR.

Language defined when creating the interaction.

Raw ASR output which includes the transcript of the audio.

If results are to be normalized, Normalized Result is added here.

If enhanced transcription with grammars is used results are added here.

Format of returned audio.

Length of generated audio data.

Offsets in milliseconds to where in audio buffer each synthesized sentence begins.

Offsets in milliseconds to where in audio buffer each synthesized word begins.

Offsets to where in audio buffer each synthesized SSML mark begins.

Offsets to where in audio voice each synthesized begins.

List of any Synthesis warnings.

Time in milliseconds since beginning of audio where word starts.

Length of word in milliseconds.

String output of word.

Value 0 to 1000 on how confident the result is.

Deployment identifier associated to the session

Valid session identifier to attached to request

UUID related to the operator (entity or person making request)

Status of request

Status of request

Deployment identifier to associate the session with

Optional unique reference for session (must be UUID) A UUID value will be auto generated if not supplied by client

UUID related to the operator (entity or person making request)

Parameters for the inbound audio resource associated with the session

The language selector the specified grammar e.g.: "en-US", "de-DE" or dialect independent "en", "de", etc.

Reference label for session grammar Note: label must consist of letters, digits, hyphens, underscores only

A grammar URL to be loaded

A string containing the raw grammar text

Optional grammar settings applied to this request

The status of the grammar load

The mode of the loaded grammar

The label for the loaded grammar

A list of strings containing word and phrase "hints" so that the transcriber recognition is more likely to recognize them. This can be used to improve the accuracy for specific words and phrases, for example, if specific commands are typically spoken by the user. This can also be used to add additional words or phrases to the transcriber's vocabulary.

A label that can be used to reference this list within a transcription request

The language selector describing which ASR resource will process request e.g.: "en-US", "de-DE" or dialect independent "en", "de", etc. Note that phrase lists are inherently language-independent, so this field is only used to direct which language-dependent resource will process the phrase load request

The status of the phrase list load.

The label for the phrase list.

Optional unique reference per request message. A UUID value will be auto generated if not supplied by client

For session-specific requests

For audio-specific requests

For interaction-specific requests

For DTMF events (part of ASR interaction)

Creates a new session and returns its ID and session related messages through response streamed callback messages

Defines the inbound audio format for the session. Must be assigned before any audio is sent and cannot later be changed.

Attach to an existing session

Explicit request to close session

Set settings to be configured for session.

Get settings for session.

Load session-specific grammar

Load session-specific phrase list

Explicit request to cancel all session related interactions and processing in progress

Session identifier (will be returned from initial call)

Optional reference to corresponding request correlation_id

VAD event notification

Final result notification

Partial result notification

Session event notification (typically errors)

Response for explicit session close request

Response to audio pull request

Response to get settings for session.

Response to create AMD interaction request

Response to create ASR interaction request

Response to create CPA interaction request

Response to create TTS interaction request

Response to create a grammar parse request

Response to create a normalize text request

Response to interaction get settings request

Response to interaction request results

Response to create Transcription interaction request

Response for session load phrase list request

Response for session load grammar request

Response to interaction cancel

Response to explicit request to close interaction

Response for explicit session cancel request

Enabled answering machine beep detection Default: true

Which string is returned in response to an AMD beep detection Default: AMD

Enable fax tone detection Default: true

Which string is returned in response to a fax tone detection Default: FAX

Enable SIT detection Default: true

Which string is returned in response to specified SIT detection Default: "SIT REORDER LOCAL"

Which string is returned in response to specified SIT detection Default: "SIT VACANT CODE"

Which string is returned in response to specified SIT detection Default: "SIT NO CIRCUIT LOCAL"

Which string is returned in response to specified SIT detection Default: "SIT INTERCEPT"

Which string is returned in response to specified SIT detection Default: "SIT REORDER DISTANT"

Which string is returned in response to specified SIT detection Default: "SIT NO CIRCUIT DISTANT"

Which string is returned in response to specified SIT detection Default: "SIT OTHER"

Enable busy tone detection Default: true

Which string is returned in response to a busy tone detection Default: BUSY

Maximum number of milliseconds the tone detection algorithm should listen for input before timing out.

For multi-channel audio, this is the channel number being referenced. Range is from 0 to N. Default channel 0 will be used if not specified

Select which audio mode is used Default: AUDIO_CONSUME_MODE_STREAMING

Specify where audio consume starts when "streaming" mode is used Default: STREAM_START_LOCATION_STREAM_BEGIN

Optional offset in milliseconds to adjust the audio start point. Range: Value in milliseconds, positive or negative. Default: 0

Optional maximum audio to process. Value of 0 means process all audio sent Range: Positive value in milliseconds Default: 0

Maximum amount of speech for human residence classification Default: 1800

Maximum amount of speech for human business classification. Human speech lasting longer than this will be classified as unknown speech Default: 3000

Maximum amount of silence to allow before human speech is detected. If This timeout is reached, the classification will be returned as unknown silence. Default: 5000

Maximum amount of time the CPA algorithm is allowed to perform human or machine classification. Only use this if you understand the implications (lower accuracy). Default: 0 (disabled)

When true (enabled), certain ASR and TTS data will not be logged. This provides additional security for sensitive data such as account numbers and passwords that may be used within applications. Anywhere that potentially sensitive data would have been recorded will be replaced with _SUPPRESSED in the logs. Default: false

Optional data (i.e. could be string, JSON, delimited lists, etc.) set by user, for external purposes. Not used by LumenVox

Optional tag for logging. Reserved for future use.

The default tag-format for loaded grammars if not otherwise specified. Default: TAG_FORMAT_SEMANTICS_1_2006

Enables or disables the verification of a peer's certificate using a local certificate authority file upon HTTPS requests. Set to false (disabled) to skip verification for trusted sites. Default: true

Maximum milliseconds to allow for grammar loading. If this is exceeded, a timeout error will be raised. Range 1000-2147483647 (~600 hours) Default: 200000 (~3.333 minutes)

Compatibility mode for certain media server operations. Only change from the default if you understand the consequences. Range: 0-1 Default: 0

Optional settings related to all interactions

Optional settings defining how audio is consumed/used by the interaction

Optional Voice Activity Detection settings for interaction

Optional grammar settings for interaction

Optional recognition settings for interaction

Optional Call Progress Analysis settings for interaction

Optional Tone Detection (AMD) settings for interaction

Optional settings specifying which text normalization steps should be performed on output of interaction.

Optional settings specifying boost options for phrases

Logging verbosity setting Default: LOGGING_VERBOSITY_INFO

Set to true to enable inverse text normalization (going from spoken form → written form (e.g. twenty two → 22) Default: false

Set to true to enable punctuation and capitalization normalization Default: false

Set to true to enable redaction of sensitive information Default: false

Number of milliseconds text normalization should await results before timing out Possible values: 0 - 1000000 Default: 5000

Probability score boost raises or lowers the probability the words or phrases are recognized. A negative value lowers the probability the word is returned in results. Range: -10.0 to 5.0 (very probable) Default: 0

Maximum number of recognition hypotheses to be returned. Specifically, the maximum number of `NBest` messages within each `AsrInteractionResult`. Default: 1

Controls how aggressively the ASR trims leading silence from input audio. Range: 0 (very aggressive) to 1000 (no silence trimmed) Default: 970

When true, partial results callbacks will be enabled for the interaction Default: false

Confidence threshold. Range 0 to 1000; applies to grammar based asr interactions Default: 0

Number of milliseconds the ASR should await results before timing out Possible values: 0 - 100,000,000 Default: 10,000,000 (~2.7 hours)

Whether the session data should be archived when closed, for tuning and other diagnostic purposes Default: false

Optional data (i.e. could be string, JSON, delimited lists, etc.) set by user, for external purposes. Not used by LumenVox

Optional settings to be used for duration of session for all interactions created. These can be over-ridden at the interaction level

Optional call identifier sting used for CDR tracking. This is often associated with the telephony call-id value or equivalent.

Optional channel identifier sting used for CDR tracking. This is often associated with a telephony/MRCP channel SDP value or equivalent.

Optional delay interval for archiving in seconds Session data will persist more in redis before being written to database

Optional tag for logging. Reserved for future use.

Optional voice (if using simple text, or if not specified within SSML)

The strength of the emphasis used in the voice during synthesis. Possible Values: "strong", "moderate", "none" or "reduced".

The pitch of the audio being synthesized. Possible Values: A number followed by "Hz", a relative change, or one of the following values: "x-low", "low", "medium", "high", "x-high", or "default". See the SSML standard for details.

The contour of the audio being synthesized. Possible Values: Please refer to the SSML standard on pitch contour for details.

The speaking rate of the audio being synthesized. Possible Values: A relative change or "x-slow", "slow", "medium", "fast", "x-fast", or "default". See the SSML standard for details.

The duration of time it will take for the synthesized text to play. Possible Values: A time, such as "250ms" or "3s".

The volume of the audio being synthesized. Possible Values: A number, a relative change or one of: "silent", "x-soft", "soft", "medium", "loud", "x-loud", or "default". See the SSML specification for details.

The age of the voice used for synthesis. Possible Values: A non-negative integer.

The default TTS gender to use if none is specified. Possible Values: Either neutral (which uses the default), male, or female.

When `false`, all audio as specified in AudioConsumeSettings is used for processing. In streaming audio mode, InteractionFinalizeProcessing() would need be called to finish processing When `true`, VAD is used to determine when the speaker starts and stops speaking. When using VAD in batch audio mode, the engine will look for speech begin within the designated audio to process and will stop processing audio when end of speech is found, which may mean that all audio loaded is not processed.

Maximum silence, in ms, allowed while waiting for user input (barge-in) before a timeout is reported. Range: -1 (infinite) to positive integer number of milliseconds Default: -1 (infinite)

After barge-in, STREAM_STATUS_END_SPEECH_TIMEOUT will occur if end-of-speech not detected in time specified by this property. This is different from the eos_delay_ms; This value represents the total amount of time a caller is permitted to speak after barge-in is detected. Range: a positive number of milliseconds or -1 (infinite) Default: -1 (infinite)

Determines noise reduction mode. Default: NOISE_REDUCTION_DEFAULT

A higher value makes the VAD more sensitive towards speech, and less sensitive towards non-speech, which means that the VAD algorithm must be more sure that the audio is speech before triggering barge in. Raising the value will reject more false positives/noises. However, it may mean that some speech that is on the borderline may be rejected. This value should not be changed from the default without significant tuning and verification. Range: Integer value from 0 to 100 Default: 50

Milliseconds of silence after speech before processing begins. Range: A positive integer number of milliseconds Default: 800

Determines how much louder the speaker must be than the background noise in order to trigger barge-in. The smaller this value, the easier it will be to trigger barge-in. Range: Integer range from 0 to 100 Default: 50

Accurate VAD depends on a good estimation of the acoustic environment. The VAD module uses the first couple frames of audio to estimate the acoustic environment, such as noise level. The length of this period is defined by this parameter. Range: A positive integer number of milliseconds. Default: 100

The volume required to trigger barge-in. The smaller the value, the more sensitive barge-in will be. This is primarily used to deal with poor echo cancellation. By setting this value higher (less sensitive) prompts that are not properly cancelled will be less likely to falsely cancel barge-in. Range: Integer range from 0 to 100 Default: 50

The length of audio to be wound back at the beginning of voice activity. This is used primarily to counter instances where barge-in does not accurately capture the very start of speech. The resolution of this parameter is 1/8 of a second. Range: A positive integer number of milliseconds Default: 480