Transplantation of Conversational Speaking Style with Interjections in Sequence-to-Sequence Speech Synthesis

Raul Fernandez¹, David Haws¹ Guy Lorberbom¹ Slava Shechtman¹ and Alexander Sorin¹
¹Authors are listed alphabetically and contributed equally

Submitted to Interspeech 2022

Inventory and definitions of Dialog Acts used to label events within dialogues

<agreement>: Confirmations of what the User said; often employs idiomatic lexical constructions.
<greeting>: Opening salutations, welcomings; often lexically formulaic.
<farewell>: Closing salutations, good byes; often lexically formulaic.
<empathy>: Negatively-valenced content, including empathetic apologies.
<instruction>: Explanatory or summarizing content; often used to give directives; may include lists; realized with a more objective and less affective tone.
<positive_feedback>: Positively-valenced content including feedback, good news, etc.
<surprise>: Expressions of surprise, realizations or discovery.
<thanks>: Formulaic exchanges of thanks.
<uncertainty>: Expressions of confusion; may precede <surprise> in some cases.
<waiting>: Holding requests, often phrased as polite questions, begging for some time (and understanding) from the User.

Orthographic forms and functions of interjections

<aha>: Exclamation of surprise, can be emphatic.
<oh>: General-purpose exclamation (often turn-initial) signaling realization, surprise, etc.
<hmm>: Interjection of uncertainty or confusion, but also surprise (often less prominent than the <aha> exclamation in the latter role).
<huh>: Aspirated hesitation conveying mild surprise (when with declarative intonation), occurs more naturally turn-initial.
<uh>: General-purpose oral hesitation/filler.
<uh-huh>: Oral backchannel to convey positive feedback or express agreement.
<um>: General-purpose nasal hesitation/filler.

Audio Samples

Base - multi-speaker multi-style TTS system (NAT2 with Hierarchical Prosody Controls), trained just on the recorded data.
Adv - The Base system with Adversarial Speaker Classifier applied on the Front-End Encoder output, trained just on the recorded data.
Aug - The Base system, trained on the recorded voice corpora, excluding the conversational voice corpus, and augmented with the VC-generated conversational data (for all the three neutral speakers in the training corpora).
Target Speaker - Target speaker, neutral style reference (generated by Adv)
Target Style - Conversational speaker, conversational style reference (generated by Base)

Speaker F1
Systems with Transpanted Style			References		Script
Base	Adv	Aug	Target Speaker	Target Style	Script
					<waiting> <uh>, one sec, let me take a look. </waiting> <agreement> Yes, definitely! All rooms include breakfast, which is served between 7 and 10 in the morning. </agreement>
					<surprise> <aha>, I think I figured it out! I think you're trying to use the wrong email address to log in. </surprise>
					Let me see. <waiting> <uh>, it looks like the item may be out of stock now, but let me put you on hold while I take a look at this, OK? </waiting>
					<positive_feedback> <oh> no problem! I'll show you! </positive_feedback> <instruction> Start by logging in, and then go to where it says "Personal Banking" at the top of the page. </instruction>
					<empathy> I apologize John. But unfortunately your order is already at the warehouse waiting to go out, and, <uh>, I can't make any changes. Sorry. </empathy>
					<greeting> Hi Ralph! I'm doing great, thanks! I'll be happy to help you solve your issue. What can I do for you today? </greeting>
					<waiting> I'm so sorry about the wait. Please bear with me while my system comes back. <hmm>, I'm afraid it's <emphasis> really </emphasis> slow today. </waiting>
					<instruction> I'm mailing you a link so that I can access your desktop remotely. Click on it, and check the box that says <emphasis> accept. <emphasis> Then I will take over. <instruction>
					<surprise> <oh> , I think I know what happened! You're using the wrong credentials. </surprise> Let me send your username to the email address we have on file.
					<empathy> You're just outside our free delivery area. I'm sorry about the inconvenience. </empathy> <positive_feedback> But I'll be happy to expedite your order and get it to you A.S.A.P. </positive_feedback>
					Let me look into it. <surprise> <aha>, I figured it out! Your first payment bounced, and we weren't able to withdraw from your account until 2 days after the due date. That would explain it! </surprise>
					<agreement> <uh-huh>, I agree! </agreement> <uncertainty> I'm not sure why there's an overdraft fee on your statement. I have to say I find this a bit strange. </uncertainty>
					<empathy> <huh> looks like we've made a mistake. My apologies. </empathy> I'm gonna go ahead and reimburse you for the price of the extra item.
					<positive_feedback>Well, I have some excellent news! The loan was approved at a fixed interest rate of only 3 percent! </positive_feedback>
					<uncertainty> That's confusing. I'm not sure what happened. Did you report one of them stolen or damaged by any chance? </uncertainty>
					Let me confirm your search. <agreement> You want a reservation for <emphasis> 4 </emphasis> people, at an <emphasis> Italian </emphasis> restaurant, around <emphasis> 8 </emphasis> PM. </agreement> Is that correct?
					<uncertainty> I don't see any work being done in your area, so, <um>, I can't explain your connectivity issues. </uncertainty> Let me troubleshoot this!
					<empathy> Yes, I'm afraid so. </empathy> <waiting> Sorry, <um>, do you mind if I place you on hold for another minute? Thanks for being so patient with me here. </waiting>
					<farewell> Thanks for giving us a call Steven. Enjoy the rest of your afternoon! Bye for now! </farewell>
					<positive_feedback> Glad to help! </positive_feedback> If you wish to stay on the line, there's a brief survey at the end to help us improve our service. <thanks> Thanks in advance if you choose to participate. </thanks>

Speaker F2
Systems with Transpanted Style			References		Script
Base	Adv	Aug	Target Speaker	Target Style	Script
					<waiting> <uh>, one sec, let me take a look. </waiting> <agreement> Yes, definitely! All rooms include breakfast, which is served between 7 and 10 in the morning. </agreement>
					<surprise> <aha>, I think I figured it out! I think you're trying to use the wrong email address to log in. </surprise>
					Let me see. <waiting> <uh>, it looks like the item may be out of stock now, but let me put you on hold while I take a look at this, OK? </waiting>
					<positive_feedback> <oh> no problem! I'll show you! </positive_feedback> <instruction> Start by logging in, and then go to where it says "Personal Banking" at the top of the page. </instruction>
					<empathy> I apologize John. But unfortunately your order is already at the warehouse waiting to go out, and, <uh>, I can't make any changes. Sorry. </empathy>
					<greeting> Hi Ralph! I'm doing great, thanks! I'll be happy to help you solve your issue. What can I do for you today? </greeting>
					<waiting> I'm so sorry about the wait. Please bear with me while my system comes back. <hmm>, I'm afraid it's <emphasis> really </emphasis> slow today. </waiting>
					<instruction> I'm mailing you a link so that I can access your desktop remotely. Click on it, and check the box that says <emphasis> accept. <emphasis> Then I will take over. <instruction>
					<surprise> <oh> , I think I know what happened! You're using the wrong credentials. </surprise> Let me send your username to the email address we have on file.
					<empathy> You're just outside our free delivery area. I'm sorry about the inconvenience. </empathy> <positive_feedback> But I'll be happy to expedite your order and get it to you A.S.A.P. </positive_feedback>
					Let me look into it. <surprise> <aha>, I figured it out! Your first payment bounced, and we weren't able to withdraw from your account until 2 days after the due date. That would explain it! </surprise>
					<agreement> <uh-huh>, I agree! </agreement> <uncertainty> I'm not sure why there's an overdraft fee on your statement. I have to say I find this a bit strange. </uncertainty>
					<empathy> <huh> looks like we've made a mistake. My apologies. </empathy> I'm gonna go ahead and reimburse you for the price of the extra item.
					<positive_feedback>Well, I have some excellent news! The loan was approved at a fixed interest rate of only 3 percent! </positive_feedback>
					<uncertainty> That's confusing. I'm not sure what happened. Did you report one of them stolen or damaged by any chance? </uncertainty>
					Let me confirm your search. <agreement> You want a reservation for <emphasis> 4 </emphasis> people, at an <emphasis> Italian </emphasis> restaurant, around <emphasis> 8 </emphasis> PM. </agreement> Is that correct?
					<uncertainty> I don't see any work being done in your area, so, <um>, I can't explain your connectivity issues. </uncertainty> Let me troubleshoot this!
					<empathy> Yes, I'm afraid so. </empathy> <waiting> Sorry, <um>, do you mind if I place you on hold for another minute? Thanks for being so patient with me here. </waiting>
					<farewell> Thanks for giving us a call Steven. Enjoy the rest of your afternoon! Bye for now! </farewell>
					<positive_feedback> Glad to help! </positive_feedback> If you wish to stay on the line, there's a brief survey at the end to help us improve our service. <thanks> Thanks in advance if you choose to participate. </thanks>