Go to the
first,
previous,
next,
last section,
table of contents.
The utterance structure lies at the heart of Festival. This chapter
describes its basic form and the functions available
to manipulate it.
Festival's basic object for synthesis is the utterance. An
represents some chunk of text that is to be rendered as speech. In
general you may think of it as a sentence but in many cases it wont
actually conform to the standard linguistic syntactic form of a
sentence. In general the process of text to speech is to take an
utterance which contaisn a simple string of characters and convert it
step by step, filling out the utterance structure with more information
until a waveform is built that says what the text contains.
The processes involved in convertion are, in general, as follows
- Tokenization
-
Converting the string of characters into a list of tokens. Typically
this means whitespace separated tokesn of the original text string.
- Token identification
-
identification of general types for the tokens, usually this is trivial
but requires some work to identify tokens of digits as years, dates,
numbers etc.
- Token to word
-
Convert each tokens to zero or more words, expanding numbers,
abbreviations etc.
- Part of speech
-
Identify the syntactic part of speech for the words.
- Prosodic phrasing
-
Chunk utterance into prosodic phrases.
- Lexical lookup
-
Find the pronucnation of each word from a lexicon/letter to sound
rule system including phonetic and syllable structure.
- Intonational accents
-
Assign intonation accents to approrpiate syllables.
- Assign duration
-
Assign duration to each phone in the utterance.
- Generate F0 contour (tune)
-
Generate tune based on accents etc.
- Render waveform
-
Render waveform from phones, duration and F) target values, this
itself may take several steps including unit selection (be they
diphones or other sized units), imposition of dsesired prosody
(duration and F0) and waveform reconstruction.
The number of steps and what actually happens may vary and is dependent
on the particular voice selected and the utterance's type,
see below.
Each of these steps in Festival is achived by a module which
will typically add new information to the utterance structure.
An utterance structure consists of a set of items which may be
part of one or more relations. Items represent things like words
and phones, though may also be used to represent less concrete objects
like noun phrases, and nodes in metrical trees. An item contains a set
of features, (name and value). Relations are typically simple lists of
items or trees of items. For example the the Word relation is a
simple list of items each of which represent a word in the utternace.
Those words will also be in other relations, such as the
SylStructure relation where the word will be the top of a tree
structure containing its syllables and segments.
Unlike previous versions of the system items (then called stream items)
are not in any particular relations (or stream). And are merely part of
the relations they are within. Importantly this allows much more general
relations to be made over items that was allowed in the previous
system. This new architecture is the continuation of our goal
of providing a general efficient structure for representing complex
interrelated utterance objects.
The architecture is fully general and new items and relations may
be defined at run time, such that new modules may use any relations
they wish. However within our standard English (and other voices)
we have used a specific set of relations ass follows.
- Token
-
a list of trees. This is first formed as a list of tokens found
in a character text string. Each root's daughters are the Word's
that the token is related to.
- Word
-
a list of words. These items will also appear as daughters (leaf nodes)
of the
Token relation. They may also appear in the Syntax
relation (as leafs) if the parser is used. They will also be leafs
of the Phrase relation.
- Phrase
-
a list of trees. This is a list of phrase roots whose daughters are
the
Word's within those phrases.
- Syntax
-
a single tree. This, if the probabilistic parser is called, is a syntactic
binary branching tree over the members of the
Word relation.
- SylStructure
-
a list of trees. This links the
Word, Syllable and
Segment relations. Each Word is the root of a tree
whose immediate daughters are its syllables and their daughters in
turn as its segments.
- Syllable
-
a list of syllables. Each member will also be in a the
SylStructure relation. In that relation its parent will be the
word it is in and its daughters will be the segments that are in it.
Syllables are also in the Intonation relation giving links to
their related intonation events.
- Segment
-
a list of segments (phones). Each member (except silences) will be leaf
nodes in the
SylStructure relation. These may also be in the
Target relation linking them to F0 target points.
- IntEvent
-
a list of intonation events (accents and bounaries). These are related
to syllables through the
Intonation relation as leafs on that
relation. Thus their parent in the Intonation relation is the
syllable these events are attached to.
- Intonation
-
a list of trees relating syllables to intonation events. Roots of
the trees in
Intonation are Syllables and their daughters
are IntEvents.
- Wave
-
a single item with a feature called
wave whose value
is the generated waveform.
This is a non-exhaustive list some modules may add other relations
and not all utterance may have all these relations, but the above
is the general case.
The primary purpose of types is to define which modules are to be
applied to an utterance. UttTypes are defined in
`lib/synthesis.scm'. The function defUttType defines which
modules are to be applied to an utterance of that type. The function
utt.synth is called applies this list of module to an utterance
before waveform synthesis is called.
For example when a Segment type Utterance is synthesized it needs
only have its values loaded into a Segment relation and a
Target relation, then the low level waveform synthesis module
Wave_Synth is called. This is defined as follows
(defUttType Segments
(Initialize utt)
(Wave_Synth utt))
A more complex type is Text type utterance which requires many
more modules to be called before a waveform can be synthesized
(defUttType Text
(Initialize utt)
(Text utt)
(Token utt)
(POS utt)
(Phrasify utt)
(Word utt)
(Intonation utt)
(Duration utt)
(Int_Targets utt)
(Wave_Synth utt)
)
The Initialize module should normally be called for all
types. It loads the necessary relations from the input form
and deletes all other relations (if any exist) ready for synthesis.
Modules may be directly defined as C/C++ functions and declared with a
Lisp name or simple functions in Lisp that check some global parameter
before calling a specific module (e.g. choosing between different
intonation modules).
These types are used when calling the function
utt.synth and individual modules may be called explicitly by
hand if required.
Because we expect waveform synthesis methods to themselves become
complex with a defined set of functions to select, join, and modify
units we now support an addition notion of SynthTypes like
UttTypes these define a set of functions to apply
to an utterance. These may be defined using the defSynthType
function. For example
(defSynthType Festival
(print "synth method Festival")
(print "select")
(simple_diphone_select utt)
(print "join")
(cut_unit_join utt)
(print "impose")
(simple_impose utt)
(simple_power utt)
(print "synthesis")
(frames_lpc_synthesis utt)
)
A SynthType is selected by naming as the value of the
parameter Synth_Method.
Duration the application of the function utt.synth there are
three hooks applied. This allows addition control of the synthesis
process. before_synth_hooks is applied before any modules are
applied. after_analysis_hooks is applied at the start of
Wave_Synth when all text, linguistic and prosodic processing have
been done. after_synth_hooks is applied after all modules have
been applied. These are useful for things such as, altering the volume
of a voice that happens to be quieter than others, or for example
outputing information for a talking head before waveform synthesis
occurs so preparation of the facial frames and synthesizing the waveform
may be done in parallel. (see `festival/examples/th-mode.scm' for
an example use of these hooks for a talking head text mode.)
A number of utterance types are currently supported. It is easy to add
new ones but the standard distribution includes the following.
Text
-
Raw text as a string.
(Utterance Text "This is an example")
Words
-
A list of words
(Utterance Words (this is an example))
Words may be atomic or lists if further features need to be specified.
For example to specify a word and its part of speech you
can use
(Utterance Words (I (live (pos v)) in (Reading (pos n) (tone H-H%))))
Note: the use of the tone feature requires an intonation mode that
supports it.
Any feature and value named in the input will be added to the Word
item.
Phrase
-
This allows explicit phrasing and features on Tokens to be specified.
The input consists of a list of phrases each contains a list of tokens.
(Utterance
Phrase
((Phrase ((name B))
I saw the man
(in ((EMPH 1)))
the park)
(Phrase ((name BB))
with the telescope)))
ToBI tones and accents may also be specified on Tokens but these will
only take effect if the selected intonation method uses them.
Segments
-
This allows specification of segments, durations and F0 target
values.
(Utterance
Segments
((# 0.19 )
(h 0.055 (0 115))
(@ 0.037 (0.018 136))
(l 0.064 )
(ou 0.208 (0.0 134) (0.100 135) (0.208 123))
(# 0.19)))
Note the times are in seconds NOT milliseconds. The format of
each segment entry is segment name, duration in seconds, and list of
target values. Each target value consists of a pair of point into the
segment (in seconds) and F0 value in Hz.
Phones
-
This allows a simple specification of a list of phones. Synthesis
specifies fixed durations (specified in
FP_duration, default 100
ms) and monotone intonation (specified in FP_F0, default 120Hz).
This may be used for simple checks for waveform synthesizers etc.
(Utterance Phones (# h @ l ou #))
Note the function SayPhones allows synthesis and playing of
lists of phones through this utterance type.
Wave
-
A waveform file. Synthesis here simply involves loading
the file.
(Utterance Wave fred.wav)
Others are supported, as defined in `lib/synthesis.scm' but are
used internally by various parts of the system. These include
Tokens used in TTS and SegF0 used by utt.resynth.
The module is the basic unit that does the work of synthesis. Within
Festival there are duration modules, intonation modules, wave synthesis
modules etc. As stated above the utterance type defines the set of
modules which are to be applied to the utterance. These modules in turn
will create relations and items so that ultimately a waveform is
generated, if required.
Many of the chapters in this manual are solely concerned with particular
modules in the system. Note that many modules have internal choices,
such as which duration method to use or which intonation method to
use. Such general choices are often done through the Parameter
system. Parameters may be set for different features like
Duration_Method, Synth_Method etc. Formerly the values
for these parameters were atomic values but now they may be the
functions themselves. For example, to select the Klatt duration rules
(Parameter.set 'Duration_Method Duration_Klatt)
This allows new modules to be added without requiring changes to
the central Lisp functions such as Duration, Intonation,
and Wave_Synth.
There are a number of standard functions that allow one to access parts
of an utterance and traverse through it.
Functions exist in Lisp (and of course C++) for accessing an utterance.
The Lisp access functions are
- `(utt.relationnames UTT)'
-
returns a list of the names of the relations currently created in
UTT.
- `(utt.relation.items UTT RELATIONNAME)'
-
returns a list of all items in
RELATIONNAME in UTT. This
is nil if no relation of that name exists. Note for tree relation will
give the items in pre-order.
- `(utt.relation_tree UTT RELATIONNAME)'
-
A Lisp tree presentation of the items
RELATIONNAME in UTT.
The Lisp bracketing reflects the tree structure in the relation.
- `(utt.relation.leafs UTT RELATIONNAME)'
-
A list of all the leafs of the items in
RELATIONNAME in
UTT. Leafs are defined as those items with no daughters within
that relation. For simple list relations utt.relation.leafs and
utt.relation.items will return the same thing.
- `(utt.relation.first UTT RELATIONNAME)'
-
returns the first item in
RELATIONNAME. Returns nil
if this relation contains no items
- `(utt.relation.last UTT RELATIONNAME)'
-
returns the last (the most next) item in
RELATIONNAME. Returns
nil if this relation contains no items
- `(item.feat ITEM FEATNAME)'
-
returns the value of feature
FEATNAME in ITEM. FEATNAME
may be a feature name, feature function name, or pathname (see below).
allowing reference to other parts of the utterance this item is in.
- `(item.features ITEM)'
-
Returns an assoc list of feature-value pairs of all local features on
this item.
- `(item.name ITEM)'
-
Returns the name of this
ITEM. This could also be accessed
as (item.feat ITEM 'name).
- `(item.set_name ITEM NEWNAME)'
-
Sets name on
ITEM to be NEWNAME. This is equivalent to
(item.set_feat ITEM 'name NEWNAME)
- `(item.set_feat ITEM FEATNAME FEATVALUE)'
-
set the value of
FEATNAME to FEATVALUE in ITEM.
FEATNAME should be a simple name and not refer to next,
previous or other relations via links.
- `(item.relation ITEM RELATIONNAME)'
-
Return the item as viewed from
RELATIONNAME, or nil if
ITEM is not in that relation.
- `(item.relationnames ITEM)'
-
Return a list of relation names that this item is in.
- `(item.relationname ITEM)'
-
Return the relation name that this item is currently being viewed as.
- `(item.next ITEM)'
-
Return the next item in
ITEM's current relation, or nil
if there is no next.
- `(item.prev ITEM)'
-
Return the previous item in
ITEM's current relation, or nil
if there is no previous.
- `(item.parent ITEM)'
-
Return the parent of
ITEM in ITEM's current relation, or
nil if there is no parent.
- `(item.daughter1 ITEM)'
-
Return the first daughter of
ITEM in ITEM's current relation, or
nil if there are no daughters.
- `(item.daughter2 ITEM)'
-
Return the second daughter of
ITEM in ITEM's current relation, or
nil if there is no second daughter.
- `(item.daughtern ITEM)'
-
Return the last daughter of
ITEM in ITEM's current relation, or
nil if there are no daughters.
- `(item.leafs ITEM)'
-
Return a list of all lefs items (those with no daughters) dominated
by this item.
- `(item.next_leaf ITEM)'
-
Find the next item in this relation that has no daughters. Note this
may traverse up the tree from this point to search for such an item.
As from 1.2 the utterance structure may be fully manipulated from
Scheme. Relations and items may be created and deleted, as easily
as they can in C++;
- `(utt.relation.present UTT RELATIONNAME)'
-
returns
t if relation named RELATIONNAME is present, nil
otherwise.
- `(utt.relation.create UTT RELATIONNAME)'
-
Creates a new relation called
RELATIONNAME. If this relation
already exists it is deleted first and items in the relation are
derefenced from it (deleting the items if they are no longer referenced
by any relation). Thus create relation guarantees an empty relation.
- `(utt.relation.delete UTT RELATIONNAME)'
-
Deletes the relation called
RELATIONNAME in utt. All items in
that relation are derefenced from the relation and if they are no
longer in any relation the items themselves are deleted.
- `(utt.relation.append UTT RELATIONNAME ITEM)'
-
Append
ITEM to end of relation named RELATIONNAME in
UTT. Returns nil if there is not relation named
RELATIONNAME in UTT otherwise returns the item
appended. This new item becomes the last in the top list.
ITEM item may be an item itself (in this or another relation)
or a LISP description of an item, which consist of a list containing
a name and a set of feature vale pairs. It ITEM is nil
or inspecified an new empty item is added. If ITEM is already
in this relation it is dereferenced from its current position (and
an emtpy item re-inserted).
- `(item.insert ITEM1 ITEM2 DIRECTION)'
-
Insert
ITEM2 into ITEM1's relation in the direction
specified by DIRECTION. DIRECTION may take the
value, before, after, above and below.
If unspecified, after is assumed. Note it is not recommended
to insert above and below and the functions item.insert_parent
and item.append_daughter should normally be used for tree building.
Inserting using before and after within daughters is
perfectly safe.
- `(item.append_daughter PARENT DAUGHTER)'
-
Append
DAUGHTER, an item or a description of an item to
the item PARENT in the PARENT's relation.
- `(item.insert_parent DAUGHTER NEWPARENT)'
-
Insert a new parent above
DAUGHTER. NEWPARENT may
be a item or the description of an item.
- `(item.delete ITEM)'
-
Delete this item from all relations it is in. All daughters of this
item in each relations are also removed from the relation (which may in
turn cause them to be deleted if they cease to be referenced by any
other relation.
- `(item.relation.remove ITEM)'
-
Remove this item from this relation, and any of its daughters. Other
relations this item are in remain untouched.
- `(item.move_tree FROM TO)'
-
Move the item
FROM to the position of TO in TO's
relation. FROM will often be in the same relation as TO
but that isn't necessary. The contents of TO are dereferenced.
its daughters are saved then descendants of FROM are
recreated under the new TO, then TO's previous
daughters are derefenced. The order of this is important as FROM
may be part of TO's descendants. Note that if TO
is part of FROM's descendants no moving occurs and nil
is returned. For example to remove all punction terminal nodes in
the Syntax relation the call would be something like
(define (syntax_relation_punc p)
(if (string-equal "punc" (item.feat (item.daughter2 p) "pos"))
(item.move_tree (item.daughter1 p) p)
(mapcar syntax_remove_punc (item.daughters p))))
- `(item.exchange_trees ITEM1 ITEM2)'
-
Exchange
ITEM1 and ITEM2 and their descendants in
ITEM2's relation. If ITEM1 is within ITEM2's
descendents or vice versa nil is returns and no exchange takes
place. If ITEM1 is not in ITEM2's relation, no
exchange takes place.
Daughters of a node are actually represented as a list whose first
daughter is double linked to the parent. Although being aware of
this structure may be useful it is recommended that all access go through
the tree specific functions *.parent and *.daughter*
which properly deal with the structure, thus is the internal structure
ever changes in the future only these tree access function need be
updated.
With the above functions quite elaborate utterance manipulations can
be performed. For example in post-lexical rules where modifications
to the segments are required based on the words and their context.
See section 13.8 Post-lexical rules for an example of using various
utterance access functions.
In previous versions items had a number of predefined features. This is
no longer the case and all features are optional. Particularly the
start and end features are no longer fixed, though those
names are still used in the relations where yjeu are appropriate.
Specific functions are provided for the name feature but they are
just short hand for normal feature access. Simple features directly access
the features in the underlying EST_Feature class in an item.
In addition to simple features there is a mechanism for relating
functions to names, thus accessing a feature may actually call a
function. For example the features num_syls is defined as a
feature function which will count the number of syllables in the
given word, rather than simple access a pre-existing feature. Feature
functions are usually dependent on the particular realtion the
item is in, e.g. some feature functions are only appropriate for
items in the Word relation, or only appropriate for those in the
IntEvent relation.
The third aspect of feature names is a path component. These are
parts of the name (preceding in .) that indicated some
trversal of the utterance structure. For example the features
name will access the name feature on the given item. The
feature n.name will return the name feature on the next item
(in that item's relation). A number of basic direction
operators are defined.
n.
-
next
p.
-
previous
nn.
-
next next
pp.
-
previous
parent.
-
daughter1.
-
first daughter
daughter2.
-
second daughter
daughtern.
-
last daughter
first.
-
most previous item
last.
-
most next item
Also you may specific traversal to another relation relation, though
the R:<relationame>. operator. For example given an Item
in the syllable relation R:SylStructure.parent.name would
give the name of word the syllable is in.
Some more complex examples are as follows, assuming we are starting
form an item in the Syllable relation.
- `stress'
-
This item's lexical stress
- `n.stress'
-
The next syllable's lexical stress
- `p.stress'
-
The previous syllable's lexical stress
- `R:SylStructure.parent.name'
-
The word this syllable is in
- `R:SylStructure.parent.R:Word.n.name'
-
The word next to the word this syllable is in
- `n.R:SylStructure.parent.name'
-
The word the next syllable is in
- `R:SylStructure.daughtern.ph_vc'
-
The phonetic feature
vc of the final segment in this syllable.
A list of all feature functions is given in an appendix of this
document. See section 32 Feature functions. New functions may also be added
in Lisp.
In C++ feature values are of class EST_Val which may be a string,
int, or a float (or any arbitrary object). In Scheme this distinction
cannot not always be made and sometimes when you expect an int you
actually get a string. Care should be take to ensure the right matching
functions are use in Scheme. It is recommended you use
string-append or string-match as they will always work.
If a pathname does not identify a valid path for the particular
item (e.g. there is no next) "0" is returned.
When collecting data from speech databases it is often useful to collect
a whole set of features from all utterances in a database. These
features can then be used for building various models (both CART tree
models and linear regression modules use these feature names),
A number of functions exist to help in this task. For example
(utt.features utt1 'Word '(name pos p.pos n.pos))
will return a list of word, and part of speech context for each
word in the utterance.
See section 26.2 Extracting features for an example of extracting sets
of features from a database for use in building stochastic models.
A number of functions are available to allow an utterance's
structure to be made available for other programs.
The whole structure, all relations, items and features may be
saved in an ascii format using the function utt.save. This
file may be reloaded using the utt.load function. Note the
waveform is not saved using the form.
Individual aspects of an utterance may be selectively saved. The
waveform itself may be saved using the function utt.save.wave.
This will save the waveform in the named file in the format specified
in the Parameter Wavefiletype. All formats supported by
the Edinburgh Speech Tools are valid including nist, esps,
sun, riff, aiff, raw and ulaw. Note
the functions utt.wave.rescale and utt.wave.resample may
be used to change the gain and sample frequency of the waveform before
saving it. A waveform may be imported into an existing utterance with
the function utt.import.wave. This is specifically designed to
allow external methods of waveform synthesis. However if you just wish
to play an external wave or make it into an utterance you should
consider the utterance Wave type.
The segments of an utterance may be saved in a file using the function
utt.save.segs which saves the segments of the named utterance in
xlabel format. Any other stream may also be saved using the more
general utt.save.relation which takes the additional argument of
a relation name. The names of each item and the end feature of each
item are saved in the named file, again in Xlabel format, other features
are saved in extra fields. For more elaborated saving methods you can
easily write a Scheme function to save data in an utterance in whatever
format is required. See the file `lib/mbrola.scm' for an example.
A simple function to allow the displaying of an utterance in
Entropic's Xwaves tool is provided by the function display.
It simply saves the waveform and the segments and sends appropriate
commands to (the already running) Xwaves and xlabel programs.
A function to synthesize an externally specified utterance is provided
for by utt.resynth which takes two filename arguments, an xlabel
segment file and an F0 file. This function loads, synthesizes and plays
an utterance synthesized from these files. The loading is provided by
the underlying function utt.load.segf0.
Go to the first, previous, next, last section, table of contents.
