本書是"計算語言學與語言科技原文叢書"中的一冊。文本和語音系統是計算語言學和語言技術的主要應用方向，涉及的范圍很好廣泛，幾乎包括所有的語言技術應用系統。本書涵蓋的內容很好豐富，既詳細介紹了各種文本和語音系統及其評測的基礎理論，又結合大量實例針對系統評測進行了明確的說明和解釋。內容深入淺出，適合有一定背景知識的初學者或相關領域的科研工作者閱讀，也可作為計算語言學專業教材。
戴布克耶Dybkjaer，丹麥南丹麥大學教授。
導讀1
Preface 15
Contributing Authors 17
Introduction 21
Laila Dybkjar， Holmer Hemsen and Wolfgang Minker
1 SPEECH AND SPEAKER RECOGNITION EVALUATION 1
Sadaoki Furui
1.Introduction 1
2.Principles of Speech Recognition 2
3.Categories of Speech Recognition Tasks 3
4.Evaluation of Speech Recognition Systems 6
5.Principles of Speaker Recognition 12
6.Categories of Speaker Recognition Tasks 13
7.Normalization and Adaptation Techniques 15
8.Evaluation of Speaker Recognition Systems 17
9.Factors Affecting the Performance and Evaluation Paradigm
Design for Speech and Speaker Recognition Systems 22
10.Svstem-Level Evaluation of Speech and Speaker Rognition23
11.Conclusion 24
References 24
2 EVALUATION OF SPEECH SYNTHESIS 29
Nick Campbell
1.Introduction 29
2.Components of Computer Speech 30
3.Evaluation Methodologies 36
4.Organised Evaluations and Assessment 41
5.Speaking to (and on Behalf of) People 45
6.Conclusion 45
References 48
3 MODELLING AND EVALUATING VERBAL AND NON-VERBAL
COMMUNICATION IN TALKING ANIMATED INTERFACE AGENTS
Bjarn Granstram and David House
1.Introduction 65
2.KTH Parametric Multimodal Speech Synthesis 67
3.Data Collection and Data-Driven Visual Synthesis 69
4.Evaluating Intelligibility and Information Presentation 71
5.Evaluating Visual Cues for Prominence 77
6.Evaluating Prosody and Interaction 80
7.Evaluating Visual Cues to Sentence Mode 84
8.Evaluation of Agent Expressiveness and Attitude 85
9.Agent and System Evaluation Studies 87
10.Future Challenges in Modelling and Evaluation 91
References 92
4 EVALUATING PART-OF-SPEECH TAGGING AND PARSING 99
Patrick Paroubek
1.POS Tagging 99
2.Parsing 102
3.Evaluation and Natural Language Processing 105
4.POS Tagging Evaluation Methodology 110
5.Methodology and Evaluation Measures for Parsing 114
6.Conclusion 117
References 118
5 GENERAL PRINCIPLES OF USER-ORIENTED EVALUATION 125
Margaret King
1.A Historical Note 126
2.What is User-Oriented Evaluation？ 128
3.A First Principle： Quality is Decided by Users 129
4.A Second Principle： Users do not Have the Same Needs 130
5.A Third Principle： Quality can be Characterized 135
6.A Fourth Principle： Quality can be Measured 148
7.Combining the Particular and the General： The Ideal 153
8.Conclusion 154
References 156
6 AN OVERVIEW OF EVALUATION METHODS IN TREC AD HOC INFO-
RMATION RETRIEVAL AND TREC QUESTION ANSWERING163
Simone Teufel
1.Introduction 163
2.Evaluation Criteria 169
3.Evaluation Metrics 172
4.Real-World Performance 182
5.Conclusion 183
References 185
7 SPOKEN DIALOGUE SYSTEMS EVALUATION 187
Niels Ole Bernsen， Laila Dybkjar and Wolfgang Minker
1.Introduction 187
2.Evaluation Methods and Criteria 188
3.Evaluation of the NICE Hans Christian Andersen Prototype 193
4.Evaluation of the SENECA Prototype 207
5.Conclusion 216
References 218
8 LINGUISTIC RESOURCES， DEVELOPMENT，AND EVALUATION OF
TEXT AND SPEECH SYSTEMS221
Christopher Cieri
1.Introduction 222
2.The Linguistic Resource Landscape 222
3.Background on Linguistic Data and Annotation 234
4.Data Planning for Technology Development and Evaluation 237
5.Finding Resources 240
6.Building Resources 242
7.Conclusion 259
References 260
9 TOWARDS INTERNATIONAL STANDARDS FOR LANGUAGE
RESOURCES263
Nancy Ide and Laurent Romary
1.Introduction 263
2.Background 265
3.The Linguistic Annotation Framework 268
4.Putting it All Together 278
5.Conclusion 282
References 283
Index 285
Akers, G.and Lennig, M.(1985).Intonation in Text-to-Speech Synthesis: Evaluation of Algorithms.Journal of the Acoustical Society of America, 77(6):2157-2165.
Benoit, C.and Pols, L.C.W.(1992).On the Assessment of Synthetic Speech.In Bailly, G., Benoit, C., and Sawallis, T., editors, Talking Machines: Theories, Models and Designs, pages 435-441, Elsevier, North Holland, Amsterdam, The Netherlands.
Benoit, C., van Erp, A., Grice, M., Hazan, V., and Jekosch, U.(1989).Multilin- gual Synthesiser Assessment Using Semantically Unpredictable Sentences.In Proceedings of the European Conference on Speech Communication and Technology (Eurospeech), volume 2, pages 633-636, Paris, France.
Bernstein, J.(1982).Evaiuating Synthetic Speech.In Proceedings of the NBS Workshop on Standardization for Speech I/O Technology, pages 87-91, Gaithersburg, Maryland, USA.
Bernstein, J.and Pisoni, D.B.(1980).Unlimited Text-to-Speech System: Deion and Evaluation of a Microprocessor Based Device.In Proceedings of the Internarional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 576-579, Denver, Colorado, USA,
Bladon, A.(1990).Evaluating the Prosody of Text-to-Speech Synthesizers.In Proceedings of Speech Tech, pages 215-220, New York, USA.
Boeffard, O., Cherbonnel, B., Emerard, F., and White, S.(1993).Automatic Segmentation and Quality Evaluation of Speech Unit Inventories for Concatenation-Based, Multilingual PSOLA Text-to-Speech Systems.In Proceedings of the European Conference on Speech Communication an.d Technology (Eurospeech), volume 2, pages 1449-1452, Berlin, Germany.
Boogaar, T.and Silverman, K, (1992).Evaluating the Overall Comprehensibility of Speech Synthesizers.In Proceedings of the International Conference on Spoken Language Processing (ICSLP), pages 1207-1210, Banff, Canada.
