text:
·· Composition (98)
·· Commissioned report (926)
·· Other (993)
Published in 1978-2014 (239)
In preparation (1)
Does not have full text (257)
Marisa Merz: Actions, Interactions and Performative Sculpture
Kittler, T., 7 Jun 2019, Entrare nell'opera: Processes and Performative Attitudes in Arte Povera. Meyer-Stoll, C. (ed.). Vaduz: Kunstmuseum Liechtenstein
Research output: Chapter in Book/Report/Conference proceeding › Other chapter contribution
Childe Harold's Pilgrimage I and II
Minta, S. M. J., 2018, Oxford Handbook of Lord Byron. Oxford University Press
Archaeology and palaeolandscapes of the continental shelf: An introduction
Bailey, G., Harff, J. & Sakellariou, D., 1 Jun 2017, Under the Sea: Archaeology and Palaeolandscapes of the Continental Shelf. Bailey, G., Harff, J. & Sakellariou, D. (eds.). Heidelberg: Springer, p. 1-17 17 p. (Coastal Research Library; vol. 20).
Music perception and performance in virtual acoustic spaces
Brereton, J., 7 Apr 2017, Body, Sound and Space in Music and Beyond: Multimodal Explorations. Wöllner, C. (ed.). Routledge , (SEMPRE Studies in The Psychology of Music).
Morgan, C. L., 2017, The Interactive Past: Archaeology, Heritage, and Video Games. Mol, A., Ariese-Vandemeulebroucke, C., Boom, K. & Politopoulos, A. (eds.). Sidestone Press, p. 233-235
Life-writing and the Victorians
Broughton, T., Aug 2016, The Oxford Handbook of Victorian Literary Culture . John, J. (ed.). Oxford: Oxford University Press
Discovering Relational Structure in Program Synthesis Problems with Analogical Reasoning
Swan, J. & Krawiec, K., 1 Jul 2016, Genetic Programming Theory and Practice XIV. Riolo, R., Tozier, B. & Goldman, B. (eds.). Ann Arbor, USA: Springer, (Genetic and Evolutionary Computation).
Chemometric Methods for Improved Food Safety and Traceability
Wilson, J. C., Jan 2016, UK Success Stories in Industrial Mathematics. Aston, P., Mullholland, A. & Tant, K. (eds.). Springer Science Business Media, p. 1228-1234 7 p. 27
Death as a Biological Category
Holland, S. M., 2016, Handbook of the Philosophy of Medicine. Schramme, T. & Edwards, S. (eds.). Springer
Evolving Diverse Strategies Through Combined Phenotypic Novelty and Objective Function Search
Smith, D., Tokarchuk, L. & Fernando, C., 30 Apr 2015, Applications of Evolutionary Computation: 18th European Conference, EvoApplications 2015, Copenhagen, Denmark, April 8-10, 2015, Proceedings. Springer, p. 344-354 11 p.
Lodge, P. & Stoneham, T., 11 Feb 2015, Locke and Leibniz on Substance. Routledge, p. 1-7 7 p.
A Practical Guide to Regime Switching in Financial Economics
Clacher, I., Freeman, M., Hillier, D., Kemp, M. & Zhang, Q., 2015, Quantitative Financial Risk Management: Theory and Practice. Zopounidis, C. & Galariotis, E. (eds.). John Wiley Sons, Inc, p. 71-97 27 p.
Automatically Detecting Fallacies in System Safety Arguments
Kelly, T., Wells, S., Yuan, T. & Manandhar, S., 2015, Proceedings of the 15th International Workshop on Computation Models of Natural Argument. p. 1-10 10 p.
Localized Power Structure, Conflict and Poverty
Bhattacharya, A., Kar, A. & Nandi, A., 2015, The building process of local institutions and their implications for poverty. Deshpande, A. (ed.). p. 14-17 4 p.
Radio station jingles: How statistical learning applies to a special genre of audio logos
Muellensiefen, D., Egermann, H. & Burrows, S., 2015, Audio Branding Yearbook 2014-2015. Bronner, K., Hirt, R. & Ringe, C. (eds.). Baden-Baden: Nomos, p. 1-21 21 p.
SCJ-Circus: a refinement-oriented formal notation for Safety-Critical Java
Miyazawa, A. & Cavalcanti, A. L. C., 2015, REFINE Workshop.
Similarity Analysis from Limiting Quantum Walks
Curado, M., Escolano, F., Hancock, E., Nourbakhsh, F., Pelillo, M. & Hancock, E. R., 2015, Similarity-Based Pattern Recognition: Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, October 12–14, 2015, Proceedings. Feragen, A., Pelillo, M. & Loog, M. (eds.). Springer International Publishing, Vol. 9370. p. 38-53 16 p. (Lecture Notes in Computer Science).
Visualising Magnitude: Graphical Number Representations Help Users Detect Large Number Entry Errors
Borghouts, J. W., Soboczenski, F., Cairns, P. & Brumby, D. P., 2015, HFES 2015 International Annual Meeting.
Berger, R., Ensor, J., Wilson, K., Phukan, I. & Dasgupra, S., 2014, Community-based adaptation to climate change: Scaling it up. Schipper, E. L. F., Ayers, J., Reid, H., Huq, S. & Rahman, A. (eds.). Oxon: Earthscan from Routledge, p. 22-35 14 p.
Emerging lessons for community-based adaptation
Ensor, J., 2014, Community-based adaptation to climate change: Emerging lessons. Ensor, J., Berger, R. & Huq, S. (eds.). Rugby: Practical Action Publishing, p. 183-197 15 p.
Arroyo-Kalin, M. (ed.), Orton, D. C. (ed.) & Fuller, D. (ed.), 2014, Encyclopaedia of Global Archaeology. Smith, C. (ed.). New York: Springer
Formal Refinement in SysML
Miyazawa, A. & Cavalcanti, A. L. C., 2014, 11th International Conference on Integrated Formal Methods: Proceedings. Albert, E. & Sekerinski, E. (eds.). SPRINGER, p. 155-170 16 p. (Lecture Notes in Computer Science; vol. 8739).
Introduction to SoniHED Conference Proceedings
Pauletto, S., Cambridge, H. M. & Rudnicki, R., 2014, Proceedings of SoniHED - Conference on Sonification of Health and Environmental Data. York: University of York, 72 p.
Nitrogen deposition effects on ecosystem services and interactions with other pollutants and climate change
Erisman, J. W., Leach, A., Adams, M., Agboola, J. I., Ahmetaj, L., Alard, D., Austin, A., Awodun, M. A., Bareham, S., Bird, T. L. & McClean, C. J., 2014, Nitrogen deposition, critical loads and biodiversity. Sutton, M., Mason, K., Sheppard, L., Sverdrup, H., Haeuber, R. & Hicks, W. K. (eds.). Springer Netherlands, p. 493-505 13 p.
Pattern Recognition with Non-Euclidean Similarities
Hancock, E. R., Xu, E. & Wilson, R. C., 2014, Man-Machine Interactions 3. Gruca, D. A., Czach?rski, T. & Kozielski, S. (eds.). Springer International Publishing, p. 3-15 13 p. (Advances in Intelligent Systems and Computing; vol. 242).
Software Deployment for Distributed Embedded Real-Time Systems of Automotive Applications
Polzlbauer, F., Bate, I. & Brenner, E., 2014, Embedded and Real Time System Development - A Software Engineering Perspective. Springer, p. 305-328 24 p.
Stormy Seas: Anglo-American Negotiations on Ocean Surveillance
Robinson, S., 2014, The Surveillance Imperative: Geosciences during the Cold War and Beyond. Springer, p. 105-124 20 p.
The effects of atmospheric nitrogen deposition on terrestrial and freshwater biodiversity
Baron, J. S., Barber, M., Adams, M., Agboola, J. I., Allen, E. B., Bealey, W. J., Bobbink, R., Bobrovsky, M. V., Bowman, W. D., Branquinho, C. & Payne, R. J., 2014, Nitrogen deposition, critical loads and biodiversity. Sutton, M. A., Mason, K. E., Sheppard, L. J., Sverdrup, H., Haeuber, R. & Hicks, W. (eds.). SPRINGER, p. 465-480 16 p.
An overview of competition in UK health care
Street, A. D., Dec 2013, Competition in UK health care: reflections on an expert workshop. Charlesworth, A. & Kelly, E. (eds.). Institute for Fiscal Studies and Nuffield Trust
Poore, B., Meerzon, Y. & Meyer-Dinkgrafe, D., Dec 2013, Observing Theatre: Spirituality and Subjectivity in the Performing Arts. Meyer-Dinkgrafe, D. (ed.). Editions Rodopi B.V., p. 15-60 (Consciousness, Literature and the Arts; vol. 36).
Feedback from Contributors and Discussion
Poore, B., Meerzon, Y. & Meyer-Dinkgrafe, D., Jan 2013, Observing Theatre: Spirituality and Subjectivity in the Performing Arts. Meyer-Dinkgräfe, D. (ed.). Amsterdam: Editions Rodopi B.V., p. 169-215 47 p. (Consciousness, Literature and the Performing Arts; vol. 36).
A Continuous-Time Quantum Walk Kernel for Unattributed Graphs
Rossi, L., Torsello, A. & Hancock, E. R., 2013, Graph-Based Representations in Pattern Recognition. Kropatsch, W., Artner, N., Haxhimusa, Y. & Jiang, X. (eds.). Springer Berlin / Heidelberg, Vol. 7877. p. 101-110 (Lecture Notes in Computer Science).
A Fast Jensen-Shannon Subgraph Kernel
Bai, L. & Hancock, E. R., 2013, Image Analysis and Processing – ICIAP 2013. Petrosino, A. (ed.). PART 1 ed. Springer Berlin / Heidelberg, Vol. 8156. p. 181-190 (Lecture Notes in Computer Science).
A Graph Embedding Method Using the Jensen-Shannon Divergence
Bai, L., Hancock, E. R. & Han, L., 2013, Computer Analysis of Images and Patterns. Wilson, R., Hancock, E., Bors, A. & Smith, W. (eds.). PART 1 ed. Springer Berlin / Heidelberg, Vol. 8047. p. 102-109 (Lecture Notes in Computer Science).
A Quantum Jensen-Shannon Graph Kernel Using the Continuous-Time Quantum Walk
Bai, L., Hancock, E. R., Torsello, A. & Rossi, L., 2013, Graph-Based Representations in Pattern Recognition. Kropatsch, W., Artner, N., Haxhimusa, Y. & Jiang, X. (eds.). Springer Berlin / Heidelberg, Vol. 7877. p. 121-131 (Lecture Notes in Computer Science).
Analysis of Wave Packet Signature of a Graph
Aziz, F., Wilson, R. C. & Hancock, E. R., 2013, Computer Analysis of Images and Patterns. Wilson, R., Hancock, E., Bors, A. & Smith, W. (eds.). PART 1 ed. Springer Berlin / Heidelberg, Vol. 8047. p. 128-136 (Lecture Notes in Computer Science).
Analysis of the Schrödinger Operator in the Context of Graph Characterization
Suau, P., Hancock, E. R. & Escolano, F., 2013, Similarity-Based Pattern Recognition. Hancock, E. & Pelillo, M. (eds.). Springer Berlin / Heidelberg, Vol. 7953. p. 190-203 (Lecture Notes in Computer Science).
Attributed Graph Similarity from the Quantum Jensen-Shannon Divergence
Rossi, L., Torsello, A. & Hancock, E. R., 2013, Similarity-Based Pattern Recognition. Hancock, E. & Pelillo, M. (eds.). Springer Berlin / Heidelberg, Vol. 7953. p. 204-218 (Lecture Notes in Computer Science).
Complexity Fusion for Indexing Reeb Digraphs
Escolano, F., Hancock, E. R. & Biasotti, S., 2013, Computer Analysis of Images and Patterns. Wilson, R., Hancock, E., Bors, A. & Smith, W. (eds.). PART 1 ed. Springer Berlin / Heidelberg, Vol. 8047. p. 120-127 (Lecture Notes in Computer Science).
Eigenvector Sign Correction for Spectral Correspondence Matching
Haseeb, M. & Hancock, E. R., 2013, Image Analysis and Processing – ICIAP 2013. Petrosino, A. (ed.). PART 2 ed. Springer Berlin / Heidelberg, Vol. 8157. p. 41-50 (Lecture Notes in Computer Science).
Entropy and Heterogeneity Measures for Directed Graphs
Ye, C., Wilson, R. C., Comin, C., F. Costa, L. & Hancock, E. R., 2013, Similarity-Based Pattern Recognition. Hancock, E. & Pelillo, M. (eds.). Springer Berlin / Heidelberg, Vol. 7953. p. 219-234 (Lecture Notes in Computer Science).
Estimating Complex Refractive Index Using Ellipsometry
Saman, G. & Hancock, E. R., 2013, Image Analysis and Processing – ICIAP 2013. Petrosino, A. (ed.). PART 1 ed. Springer Berlin / Heidelberg, Vol. 8156. p. 201-210 (Lecture Notes in Computer Science).
Gaussian Wave Packet on a Graph
Aziz, F., Wilson, R. C. & Hancock, E. R., 2013, Graph-Based Representations in Pattern Recognition. Kropatsch, W., Artner, N., Haxhimusa, Y. & Jiang, X. (eds.). Springer Berlin / Heidelberg, Vol. 7877. p. 224-233 (Lecture Notes in Computer Science).
Graph Characteristics from the Schrödinger Operator
Suau, P., Hancock, E. R. & Escolano, F., 2013, Graph-Based Representations in Pattern Recognition. Kropatsch, W., Artner, N., Haxhimusa, Y. & Jiang, X. (eds.). Springer Berlin / Heidelberg, Vol. 7877. p. 172-181 (Lecture Notes in Computer Science).
Graph Characterization Using Gaussian Wave Packet Signature
Aziz, F., Wilson, R. C. & Hancock, E. R., 2013, Similarity-Based Pattern Recognition. Hancock, E. & Pelillo, M. (eds.). Springer Berlin / Heidelberg, Vol. 7953. p. 176-189 (Lecture Notes in Computer Science).
Great Post-Communist Experiment in Eastern Europe and Central Asia
Wagstaff, A. & Moreno-Serra, R., 2013, Scaling Up Affordable Health Insurance: Staying the Course. The World Bank
Heterogeneity Index for Directed Graphs
Ye, C., Wilson, R. C. & Hancock, E. R., 2013, Computer Analysis of Images and Patterns. Wilson, R., Hancock, E., Bors, A. & Smith, W. (eds.). PART 2 ed. Springer Berlin / Heidelberg, Vol. 8048. p. 424-431 (Lecture Notes in Computer Science).
Information-Theoretic Dissimilarities for Graphs
Escolano, F., Hancock, E. R., Liu, M. & Lozano, M., 2013, Similarity-Based Pattern Recognition. Hancock, E. & Pelillo, M. (eds.). Springer Berlin / Heidelberg, Vol. 7953. p. 90-105 (Lecture Notes in Computer Science).
Introduction to J. M. Coetzee
Attridge, D., 2013, A Between Almanach for the Year 2013. Wisniewski, T. (ed.). Gdansk: Maski, p. 25-26 2 p.
Linking Theories of Emotion-Induction to Applications in Audio Branding
Egermann, H., 2013, Audio Branding Yearbook 2012-2013. Bronner, K., Hirt, R. & Ringe, C. (eds.). Baden-Baden: Nomos, p. 155-166 12 p.
Manifold Learning and the Quantum Jensen-Shannon Divergence Kernel
Rossi, L., Torsello, A. & Hancock, E. R., 2013, Computer Analysis of Images and Patterns. Wilson, R., Hancock, E., Bors, A. & Smith, W. (eds.). PART 1 ed. Springer Berlin / Heidelberg, Vol. 8047. p. 62-69 (Lecture Notes in Computer Science).
The Archaeology of Poverty and Urban Slums
Symonds, J., 2013, Encyclopaedia of Global Archaeology. Smith, C. (ed.). New York: Springer
The Importance of Topology Evolution in NeuroEvolution: A Case Study Using Cartesian Genetic Programming of Artificial Neural Networks
Turner, A. J. & Miller, J. F., 2013, Research and Development in Intelligent Systems XXX. Bramer, M. & Petridis, M. (eds.). Springer International Publishing, p. 213-226 14 p. (Incorporating Applications and Innovations in Intelligent Systems XXI Proceedings of AI-2013, The Thirty-third SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence).
The Jorvik Centre
Symonds, J., 2013, Encyclopaedia of Global Archaeology. Smith, C. (ed.). New York: Springer, p. 4220-4221
The Society for Historical Archaeology
The Sonnet Refashioned: Muldoon's Maggot
Attridge, D., 2013, A Between Almanach for the Year 2013. Wisniewski, T. (ed.). Gdansk: Maski, p. 137-8 2 p.
The York Archaeological Trust
TurboTensors for Entropic Image Comparison
Escolano, F., Hancock, E. R., Bonev, B. & Lozano, M. A., 2013, Graph-Based Representations in Pattern Recognition. Kropatsch, W., Artner, N., Haxhimusa, Y. & Jiang, X. (eds.). Springer Berlin / Heidelberg, Vol. 7877. p. 51-60 (Lecture Notes in Computer Science).
Prosodies in Conversation
Ogden, R., 30 Nov 2012, Understanding Prosody: the role of context, function and communication. Niebuhr, O. (ed.). Berlin/New York: Walter de Gruyter & Co , p. 201-218
Simulation of materials and joints in FDTD using digital filter macro-models
Flintoft, I. D., Dawson, J. F., Xia, R., Porter, S. J. & Marvin, A., 2 Jul 2012, EuroEM 2012.
Saving with Excitement? The Premium Bond and Consuming Savers in Post-War Britain
Shin, H., Mar 2012, Genealogies of Curiosity and Material Desire: Proceedings of the History of Consumer History Conference 2012 . Shin, H. & Tanaka, Y. (eds.). Tokyo, p. 49-58 10 p.
An example of semi and unstructured interviews: illustrative box 6c
Corden, A., 2012, Understanding Research for Social Policy and Social Work. Becker, S., Bryman, A. & Ferguson, H. (eds.). 2nd ed. Bristol: The Policy Press, p. 294-295 2 p. (Understanding Welfare: Social Issues, Policy and Practice Series).
Concepts and challenges in measuring the performance of health care organizations
Smith, P. C. & Street, A., 2012, The Elgar Companion to Health Economics. Jones, A. (ed.). 2nd ed. Cheltenham: Edward Elgar, p. 341-349 9 p.
Dobrowolna nieprzejrzystosc. O dzialaniu i zxabieraniu glosu: Z Derekiem Attridge'em rozmawiaja Pawel Moscicki
Attridge, D., 2012, Wyostrzyć wzrok: J. M. Coetzee: sztuka, świat, polityka. Burzyńska, A. R. & Rapior, W. (eds.). Krakow: Homini and Fundacja Malta, p. 19-32
Evidence based policy and practice
Hardy, M., 2012, Understanding Research for Social Policy and Social Work: Themes, methods and approaches. Becker, S., Bryman, A. & Harry, F. (eds.). 2nd ed ed. The Policy Press, (Understanding Welfare; Social Issues, Policy and Practice Series).
Excavations at Gearraidh Bhailteas (Garryvaltos), Milton.
Symonds, J., 2012, From Machair to Mountains: Archaeological Survey And Excavation in South Uist.. Parker-Pearson, M. (ed.). Oxford, UK: Oxbow Books, p. 294-307 14 p. ( SEARCH Monograph ; vol. 4).
Free (and Fair) Markets without Capitalism: Political Values, Principles of Justice, and Property-Owning Democracy: Property-Owning Democracy: Rawls and Beyond
O'Neill, M., 2012, Property-Owning Democracy: Rawls and Beyond. Chichester: John Wiley & Sons, p. 75-100 26 p.
Gregor Reisch, Margarita Philosophica (cat. no. 8)
Vorholt, H., 2012, Senate House Library: University of London. Pressler, C. & Attar, K. (eds.). London: Scala
Measuring value: pharmacoeconomics theory and practice
Towse, A., Drummond, M. F. & Sorenson, C., 2012, The Oxford Handbook of The Economics of the Pharmaceutical Industry. Danzon, P. & Nicholson, S. (eds.). Oxford: Oxford University Press, p. 394-437 44 p.
Research with recently bereaved parents: illustrative box 2d
Corden, A., 2012, Understanding Research for Social Policy and Social Work. Becker, S., Bryman, A. & Ferguson, H. (eds.). 2nd ed. Bristol: The Policy Press, p. 77-78 2 p. (Understanding Welfare: Social Issues, Policy and Practice Series).
Social value judgements and ethical principles
Holland, S. M. & Tony, H., 2012, Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development..
The Risks of LSCITS: The Odds are Stacked Against Us
McDermid, J. A., 2012, Development, Operation and Management of Large-Scale Complex IT Systems: Proceedings of the 17th Monterey Workshop 2012, Oxford, UK.. Calinescu, R. & Garlan, D. (eds.). SPRINGER-VERLAG LONDON LTD, Vol. 7539 LNCS. p. 94-117 24 p.
Using verbatim quotations in writing up applied qualitative research
Corden, A. & Sainsbury, R. D., 2012, Understanding Research for Social Policy and Social Work. Becker, S., Bryman, A. & Ferguson, H. (eds.). 2nd ed. Bristol: The Policy Press, p. 356-358 3 p. (Understanding Welfare: Social Issues, Policy and Practice Series).
English Guildhalls
Giles, K., Dec 2011, The Archaeology of Medieval Europe : Twelfth to Sixteenth Centuries. Carver, M. & Klapste, J. (eds.). Aarhus: Aarhus University Press, Vol. 2. p. 396 1 p. (Archaeology of Medieval Europe; vol. 2, no. 9).
The methods of cost-effectiveness analysis to inform decisions about the use of health care interventions and programs
Walker, S., Sculpher, M. & Drummond, M., Apr 2011, The Oxford Handbook of Health Economics. Smith, P. C. & Glied, S. (eds.). Oxford: Oxford University Press, p. 733-758 26 p.
"We Walk Amid Crowds, Ride, Fly and Fall with the Hero": Avatars and Posthumanism
Ng, J., 2011, The Projected and the Prophetic. Copeland, J. J. (ed.). Vol. 176.
Aiding movement with sonification in "exercise, play and sport"
Honer, O., Hunt, A., Pauletto, S., Rober, N., Hermann, T. & Effenberg, A., 2011, The Sonification Handbook. Hermann, T., Hunt, A. & Neuhoff, J. (eds.). Logos Verlag Berlin, p. 525-553
Blocking and the Architecture of Grammar
Sells, P., 2011, Readings in Cognitive Science: Papers in Honor of Tom Wasow. Bender, E. & Arnold, J. (eds.). Stanford: CSLI Publications, p. 81-97 17 p.
Cat. nos 12, 13, 14, 16, 64, 274, 279, 363 and 364
Vorholt, H., 2011, Illuminated Manuscripts in Cambridge: A Catalogue of Western Book Illumination in the Fitzwilliam Museum and the Cambridge Colleges, 2, Italy and the Iberian Peninsula. Morgan, N., Panayotova, S. & Reynolds, S. (eds.). London: Harvey Miller, Vol. 2 vols.
How Negation Scopes in Japanese and Korean
Sells, P., 2011, Japanese/Korean Linguistics . McClure, W. & den Dikken, M. (eds.). Stanford: CSLI Publications, Vol. 18. p. 328-344 17 p. (Japanese and Korean Linguistics; vol. 18).
On the appropriateness of sound
Pauletto, S., 2011, Explorations in Sonic Interaction Design. Rocchesso, D. (ed.). Logos Verlag Berlin, p. 59-60
Reconstruction and Scope of Negation in Korean (and Japanese)
Kim, S-S. & Sells, P., 2011, Proceedings of the 7th Workshop on Altaic Formal Linguistics. Simpson, A. (ed.). Cambridge, MA: MIT Working Papers in Linguistics, p. 139-153 15 p.
Sonic Interaction Design and its relation to film and theatre
Pauletto, S., Hug, D., Barrass, S. & Luckhurst, M., 2011, Explorations in Sonic Interaction Design. Rocchesso, D. (ed.). Logos Verlag Berlin, p. 11-14
Sound, information, and experience
Rocchesso, D., Serafin, S., Behrendt, F., Bernardini, N., Bresin, R., Eckel, G., Franinovic, K., Hermann, T., Pauletto, S., Susini, P. & Visell, Y., 2011, Explorations in Sonic Interaction Design. Rocchesso, D. (ed.). Logos Verlag Berlin, p. 3-6
Trends in Nature-based Tourism
Balmford, A., Beresford, J., Green, J., Naidoo, R., Walpole, M. & Manica, A., 2011, Ecotourism and Sustainable Tourism: New Perspectives and Studies. Apple Academic Press, p. 30-37 8 p.
Supporting Explicit Interpretation of Standards and Guidance
Stephenson, Z. R. & McDermid, J. A., Oct 2010, IET Conference Publications: 5th IET International Conference on System Safety 2010. 567 CP ed. IEEE, Vol. 2010. p. 1-6 6 p.
Analysing System Failure Behaviours with PRISM
Ge, X., Paige, R. F. & McDermid, J. A., Jun 2010, Proceedings of the 2010 Fourth International Conference on Secure Software Integration and Reliability Improvement Companion (SSIRI-C). p. 130-136 6 p.
Hyperbolic Equations with Random Boundary Conditions
Brzezniak, Z. & Peszat, S., Feb 2010, RECENT DEVELOPMENT IN STOCHASTIC DYNAMICS AND STOCHASTIC ANALYSIS . Duan, J., Luo, S. & Wang, C. (eds.). New Jersey: World Scientific Publishing, Vol. 8. p. 1-21 21 p. (Interdisciplinary Mathematical Sciences; vol. 8).
Deantune and Bishopstone: The estate and the church under the Mercian kings and the South Saxon bishops
Pickles, T. & Blair, J., 2010, The Later Anglo-Saxon Settlement at Bishopstone: A Downland Manor in the Making. Thomas, G. (ed.). York: Council for British Archaeology, p. 17-22 (Council for British Archaeology Research Report; vol. 163).
Development of High Integrity Software Product Lines using Model Transformation
Hutchesson, S. G. & McDermid, J. A., 2010, Computer Safety, Reliability and Security: SAFECOMP'10 Proceedings of the 29th International Conference on Computer Safety, Reliability, and Security. Heidelberg: Springer, Vol. 6351. p. 389-401 13 p.
Ethics and the practoce of qualitative research
Shaw, I. F., 2010, SAGE Qualitative Research Methods . Atkinson, P. & Delamont, S. (eds.). London: Sage Publications Ltd
Ethics in qualitative research and evaluation
Shaw, I. F., 2010, Data Collection : SAGE Benchmarks in Social Research Methods . Vogt, W. P. (ed.). London: Sage Publications Ltd, Vol. 3. (SAGE Benchmarks in Social Research Methods ).
FGF Signalling in Vertebrate Development
Isaacs, H. V. & Pownall, B., 2010, Colloquium Series on Developmental Biology. Morgan and Claypool, (Colloquium Series on Developmental Biology).
Cussens, J., 2010, Encyclopedia of Machine Learning. Sammut, C. & Webb, G. I. (eds.). SPRINGER, p. 519-522 4 p.
Introduction to pharmacoeconomics
Drummond, M., 2010, Pharmacoeconomics for European Hospital Pharmacists. Rutten, F. F. H. & Vulto, A. G. (eds.). Mol, Belgium: Pharma Publish and Media Europe, p. 8-11 4 p.
Language Documentation and Linguistic Theory
Sells, P., 2010, Language Documentation and Description 7. Austin, P. (ed.). SOAS, London: HRELP, Department of Linguistics, p. 209-237
Occurrence and Fate of Human Pharmaceuticals in the Environment
Monteiro, S. C. & Boxall, A. B. A., 2010, REVIEWS OF ENVIRONMENTAL CONTAMINATION AND TOXICOLOGY, VOL 202. NEW YORK: Springer, p. 53-154 102 p.
Prosodic constructions in making complaints
Ogden, R., 2010, Prosody in Interaction. Barth-Weingarten, D., Reber, E. & Selting, M. (eds.). Amsterdam: John Benjamins, p. 81-103 23 p. (Studies in Discourse and Grammar; vol. 23).
The Emergence of Carolingian Latin Literature and the Court of Charlemagne (780-814)
Garrison, M., 2010, Classical and Medieval Literature Criticism (vol. 127). Detroit: Gale, Cengage Learning, Vol. 127. p. 29-48 19 p. (Classical and Medieval Literature Criticism).
ANALYSIS OF CHEMOKINE RECEPTOR ENDOCYTOSIS AND INTRACELLULAR TRAFFICKING
Kershaw, T., Wavre-Shapton, S. T., Signoret, N. & Marsh, M., 2009, CHEMOKINES, PT A. SAN DIEGO: ELSEVIER ACADEMIC PRESS INC, p. 357-+ 22 p.
Advertising, Pragmatics of
Sells, P., 2009, The Pragmatics Encyclopedia. Cummings, L. (ed.). London: Routledge, p. 5-7 3 p.
Allometry of Body Size and Abundance in 166 Food Webs
Reuman, D. C., Mulder, C., Banasek-Richter, C., Blandenier, M-F. C., Breure, A. M., Den Hollander, H., Kneitel, J. M., Raffaelli, D., Woodward, G. & Cohen, J. E., 2009, ADVANCES IN ECOLOGICAL RESEARCH, VOL 41. SAN DIEGO: ELSEVIER ACADEMIC PRESS INC, p. 144 44 p.
Bioarchaeological analysis of iodine in dental enamel: initial analysis of sheep dental enamel for elemental iodine, for the purpose of future detection of in vivo iodine deficiency in ruminants and humans
Wright, C. C., Collins, M. J., Brothwell, D. R. & Shafer, D., 2009, Holm of Papa Westray: Artefacts and human and faunal remains. p. 83-87
Biological remains
Hall, A. R., Jaques, D. & Carrott, J., 2009, A Romano-British settlement at Millfield Farm, Wheldrake, near York. Robinson, G. (ed.). p. 167-8 2 p. (Yorkshire Archaeological Journal; no. 81).
Biscopes-tun, muneca-tun and preosta-tun: dating, significance and distribution
Pickles, T., 2009, The Church in English Place-Names. Quinton, E. (ed.). Nottingham: English Place-Name Society, p. 39-108 (English Place-Name Society Extra Series; vol. 4).
Community-based adaptation and culture in theory and practice
Ensor, J. & Berger, R., 2009, Adapting to Climate Change - Thresholds, Values, Governance. Adger, W. N., Lorenzoni, I. & O'Brien, K. (eds.). Cambridge University Press
Detecting deceit: current issues
Bull, P., 2009, International Developments in Investigative Interviewing. Williamson, T., Milne, B. & Savage, S. (eds.). Cullompton, Devon, UK: Willan, p. 190-206
Diaspora Engagement in Peacebuilding: Empirical and Theoretical Challenges
Bush, K. D., 2009, Whose Peace? Critical Perspectives on the Political Economy of Peacebuilding. Palgrave Macmillan, p. 191-205
Domain Analysis of an Electronic Health records System
Ge, X., Paige, R. F. & McDermid, J. A., 2009, FOSD '09 Proceedings of the First International Workshop on Feature-Oriented Software Development . New York: ACM, p. 49-54 5 p.
Edition of 'Haraldstikki'
Townend, M. O., 2009, Poetry from the Kings' Sagas 2. Gade, K. E. (ed.). Turnhout: Brepols, p. II, 807-08 2 p. (Skaldic Poetry of the Scandinavian Middle Ages; vol. 2).
Ligand Electron Density Shape Recognition Using 3D Zernike Descriptors
Gunasekaran, P., Grandison, S., Cowtan, K. D., Mak, L., Lawson, D. M. & Morris, R. J., 2009, Pattern Recognition in Bioinformatics: 4th IAPR International Conference, PRIB 2009, Sheffield, UK, September 7-9, 2009, Proceedings . Springer, Vol. 5780 LNBI. p. 125-136 11 p. (Lecture Notes in Computer Science ).
Social health insurance and labor market outcomes: Evidence from central and eastern Europe, and central Asia
Wagstaff, A. & Moreno-Serra, R., 2009, Innovations in Health System Finance in Developing and Transitional Economies. Emerald Group Publishing Limited, p. 83-106 24 p.
Unpredicated Particles
Levin, B. & Sells, P., 2009, Reality Exploration and Discovery: Pattern Interaction in Language and Life. Uyechi, L. & Wee, L. H. (eds.). Stanford: CSLI Publications, p. 303-324 22 p.
Use of pharmacoeconomics in drug reimbursement in Australia, Canada and the United Kingdom: What can we learn from international experience?
Drummond, M. F. & Sorenson, C., 2009, Pharmacoeconomics From Theory to Practice . Arnold, R. J. G. (ed.). Boca Raton, USA: CRC Press, p. 175 -196
Delivery of nitric oxide for analysis of the function of cytochrome c'
Cole, L. J., Huston, W. M. & Moir, J. W. B., 2008, GLOBINS AND OTHER NITRIC OXIDE-REACTIVE PROTEINS, PT A. SAN DIEGO: ELSEVIER ACADEMIC PRESS INC, p. 21-33 13 p.
Drummond, M. F., Goeree, R., Modyyedi, P. & Levin, M., 2008, Users Guides to the Medical Literature. Guyatt, G., Rennie, D., Meade, O. M. & Cook, D. J. (eds.). second edition ed. Chicago: Chicago Illinois: American Medical Association
Evolution of Cooperation in an Incentive Based Business Game Environment
Bista, S. K., Dahal, K. P. & Cowling, P. I., 2008, Global Design to Gain a Competitive Edge. Springer London, p. 875-882 8 p.
Hyperheuristics: recent developments
Chakhlevitch, K. & Cowling, P., 2008, Adaptive and multilevel metaheuristics. Springer Berlin/Heidelberg, p. 3-29 27 p.
Incorporating economics evidence
Shemilt, I., Mugford, M., Byford, S., Drummond, M. F., Eisenstein, E., Knapp, M., Mallender, J., McDaid, D., Vale, L. & Walker, D., 2008, Cochrane Handbook for Systematic Reviews of Interventions, version 5.0.0. Higgins, J. P. T. & Green, S. (eds.). The Cochrane Collaboration. Available from www.cochrane-handbook.org
Korean Copular Constructions: A Lexical Sharing Approach
Kim, J-B., Sells, P. & Wescoat, M. T., 2008, Japanese/Korean Linguistics 13. Clancy, P., Hudson, M. E., Jun, S-A. & Sells, P. (eds.). Stanford: CSLI Publications, p. 179-192 14 p.
Local modelling in classification.
Szepannek, G., Schiffner, J., Wilson, J. C., Weihs, C. & Perner, P. (ed.), 2008, Advances in Data Mining: Medical Applications, E-Commerce, Marketing, and Theoretical Aspects: Proceedings of 8th Industrial Conference, ICDM 2008. 2008 ed. SPRINGER-VERLAG BERLIN, Vol. 5077. p. 153-164 12 p. (Lecture Notes in Computer Science; vol. 5077).
Long-Distance Dependencies in Tagalog: The Case for Raising
Gerassimova, V. & Sells, P., 2008, Proceedings of the 26th West Coast Conference on Formal Linguistics. Chang, C. B. & Haynie, H. J. (eds.). Somerville, Mass.: Cascadilla Proceedings Project, p. 190-198 9 p.
Measuring nitric oxide metabolism in the pathogen Neisseria meningitidis
Thomson, M. J., Stevanin, T. M. & Moir, J. W. B., 2008, GLOBINS AND OTHER NITRIC OXIDE-REACTIVE PROTEINS, PART B. SAN DIEGO: ELSEVIER ACADEMIC PRESS INC, p. 539-560 22 p.
The carbonised grain
Hall, A. R., 2008, Before Eoforwic: new light on York in the 6th-7th centuries. Spall, C. A. & Toop, N. J. (eds.). p. 11 (Medieval Archaeology ; vol. 52).
What are the factors that promote high post-16 participation of many minority ethnic groups? A focused review of the UK-based aspirations literature
Torgerson, C. J., Gorard, S., Low, G. D., Ainsworth, H., See, B. H. & Wright, K., 2008, Research Evidence in Education Library Report no 1608. London: EPPI-Centre, Social Sciences Research Unit, Institue of Education, University of London
Tilting modules for algebraic groups and finite dimensional algebras
Donkin, S., Happel, D. (ed.), Huegel, L. (ed.) & Krause, H., 1 Mar 2007, Handbook of Tilting Theory. p. 169-208 (London Mathematical Society Lecture Note Series ; no. 332).
Changes in classroom practice: executive summary
Ratcliffe, M., Hanley, P. & Osborne, J., Feb 2007, Twenty first century science pilot evaluation report. UYSEG and Nuffield Foundation, p. 12-15 4 p.
Abstraction and reformulation in the generation of constraint models - (Extended abstract)
Frisch, A. M., 2007, Abstraction, Reformulation, and Approximation, Proceedings. Miguel, I. & Ruml, W. (eds.). BERLIN: SPRINGER-VERLAG BERLIN, p. 23 2 p.
Appendix 3. Assessment of the samples rich in moss
Hall, A. R., 2007, Harnessing the Tides. : The Early Medieval Tide Mills at Nendrum Monastery. McErlean, T. & Crothers, N. (eds.). Strangford Lough, (N. Ireland Archaeol. Monograph; no. 7).
Coulomb energy differences between high-spin states in isobaric multiplets
Bentley, M. A. & Lenzi, S. M., 2007, PROGRESS IN PARTICLE AND NUCLEAR PHYSICS, VOL 59, NO 2. AMSTERDAM: ELSEVIER SCIENCE BV, p. 497-561 65 p.
E-government and the United Kingdom
Pleace, N., 2007, E-Government in Europe : Re-booting the state. Nixon, P. G. & Koutrakou, V. N. (eds.). Abingdon: Routledge, p. 61-74 14 p.
Logic-based Formalisms for Statistical Relational Learning
Cussens, J., 2007, Introduction to Statistical Relational Learning. Getoor, L. & Taskar, B. (eds.). Cambridge, MA: MIT Press
Plant macrofossils
Hall, A. R., 2007, The transformation of an early post-medieval town into a major modern city: excavation and survey of the Waverley Vaults, New Street, Edinburgh, Scotland. Toolis, R. & Sproat, D. (eds.). p. 170-71 (Post-Medieval Archaeology ; vol. 41).
Report on the Bioethical Issues raised by the research
Holland, S. M., Baldwin, T. R. & Farrant, A., 2007, Osteocord Bone from blood: Optimised isolation, characterisation and osteogenic induction of mesenchymal stem cells from umbilical cord blood..
Using a large set of low level heuristics in a hyperheuristic approach to personnel scheduling
Cowling, P. I. & Chakhlevitch, K., 2007, Evolutionary Scheduling. Springer Berlin/Heidelberg, p. 543-576 34 p.
Using economic evaluation in reimbursement decisions for health technologies: lessons from international experience
Drummond, M. F., 2007, Pharmaceutical Innovation: incentives competition and cost-benefit analysis in international perspective. Sloan, F. A. & Chee-Ruey, H. (eds.). New York: Cambridge University Press, p. 215-225 11 p.
Divine Election for Nations: A Difficult Rhetoric for Medieval Scholars?
Garrison, M., 10 Oct 2006, The Making of Christian Myths in the Periphery of Latin Christendom (c.1000-1300). Mortensen, L. B. (ed.). Copenhagen: Museum Tusculanum Press, p. 275-314 39 p.
Automated Classification of Images from Crystallisation Experiments
Wilson, J. C. & Perner, P. (ed.), 1 Feb 2006, Advances in Data Mining: Applications in Medicine, Web Mining, Marketing, Image and Signal Mining: Proceedings of the 6th Industrial Conference on Data Mining In: ICDM 2006 6th Industrial Conference on Data Mining, 14-15 July 2006, Leipzig, Germany.. 2006 ed. Springer Berlin / Heidelberg, Vol. 4065. p. 459-473 15 p. (Lecture Notes in Computer Science; vol. 4065).
Adsorption of ionisable pesticides in soils
Kah, M. & Brown, C. D., 2006, REVIEWS OF ENVIRONMENTAL CONTAMINATION AND TOXICOLOGY, VOL 188. NEW YORK: SPRINGER, p. 149-217 69 p.
Centralizer construction of the Yangian of the queer Lie superalgebra
Nazarov, M. & Sergeev, A., 2006, Studies in Lie Theory: Dedicated to A. Joseph on his Sixtieth Birthday: Progress in Mathematics. Part II ed. Vol. 243. p. 417-441 25 p.
Economic evaluation and decision makers
Drummond, M. F., 2006, The Elgar Companion to Health Economics. Jones, A. M. (ed.). Cheltenham: Edward Elgar
Evaluating the Shelter Inclusion Project: a floating support service for households accused of antisocial behaviour
Jones, A., Pleace, N. & Quilgars, D., 2006, Housing, Urban Governance and Anti-Social Behaviour. Flint, J. (ed.). Bristol: Policy Press
Factorization of symmetric polynomials
Kuznetsov, V., Sklyanin, E., Kuznetsov, V. (ed.) & Sahi, S. (ed.), 2006, Jack, Hall-Littlewoood and Macdonald Polynomials. AMER MATHEMATICAL SOC, Vol. 417. p. 239-256 18 p. (Contemporary Mathematics).
History and citizenship: linking past, present and future
Davies, I., 2006, The history teacher's yearbook 2006/7.
Quantum energy inequalities and stability conditions in quantum field theory
Fewster, C. J., 2006, Rigorous Quantum Field Theory: A Festschrift for Jacques Bros. Buchholz, D., Boutet de Monvel, A., Iagolnitzer, D. & Moschella, U. (eds.). Birkhauser, Vol. 251. p. 95-112 (Progress in Mathematics; vol. 251).
The Peak Load Problem with Feasible Storage
Gravelle, H. S. E., 2006, The Economics of Public Utilities. p. 415-36 (International Library of Critical Writings in Economics; vol. 1).
The impact of rule ranking on the quality of associative classifiers
Thabtah, F., Cowling, P. & Peng, Y., 2006, Research and Development in Intelligent Systems XXII. Springer London, p. 277-287 11 p.
Low perturbation antenna impedance measurement techniques
Jekkonen, J., Flintoft, I. D., Capstick, M. & Marvin, A., 23 Oct 2005, URSI2005, The XXVIIIth Triennial General Assembly of the International Union of Radio Science: 23-29 October 2005. New Delhi: International Union of Radio Science, 4 p.
A novel indirect method to determine the radiation impedance of an unknown antenna structure
Jekkonen, J., Flintoft, I. D., Capstick, M. H. & Marvin, A. C., 14 Feb 2005, Proceedings of the 16th International Zurich Symposium and Technical Exhibition on Electromagnetic Compatibility: 14-18 February, 2005. Zurich: Laboratory for Electromagnetic Fields and Microwave Electronics, Swiss Federal Institute of Technology Zurich, p. 179-182 4 p.
Bilevel optimisation of prices in a variety of transportation models
Smith, M. & Mahmassami, H. (ed.), 2005, Transportation and Traffic Theory Flow, Dynamics and Human Interaction: Proceedings of the 16th International Symposium on Transportation and Traffic Theory. Vol. 1. p. 1-21 21 p.
Deductive Reasoning and Statistical Inference
Cussens, J., 2005, Encyclopedia of Statistics in Behavioral Science. Chichester: John Wiley & Sons
Gready, P. & Ensor, J., 2005, Reinventing development? Translating rights-based approaches from theory into practice. Gready, P. & Ensor, J. (eds.). London: Zed Books
Linking rights and culture-implictions for rights-based approaches
Ensor, J., 2005, Reinventing development?: translating rights-based approaches from theory into practice. Gready, P. & Ensor, J. (eds.). London: Zed Books
Results from a survey among United Kingdom decision-makers
Drummond, M. F., Nixon, J. & Wilson, P., 2005, EUROMET 2004: the Influence of Economic Evaluation Studies on Health Care Decision-Making. A European Study. Eberhardt, S., Stoklossa, C. & Graf von der Schulenburg, J. M. (eds.). Amsterdam: IOS Press
New Developments in Shielding Effectiveness Measurements: Shielding Aperture
Marvin, A. C., Dawson, J. F. & Dawson, L., 1 Apr 2004, IEE New EMC Issues in Design: Techniques Tools and Components Seminar. p. 45-50 6 p.
Encountering the Other in the Classroom: An Interview
Attridge, D., 2004, Thinking Difference: Conversations with Critics. Wolfreys, J. (ed.). New York: Fordham University Press, p. 11-19 9 p. (Perspectives in Continental Philosophy).
The role of practical work in the teaching and learning of science
Committee on High School Science Laboratories: Role and Vision, 2004, America's Lab Report. Washington DC: National Academy of Sciences
Veterinary medicines in the environment
Boxall, A. B. A., Fogg, L. A., Blackwell, P. A., Kay, P., Pemberton, E. J. & Croxford, A., 2004, REVIEWS OF ENVIRONMENTAL CONTAMINATION AND TOXICOLOGY. NEW YORK: Springer-verlag, Vol. 180. p. 191 91 p.
A systematic review of the effects of context-based and Science-Technology-Society (STS) approaches in the teaching of secondary science
Bennett, J., Hogarth, S. & Lubben, F., 2003, Research Evidence in Education Library. London: EPPI-Centre, Social Research Unit, Institute of Education, p. 128
Development and diffusion of health technology
Drummond, M. F. & Lindgren, B., 2003, Learning to Live with Health Economics. Zöllner, H., Stoddart, G. & Selby Smith, C. (eds.). Copenhagen: Health Services Management Centre in Collaboration with WHO (Regional Office for Europe)
Drummond, M. F., 2003, Learning to Live with Health Economics. Zöllner, H., Stoddart, G. & Selby Smith, C. (eds.). Copenhagen: Health Services Management Centre in Collaboration with WHO (Regional Office for Europe)
Induced representation of affine Hecke algebras and canonical bases of quantum groups
Leclerc, B., Nazarov, M. & Thibon, J-Y., 2003, Studies in Memory of Issai Schur. Joseph, A., Melnikov, A. & Rentschler, R. (eds.). Springer-verlag, Vol. 210. p. 115-153 39 p. (Progress in Mathematics; vol. 210).
Macromolecular TLS refinement in REFMAC at moderate resolutions
Winn, M. D., Murshudov, G. N. & Papiz, M. Z., 2003, MACROMOLECULAR CRYSTALLOGRAPHY, PT D. SAN DIEGO: ELSEVIER ACADEMIC PRESS INC, p. 300-321 22 p.
Representations of Yangians associated with skew Young diagrams
Nazarov, M., 2002, Proceedings of the International Congress of Mathematicians: Beijing 2002, August 20-28. Tatsien, L. I. (ed.). Higher Education Pr (July 2003), Vol. 2. p. 643-654 12 p.
Software Hazard and Safety Analysis
McDermid, J. A., Damm, W. (ed.) & Olderog, E-R. (ed.), 2002, Formal Techniques in Real-Time and Fault-Tolerant Systems. Damn, W. & Olderog, E-R. (eds.). Springer-verlag, p. 23-36 14 p. (Lecture Notes in Computer Science; vol. 2469).
The use of wavelet transforms in low resolution phase extension
Main, P. & Wilson, J. C., 1 Jun 2001, Methods in Macromolecular Crystallography : Proceedings of the NATO Advanced Study Institutes on Methods in Macromolecular Crystallography and Chemical Prospsectives in Crystallography of Molecular Biology. Turk, D. & Johnson, L. (eds.). Netherlands: IOS Press, Vol. 325. p. 82-94 (Nato Science Series I : Life and Behavioural Sciences, ; vol. 325).
Counselling for depression in primary care
Rowland, N., Bower, P., Mellor Clark, J., Heywood, P. & Godfrey, C., 2001, The Cochrane Library . Oxford: Update Software
Ergodicity of p-adic multiplications and the distribution of Fibonacci numbers
Coelho, Z. & Parry, W., 2001, Topology, Ergodic Theory, Real Algebraic Geometry: Rokhlin's Memorial. Vol. 202. p. 51-70 20 p. (Translations-Series 2, Advances in the Mathematical Sciences; vol. 202).
Integrating Probabilistic and Logical Reasoning
Cussens, J., 2001, Foundations of Bayesianism. Corfield, D. & Williamson, J. (eds.). Dordrecht: Kluwer, Vol. 24. p. 241-260 20 p. (Applied Logic Series).
Bäcklund Transformations and Baxter's Q-operator
Sklyanin, E., Harnad, J. P. (ed.), Sabidussi, G. (ed.) & Winternitz, P. (ed.), 2000, Integrable Systems: From Classical to Quantum. Vol. 26. p. 227-250 (CRM Proceedings & Lecture Notes; vol. 26).
Canonicity of Bäcklund transformation: r-matrix approach 1
Sklyanin, E. & Semenov-Tian-Shansky, M. (ed.), 2000, L. D. Faddeev's Seminar on Mathematical Physics. Vol. 201. p. 277-282 6 p. (Translations Series 2: Advances in the Mathematical Sciences; vol. 201).
Experiments in Inductive Chart Parsing
Cussens, J. & Pulman, S., 2000, Learning Language in Logic. Cussens, J. & Dzeroski, S. (eds.). SPRINGER, Vol. 1925. (LNAI).
Managing budgets
Gravelle, H., Smith, P., Dusheiko, M., Jacobs, R. & Malbon, G., 2000, The National Tracker Survey of Primary Care Groups and Trusts: Progress and Challenges 1999/2000. Wilkin, D., Gillam, S. & Leese, B. (eds.). Manchester: The University of Manchester
The supply-side effects of tobacco control policies
Jacobs, R., Gale, F., Capehart, T., Zhang, P. & Jha, P., 2000, Tobacco control policies in developing countries. Jha, P. & Chaloupka, F. (eds.). New York: Oxford University Press
The taxation of tobacco products
Chaluoupka, F., Hu, T., Warner, K., Jacobs, R. & Yurekli, A., 2000, Tobacco control policies in developing countries. Jha, P. & Chaloupka, F. (eds.). New York: Oxford University Press
Time Series Analysis of Mortality and Unemployment
Gravelle, H. S. E., 2000, The economics of unemployment. Vol. 4. p. 401-09 (International Library of Critical Writings in Economics; vol. 122).
Ex Post Value Regulation of Pharmaceutical Prices
Gravelle, H., 1999, Regulation strategies and economic policies. Vol. 3. p. 125-43 (Essays in honour of Bernard Corry and Maurice Peston ; vol. 3).
Employment issues in tobacco control
van der Merwe, R., 1998, The economics of tobacco control: towards an optimal policy mix. Abedian, I., van der Merwe, R., Wilkins, N. & Jha, P. (eds.). Cape Town: Applied Fiscal Research Centre: University of Cape Town
Principles of pharmacoeconomics
Drummond, M. F., 1998, Health Economics of Dementia. Winblad, B., Wimo, A., Jönsson, B. & Karlsson, G. (eds.). London: John Wiley
The economics of tobacco control in South Africa
"Court, City and Countryside: Jan Brueghel's Peasant Weddings As Images of Social Unity Under Archducal Sovereignty"
Van Wyhe, C. H., 1998, .] Albert and Isabella, Brussels, Koninklijke Musea voor Kunst en Geschiedenis, Madrid, Prado Museum, Brepols . BREPOLS PUBLISHERS, p. 151-160
Automated refinement for protein crystallography
Lamzin, V. S. & Wilson, K. S., 1997, Methods in Enzymology. SAN DIEGO: ELSEVIER ACADEMIC PRESS INC, Vol. 277. p. 269-305 37 p.
Biotechnology in the changing health care environment: methods for economic evaluation of innovative technology
Drummond, M. F. & Mason, J. M., 1997, OECD 1997 Economic Aspects of Biotechnologies Related to Human Health: Part 1: Biotechnology and medical innovation. OECD. (ed.). Paris: OECD, Vol. OECD/GD(97)205.
Disease management: who needs it and why?
Drummond, M. F., 1997, Disease Management. Lilley, R. (ed.). Chichester: Wiley
Economic evaluation and clinical audit
Drummond, M. F. & Mason, J. M., 1997, Evaluation and Development in Clinical Practice. Miles, A. (ed.). London: The Royal Society of Medicine Press
The Australian cost-effectiveness guidelines: an update
Drummond, M. F. & Aristides, M., 1997, Guidelines for the Economic Evaluation of Pharmaceuticals: can the UK learn from Australia and Canada?. Towse, A. (ed.). London: Office of Health Economics;
The radiation-attenuated vaccine against schistosomes in animal models: Paradigm for a human vaccine?
Coulson, P. S., 1997, ADVANCES IN PARASITOLOGY, VOL 39. SAN DIEGO: ELSEVIER ACADEMIC PRESS INC, p. 271-336 66 p.
The wound response of tomato plants: analysis of local and long-range signalling events
Bowles, D. J., 1997, ESSAYS IN BIOCHEMISTRY, VOL 32, 1997. LONDON: PORTLAND PRESS LTD, p. 161-169 9 p.
Using economic data for prescribing decisions
Drummond, M. F., 1997, Promoting Cost-Effective Prescribing in the UK National Health Service. Bloor, K. & Freemantle, N. (eds.). York: Centre for Health Economics, University of York
Socioeconomic evaluation of pharmaceuticals
Drummond, M. F. & Luce, B. R., 1996, Assessment Of Health Care Technologies: case studies, key concepts and strategic issues. Szczepura, A. & Kankaanpää, J. (eds.). London: John Wiley & Sons;
The future of pharmacoeconomics
Drummond, M. F., 1996, Quality of Life and Pharmacoeconomics in Clinical Trials. Spilker, B. (ed.). 2nd ed. New York: RAVEN PRESS
Adapting economic evaluation results from one setting to another
Drummond, M. F., 1995, Economic Evaluation In Australian Health Care. Selby Smith, C. & Drummond, M. F. (eds.). Canberra: Australian Government Printing Office
Economic evaluation and government guidelines: the art of the possible
Issues in the socioeconomic evaluation of cancer: an introduction
Drummond, M. F., 1995, The Challenges in Studying the Socio-Economic Dimensions in Cancer Therapy. Champey, Y. & Hillman, A. H. (eds.). Paris: Rhône Poulenc Rorer Foundation
Limitations of current economic evaluation methodologies in the cancer field
Drummond, M. F. & Coyle, D., 1995, The Challenges in Studying the Socio-Economic Dimensions in Cancer Therapy. Champey, Y. & Hillman, A. H. (eds.). Paris: Rhône Poulenc Rorer Foundation
Recent developments in overseas economic evaluation
Drummond, M. F. & Rutten, F. F. H., 1995, Economic Evaluation In Australian Health Care. Selby Smith, C. & Drummond, M. F. (eds.). Canberra: Australian Government Printing Office
Consumer theory: duality
Gravelle, H. & Rees, R., 1994, Microeconomics: solutions manual and workbook. Gravelle, H. & Rees, R. (eds.). London: Longman-Pearson
Economic evaluation of drug therapy
Drummond, M. F. & Davies, L. M., 1994, The Economics Of Schizophrenia. Moscarelli, M. (ed.). London: Wiley
Economic evaluation of treatments for shingles
Drummond, M. F., 1994, Economics Of Shingles. Backhouse, M. (ed.). London: Synergy Publications
Health economics and anaesthesia
Drummond, M. F., 1994, Anaesthesia Rounds. Oxford: The Medicines Group
Safety Engineering and Assurance for Real-Time Systems
McDermid, J. A., Halong, W. A. (ed.) & Stoyenko, A. D. (ed.), 1994, Real-Time Computing. Springer-Verlag
The theory of the consumer
Birds of a feather Group on Safety Critical Systems
Cullyer, W. J., McDermid, J. A., Halang, W. A. (ed.) & Stoyenko, A. D. (ed.), 1993, Real-Time Computing. Springer-verlag, p. 755-756 2 p.
Cost-benefit analysis in health and health care: fine in practice, but does it work in theory?
Drummond, M. F., 1993, Efficiency In The Public Sector. Williams, A. & Giardina, E. (eds.). Aldershot: Edward Elgar Publishing
Cost-effectiveness league tables and priority setting
Mason, J. & Drummond, M. F., 1993, Purchasing And Providing Cost-Effective Health Care. Drummond, M. F. & Maynard, A. (eds.). London: Churchill Livingstone
Drug costs and health priorities
Drummond, M. F., 1993, Drug Costs And Treatment Choices. Collier, J. & Bonn, D. (eds.). London: Consumers' Association
Estimating utilities for making decisions in health care
Drummond, M. F., 1993, Analysing How We Reach Clinical Decisions. Llewelyn, H. & Hopkins, A. (eds.). London: Royal College of Physicians of London;
General practice fundholding: maverick or catalyst?
Leese, B. & Drummond, M. F., 1993, Purchasing And Providing Cost-Effective Health Care. Drummond, M. F. & Maynard, A. (eds.). London: Churchill Livingstone
Health technology policy and health services research
Drummond, M. F., 1993, Purchasing And Providing Cost-Effective Health Care. Drummond, M. F. & Maynard, A. (eds.). London: Churchill Livingstone
Methodology of economic appraisal of health technology
Drummond, M. F., 1993, Health Services Research. Fracchia, G. N. & Theofilatou, M. (eds.). Amsterdam: 10S Press for the European Community
SAM - A Tool to Support the Construction, Review and Evolution of Safety Arguments
Forder, J., Higgins, C., McDermid, J. A., Storrs, G., Redmill, F. (ed.) & Anderson, T. (ed.), 1993, Directions in Safety-Critical Systems: Proceedings of the First Safety-Critical Systems Symposium . Springer-Verlag, p. 195-216 22 p.
The contribution of health economics to cost-effective health care delivery
Medical technology in general practice in the UK: where do we go from here?
Leese, B. & Drummond, M. F., 1992, Primary Health Care: A Global View. Adamopoulos, P. N. (ed.).
The true cost of migraine
Drummond, M. F., 1992, Migraine through expert eyes. Worthing: Professional Postgraduate Services Europe
Cost-benefit in prescribing
Drummond, M. F., 1991, Prescribing In The 90s. Medicine, R. S. O. (ed.). London: Royal Society of Medicine;
Economic evaluation and the rational diffusion of health technology in Europe
Drummond, M. F., 1991, Health Care Systems And The Challenges Of Europe After 1992. Kyriopoulos, J. E. (ed.). Athens: Lambrakis Press
Drummond, M. F., 1991, Oxford Textbook of Public Health. Holland, W. W. (ed.). 2nd ed. Oxford: Oxford Medical Publications
Drummond, M. F., 1991, Textbook On Pharmaceutical Medicine. Auty, R., Mann, R. & Rawlins, M. (eds.). Carnforth: PARTHENON PUBLISHING GROUP LTD
Health outputs and priorities for health care
Drummond, M. F., 1991, A Share In Health: An Exploration Of The Role Of The DHA. Bath: Bath District Health Authority
A strategy for measurement of quality of life in National Eye Institute trials
Drummond, M. F. & Ferris, F. F., 1990, Measuring The Quality Of Life Of People With Visual Impairment. Drummond, M. F. (ed.). Washington U.S.: U.S. Department of Health and Human Services
Assessing efficiency in the new NHS
Drummond, M. F., 1990, Changing Face Of The NHS In The 1990s. Spurgeon, P. (ed.). London: Longman and Co
Economic evaluation of laser applications
Drummond, M. F., 1990, Effectiveness and Cost-Effectiveness of Laser Applications in Health Care. Banta, H. D. & Schou, I. (eds.). Frederiksberg: Lambert Academic Publishing
Economic evaluation of programmes for AIDS and HIV infection: methodological issues
Drummond, M. F. & Davies, L. M., 1990, Economic Aspects of AIDS and HIV Infection. Schwefel, D. (ed.). Berlin: Springer-verlag
Economic implications of laser use in ophthalmology
Drummond, M. F., 1990, Effectiveness And Cost-Effectiveness Of Laser Applications In Health Care. Banta, H. D. & Shou, I. (eds.). Frederiksberg: Lambert Academic Publishing
Issues and prospects for economic analysis
Drummond, M. F. & Davies, L., 1990, AIDS: The Challenge For Economic Analysis. Birmingham: Health Services Management Centre in Collaboration with WHO (Regional Office for Europe)
Measuring and valuing health states relating to visual impairment: a review of literature, concepts and methods
Ferguson, B. A., Buxton, M. J. & Drummond, M. F., 1990, Measuring the Quality of Life of People with Visual Impairment. Drummond, M. (ed.). Washington: U.S. Department of Health and Human Services
Methodological and practical issues in estimating the direct costs of AIDS/HIV: overview
Davies, L. M. & Drummond, M. F., 1990, AIDS: The Challenge For Economic Analysis. Birmingham: Health Services Management Centre in Collaboration with WHO (Regional Office for Europe)
Priority setting for AIDS and other health care programmes
Drummond, M. F., 1990, AIDS: The Challenge For Economic Analysis. Birmingham: Health Services Management Centre in Collaboration with WHO (Regional Office for Europe)
The cost-effectiveness of lipid-lowering drugs
Drummond, M. F. & McGuire, A., 1990, The Social and Economic Contexts of Coronary Heart Disease Prevention. Lewis, B. & Assmann, G. (eds.). London: Current Medical Literature
The utility approach to measuring health states
Drummond, M. F., 1990, Measuring The Quality Of Life Of People With Visual Impairment. Drummond, M. F. (ed.). Washington, U.S.: U.S. Department of Health and Human Services
Topics for economic analysis
Drummond, M. F. & Davies, L. M., 1990, AIDS: The Challenge For Economic Analysis. Birmingham: Health Services Management Centre in Collaboration with WHO (Regional Office for Europe)
Assurance in High-Integrity Software
McDermid, J. A. & Sennett, C. T. (ed.), 1989, High-integrity software. Pitman, p. 226-273 48 p.
Financial incentives to change behaviour towards health technology
Drummond, M. F., 1988, Expensive health technologies: regulatory and administrative mechanisms in Europe. Stocking, B. (ed.). Oxford: Oxford University Press
Security Models and Enterprise Models
Dobson, J. E., McDermid, J. A. & Landwehr, C. (ed.), 1988, Database Security II : Status and Prospects. II ed. IFIP - International Federation for Information Processing, p. 1-36 36 p.
The assessment of drug therapy in social and economic terms
Drummond, M. F., 1988, Trends and Changes in Drug Research and Development. Walker, B. & Walker, S. R. (eds.). Lancaster: MTP Press
The role of economics in the evaluation of care
Mugford, M. & Drummond, M. F., 1988, Effective Care in Pregnancy and Childbirth. Enkin, M. W., Keirse, M. J. N. C. & Chalmers, I. (eds.). Oxford: Oxford University Press
A world campaign to eliminate cataracts: is it worth it?
Drummond, M., 1987, To Restore Sight: The Global Conquest of Cataract Blindness. New York: Helen Keller International Inc
Assessing the costs and benefits of medicines: some current measurement issues
Drummond, M. F., 1987, Costs and Benefits of Pharmaceutical Research. Teeling-Smith, G. (ed.). London: Office of Health Economics;
Cost benefit analysis in health care: future directions
Drummond, M. F., 1987, Health Economics: Prospects for the Future. Teeling Smith, G. (ed.). London: Croom Helm
Economic Appraisal and Health Technology: background to the EC project
Drummond, M. F., 1987, Economic Appraisal of Health Technology in the European Community. Drummond, M. F. (ed.). Oxford: Oxford Medical Publications
Economic Appraisal of Health Technology: future directions
Methods for economic appraisal of health technology
Clinical and economic evaluation of diagnostic technologies: the cases of impedance plethysmography and nuclear magnetic resonance
Guyatt, G. & Drummond, M. F., 1986, Health Care Technology: Effectiveness, Efficiency and Public Policy. Feeny, D., Guyatt, G. & Tugwell, P. (eds.). Halifax: Canadian Institute for Research on Public Policy
Default Risk and the Optimal Pricing of Court Enforcement Services
Gravelle, H. S. E., 1986, Law and economics and the economics of legal regulation. Kluwer Academic Publishers, Vol. 13. p. 85-112 28 p. (International Studies in Economics and Econometrics series; vol. 13).
Guidelines for health technology assessment: diagnostic technologies
Guyatt, G., Drummond, M. F., Feeny, D., Haynes, R. B. & Tugwell, P., 1986, Health Care Technology: Effectiveness, Efficiency and Public Policy. Feeny, D., Guyatt, G. & Tugwell, P. (eds.). Halifax: Canadian Institute for Research on Public Policy
Guidelines for health technology assessment: therapeutic technologies
Guidelines for health technology assessment: economic evaluation
Drummond, M. F., 1986, Health Care Technology: Effectiveness, Efficiency and Public Policy. Feeny, D., Guyatt, G. & Tugwell, P. (eds.). Halifax: Canadian Institute for Research on Public Policy
Mobilizing resources and political will: convincing governments that prevention of blindness is cost effective
Drummond, M. F., 1986, World blindness and its prevention. Oxford: Oxford University Press, Vol. 3.
Review of evaluative studies of neonatal care for very low birth weight infants: social and economic aspects
Drummond, M. F., 1986, Perinatal care delivery systems: description and evaluation in EC Countries. Kaminski, M., Breart, G., Buekens, P., Huisjes, H. J., McIlwaine, G. & Selbmann, H. K. (eds.). Oxford: Oxford University Press
The financial burden of stroke and the economic evaluation of treatment alternatives
Drummond, M. F. & Ward, G. H., 1986, Stroke: Epidemiological, Therapeutic and Socio economic Aspects. Clifford Rose, F. (ed.). London: Royal Society of Medicine;
Social science techniques: economic studies
Drummond, M. F., 1985, Oxford Textbook of Public Health. Holland, W. W., Detels, R. & Knox, G. (eds.). Oxford: Oxford University Press, Vol. 3.
Bargaining and Efficiency in Public and Private Sector Firms
Gravelle, H. S. E., 1984, The Performance of Public Enterprises: Concepts and Measurement. Marchand, M., Pestieau, P. & Tulkens, H. (eds.). Amsterdam; New York and Oxford: North Holland, p. 193-220 28 p. (Studies in Mathematical and Managerial Economics).
Economic assessment of therapy
Drummond, M. F., 1983, Measuring the social benefits of medicines. Teeling Smith, G. (ed.). London: Office of Health Economics;
Will you achieve more than your predecessor? Evaluating effectiveness and efficiency in the hospital service
Drummond, M. F. & Yates, J. M., 1982, The management of medicine. Allen, D. & Grimes, D. (eds.). London: Pitman Medical
Evaluation and the National Health Service
Drummond, M. F., 1978, Economic Aspects of Health Services. Culyer, A. J. & Wright, K. G. (eds.). London: Martin Robertson
Financing medical education: inter relationships between medical school and teaching hospital expenditure
Culyer, A. J. & Drummond, M. F., 1978, Economic Aspects of Health Services. Culyer, A. J. & Wright, K. G. (eds.). London: Martin Robertson
Sharing resources for health in England: the case of teaching hospitals


text:
\section{Introduction}

Neural machine translation (NMT) models~\citep{sutskever,bahdanau,convs2s,transformer} are remarkably effective in modelling the distribution over target sentences conditioned on the source sentence, and yield superior translation performance compared to traditional statistical machine translation (SMT) on many language pairs. However, it is often difficult to extract a comprehensible explanation for the predictions of these models as information in the network is represented by real-valued vectors or matrices~\citep{visual-nmt}. In contrast, the translation process in SMT is `transparent' as it can identify the source word which caused a target word through word alignment. Most NMT models do not use the concept of word alignment. It is tempting to interpret encoder-decoder attention matrices~\citep{bahdanau} in neural models as (soft) alignments, but previous work has found that the attention weights in NMT are often erratic~\citep{nmt-attention-sucks} and differ significantly from traditional word alignments~\citep{nmt-challenges,nmt-where-attention}. We will discuss the difference between attention and alignment in detail in Sec.~\ref{sec:att}. The goal of this paper is explainable NMT by developing a transparent translation process for neural models. Our approach does not change the neural architecture, but represents the translation together with its alignment as a linear sequence of operations. The neural model predicts this operation sequence, and thus simultaneously generates a translation and an explanation for it in terms of alignments from the target words to the source words that generate them. The operation sequence is ``self-explanatory''; it does not explain an underlying NMT system but is rather a single representation produced by the NMT system that can be used to generate translations along with an accompanying explanatory alignment to the source sentence. We report competitive results of our method on Spanish-English, Portuguese-English, and Japanese-English, with the benefit of producing hard alignments for better interpretability. We discuss the theoretical connection between our approach and hierarchical SMT~\citep{hiero} by showing that an operation sequence can be seen as a derivation in a formal grammar.

\section{A Neural Operation Sequence Model}
\label{sec:osnmt}

Our operation sequence neural machine translation (OSNMT) model is inspired by the operation sequence model for SMT~\citep{osm}, but changes the set of operations to be more appropriate for neural sequence models. OSNMT is not restricted to a particular architecture, i.e.\ any seq2seq model such as RNN-based, convolutional, or self-attention-based models~\citep{bahdanau,transformer,convs2s} could be used.
In this paper, we use the recent Transformer model architecture~\citep{transformer} in all experiments.

In OSNMT, the neural seq2seq model learns to produce a sequence of operations. An OSNMT operation sequence describes a translation (the `compiled' target sentence) and explains each target token with a hard link into the source sentence. OSNMT keeps track of the positions of a source-side read head and a target-side write head. The read head monotonically walks through the source sentence, whereas the position of the write head can be moved from marker to marker in the target sentence. OSNMT defines the following operations to control head positions and produce output words.

\begin{itemize}
\item \texttt{POP\_SRC}: Move the read head right by one token.
\item \texttt{SET\_MARKER}: Insert a marker symbol into the target sentence at the position of the write head.
\item \texttt{JMP\_FWD}: Move the write head to the nearest marker right of the current head position in the target sentence.
\item \texttt{JMP\_BWD}: Move the write head to the nearest marker left of the current head position in the target sentence.
\item \texttt{INSERT}($t$): Insert a target token $t$ into the target sentence at the position of the write head.
\end{itemize}

Tab.~\ref{tab:osm-breakdown} illustrates the generation of a Japanese-English translation in detail. The neural seq2seq model is trained to produce the sequence of operations in the first column of Tab.~\ref{tab:osm-breakdown}. The initial state of the target sentence is a single marker symbol $X_1$. Generative operations like \texttt{SET\_MARKER} or \texttt{INSERT}($t$) insert a single symbol left of the current marker (highlighted). The model begins with a \texttt{SET\_MARKER} operation, which indicates that the translation of the first word in the source sentence is not at the beginning of the target sentence. Indeed, after ``translating'' the identities `2000' and `hr', in time step 6 the model jumps back to the marker $X_2$ and continues writing left of `2000'. The translation process terminates when the read head is at the end of the source sentence. The final translation in plain text can be obtained by removing all markers from the (compiled) target sentence.

\subsection{OSNMT Represents Alignments}
\label{sec:osnmt-align}

The word alignment can be derived from the operation sequence by looking up the position of the read head for each generated target token. The alignment for the example in Tab.~\ref{tab:osm-breakdown} is shown in Fig.~\ref{fig:alignment}. Note that similarly to the IBM models~\citep{ibm} and the OSM for SMT~\citep{osm}, our OSNMT can only represent 1:$n$ alignments. Thus, each target token is aligned to exactly one source token, but a source token can generate any number of (possibly non-consecutive) target tokens.

\begin{table*}[t!]
\begin{center}
\small
\begin{tabular}{r|l|l|l|}
\cline{2-4}
 & \bf Operation & \bf Source sentence & \bf Target sentence (compiled) \\ \cline{2-4}

 &		& \begin{CJK}{UTF8}{min}\colorbox{blue!15}{2000} hr の 安定 動作 を 確認 し た\end{CJK}	& \colorbox{blue!15}{$X_1$} \\
1 & \texttt{SET\_MARKER}		& \begin{CJK}{UTF8}{min}\colorbox{blue!15}{2000} hr の 安定 動作 を 確認 し た\end{CJK}	& $X_2$ \colorbox{blue!15}{$X_1$} \\
2 & 2000		& \begin{CJK}{UTF8}{min}\colorbox{blue!15}{2000} hr の 安定 動作 を 確認 し た\end{CJK}	& $X_2$ 2000 \colorbox{blue!15}{$X_1$} \\
3 & \texttt{POP\_SRC}		& \begin{CJK}{UTF8}{min}2000 \colorbox{blue!15}{hr} の 安定 動作 を 確認 し た\end{CJK}	& $X_2$ 2000 \colorbox{blue!15}{$X_1$} \\
4 & hr		& \begin{CJK}{UTF8}{min}2000 \colorbox{blue!15}{hr} の 安定 動作 を 確認 し た\end{CJK}	& $X_2$ 2000 hr \colorbox{blue!15}{$X_1$} \\
5 & \texttt{POP\_SRC}		& \begin{CJK}{UTF8}{min}2000 hr \colorbox{blue!15}{の} 安定 動作 を 確認 し た\end{CJK}	& $X_2$ 2000 hr \colorbox{blue!15}{$X_1$} \\
6 & \texttt{JMP\_BWD}		& \begin{CJK}{UTF8}{min}2000 hr \colorbox{blue!15}{の} 安定 動作 を 確認 し た\end{CJK}	& \colorbox{blue!15}{$X_2$} 2000 hr $X_1$ \\
7 & \texttt{SET\_MARKER}		& \begin{CJK}{UTF8}{min}2000 hr \colorbox{blue!15}{の} 安定 動作 を 確認 し た\end{CJK}	& $X_3$ \colorbox{blue!15}{$X_2$} 2000 hr $X_1$ \\
8 & of		& \begin{CJK}{UTF8}{min}2000 hr \colorbox{blue!15}{の} 安定 動作 を 確認 し た\end{CJK}	& $X_3$ of \colorbox{blue!15}{$X_2$} 2000 hr $X_1$ \\
9 & \texttt{POP\_SRC}		& \begin{CJK}{UTF8}{min}2000 hr の \colorbox{blue!15}{安定} 動作 を 確認 し た\end{CJK}	& $X_3$ of \colorbox{blue!15}{$X_2$} 2000 hr $X_1$ \\
10 & \texttt{JMP\_BWD}		& \begin{CJK}{UTF8}{min}2000 hr の \colorbox{blue!15}{安定} 動作 を 確認 し た\end{CJK}	& \colorbox{blue!15}{$X_3$} of $X_2$ 2000 hr $X_1$ \\
11 & stable		& \begin{CJK}{UTF8}{min}2000 hr の \colorbox{blue!15}{安定} 動作 を 確認 し た\end{CJK}	& stable \colorbox{blue!15}{$X_3$} of $X_2$ 2000 hr $X_1$ \\
12 & \texttt{POP\_SRC}		& \begin{CJK}{UTF8}{min}2000 hr の 安定 \colorbox{blue!15}{動作} を 確認 し た\end{CJK}	& stable \colorbox{blue!15}{$X_3$} of $X_2$ 2000 hr $X_1$ \\
13 & operation	& \begin{CJK}{UTF8}{min}2000 hr の 安定 \colorbox{blue!15}{動作} を 確認 し た\end{CJK}	& stable operation \colorbox{blue!15}{$X_3$} of $X_2$ 2000 hr $X_1$ \\
14 & \texttt{POP\_SRC}		& \begin{CJK}{UTF8}{min}2000 hr の 安定 動作 \colorbox{blue!15}{を} 確認 し た\end{CJK}	& stable operation \colorbox{blue!15}{$X_3$} of $X_2$ 2000 hr $X_1$ \\
15 & \texttt{JMP\_FWD}		& \begin{CJK}{UTF8}{min}2000 hr の 安定 動作 \colorbox{blue!15}{を} 確認 し た\end{CJK}	& stable operation $X_3$ of \colorbox{blue!15}{$X_2$} 2000 hr $X_1$ \\
16 & \texttt{JMP\_FWD}		& \begin{CJK}{UTF8}{min}2000 hr の 安定 動作 \colorbox{blue!15}{を} 確認 し た\end{CJK}	& stable operation $X_3$ of $X_2$ 2000 hr \colorbox{blue!15}{$X_1$} \\
17 & was		& \begin{CJK}{UTF8}{min}2000 hr の 安定 動作 \colorbox{blue!15}{を} 確認 し た\end{CJK}	& stable operation $X_3$ of $X_2$ 2000 hr was \colorbox{blue!15}{$X_1$} \\
18 & \texttt{POP\_SRC}		& \begin{CJK}{UTF8}{min}2000 hr の 安定 動作 を \colorbox{blue!15}{確認} し た\end{CJK}	& stable operation $X_3$ of $X_2$ 2000 hr was \colorbox{blue!15}{$X_1$} \\
19 & \texttt{POP\_SRC}		& \begin{CJK}{UTF8}{min}2000 hr の 安定 動作 を 確認 \colorbox{blue!15}{し} た\end{CJK}	& stable operation $X_3$ of $X_2$ 2000 hr was \colorbox{blue!15}{$X_1$} \\
20 & confirmed	& \begin{CJK}{UTF8}{min}2000 hr の 安定 動作 を 確認 \colorbox{blue!15}{し} た\end{CJK}	& stable operation $X_3$ of $X_2$ 2000 hr was confirmed \colorbox{blue!15}{$X_1$} \\
21 & \texttt{POP\_SRC}		& \begin{CJK}{UTF8}{min}2000 hr の 安定 動作 を 確認 し \colorbox{blue!15}{た} \end{CJK}	& stable operation $X_3$ of $X_2$ 2000 hr was confirmed \colorbox{blue!15}{$X_1$} \\

\cline{2-4}
\end{tabular}
\end{center}
\caption{\label{tab:osm-breakdown} Generation of the target sentence ``stable operation of 2000 hr was confirmed'' from the source sentence ``\begin{CJK}{UTF8}{min}2000 hr の 安定 動作 を 確認 し た\end{CJK}''. The neural model produces the linear sequence of operations in the first column. The positions of the source-side read head and the target-side write head are highlighted. The marker in the target sentence produced by the $i$-th \texttt{SET\_MARKER} operation is denoted with `$X_{i+1}$'; $X_1$ is the initial marker. We denote \texttt{INSERT}($t$) operations as $t$ to simplify notation.
}
\end{table*}

\subsection{OSNMT Represents Hierarchical Structure}
\label{sec:osnmt-hiero}

We can also derive a tree structure from the operation sequence in Tab.~\ref{tab:osm-breakdown} (Fig.~\ref{fig:tree}) in which each marker is represented by a nonterminal node with outgoing arcs to symbols inserted at that marker. The target sentence can be read off the tree by depth-first search traversal (post-order).

More formally, synchronous context-free grammars (SCFGs) generate pairs of strings by pairing two context-free grammars. Phrase-based hierarchical SMT~\citep{hiero} uses SCFGs to model the relation between the source sentence and the target sentence. Multitext grammars (MTGs) are a generalization of SCFGs to more than two output streams~\citep{mtg,gmtg}. We find that an OSNMT sequence can be interpreted as sequence of rules of a tertiary MTG $\mathcal{G}$ which generates 1.) the source sentence, 2.) the target sentence, and 3.) the position of the target side write head. The start symbol of $\mathcal{G}$ is

\begin{equation}
[(S), (X_1), (P_1)]^\mathsf{T}
\end{equation}
which initializes the source sentence stream with a single nonterminal $S$, the target sentence with the initial marker $X_1$ and the position of the write head with 1 ($P_1$). Following \citet{gmtg} we denote rules in $\mathcal{G}$ as

\begin{equation}
[(\alpha_1),(\alpha_2),(\alpha_3)]^\mathsf{T}\rightarrow [(\beta_1),(\beta_2),(\beta_3)]^\mathsf{T}
\end{equation}
where $\alpha_1, \alpha_2,\alpha_3$ are single nonterminals or empty, $\beta_1,\beta_2,\beta_3$ are strings of terminals and nonterminals, and $\alpha_i\rightarrow\beta_i$ for all $i\in\{1,2,3\}$ with nonempty $\alpha_i$ are the rewriting rules for each of the three individual components which need to be applied simultaneously. \texttt{POP\_SRC} extends the source sentence prefix in the first stream by one token.

\begin{figure}[!t]
\centering
\includegraphics[width=0.9\linewidth]{alignment.png}
\caption{The translation and the alignment derived from the operation sequence in Tab.~\ref{tab:osm-breakdown}.}
\label{fig:alignment}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=0.85\linewidth]{tree.png}
\caption{Target-side tree representation of the operation sequence in Tab.~\ref{tab:osm-breakdown}.}
\label{fig:tree}
\end{figure}

\begin{equation}
\mathtt{POP\_SRC}:\forall s\in\mathcal{V}_{src}: \begin{bmatrix}
(S)\\
()\\
()
\end{bmatrix}\rightarrow \begin{bmatrix}
(sS)\\
()\\
()
\end{bmatrix}
\label{eq:pop-rule}
\end{equation}
where $\mathcal{V}_{src}$ is the source language vocabulary. A jump from marker $X_i$ to $X_j$ is realized by replacing $P_i$ with $P_j$ in the third grammar component:

\begin{equation}
\mathtt{JMP}:\forall i,j\in \mathcal{N}: [(),(), P_i]^\mathsf{T} \rightarrow [(),(),(i P_j)]^\mathsf{T}
\label{eq:jump-rule}
\end{equation}
where $\mathcal{N}=\{k\in\mathbb{N}|k\leq n\}$ is the set of the first $n$ natural numbers for a sufficiently large $n$. The generative operations (\texttt{SET\_MARKER} and \texttt{INSERT}($t$)) insert symbols into the second component.

\begin{table}[t!]
\begin{center}
\small
\begin{tabular}{l|l}
\textbf{Derivation} & \textbf{OSNMT} \\
\hline
$[(S), (X_1), P_1]^\mathsf{T}$ & \\
$\overset{\text{Eq.~\ref{eq:pop-rule}}}{\rightarrow}[(\text{2000 } S), (X_1), P_1]^\mathsf{T}$ & \texttt{SET\_MARKER} \\
$\overset{\text{Eq.~\ref{eq:set-marker-rule}}}{\rightarrow} [(\text{2000 } S), (X_2X_1), (P_1)]^\mathsf{T}$ & 2000 \\
$\overset{\text{Eq.~\ref{eq:token-rule}}}{\rightarrow} [(\text{2000 } S), (X_2\text{ 2000 }X_1), (P_1)]^\mathsf{T}$ & \texttt{POP\_SRC} \\
$\overset{\text{Eq.~\ref{eq:pop-rule}}}{\rightarrow} \begin{bmatrix}
(\text{2000 hr } S)\\
(X_2\text{ 2000 }X_1)\\
(P_1)
\end{bmatrix}$ & hr \\
$\overset{\text{Eq.~\ref{eq:token-rule}}}{\rightarrow} \begin{bmatrix}
(\text{2000 hr } S)\\
(X_2\text{ 2000 hr }X_1)\\
(P_1)
\end{bmatrix}$ & \texttt{POP\_SRC} \\
$\overset{\text{Eq.~\ref{eq:pop-rule}}}{\rightarrow} \begin{bmatrix}
(\text{2000 hr \begin{CJK}{UTF8}{min}の\end{CJK} } S) \\
(X_2\text{ 2000 hr }X_1) \\
(P_1)
\end{bmatrix}$ & \texttt{JMP\_BWD} \\
$\overset{\text{Eq.~\ref{eq:jump-rule}}}{\rightarrow} \begin{bmatrix}
(\text{2000 hr \begin{CJK}{UTF8}{min}の\end{CJK} } S)\\
(X_2\text{ 2000 hr }X_1)\\
(\text{1 }P_2)
\end{bmatrix}$ & \texttt{SET\_MARKER} \\
$\overset{\text{Eq.~\ref{eq:set-marker-rule}}}{\rightarrow} \begin{bmatrix}
(\text{2000 hr \begin{CJK}{UTF8}{min}の\end{CJK} } S)\\
(X_3X_2\text{ 2000 hr }X_1)\\
(\text{1 }P_2)
\end{bmatrix}$ & of \\
$\overset{\text{Eq.~\ref{eq:token-rule}}}{\rightarrow} \begin{bmatrix}
(\text{2000 hr \begin{CJK}{UTF8}{min}の\end{CJK} } S) \\
(X_3\text{ of }X_2\text{ 2000 hr }X_1)\\
(\text{1 }P_2)
\end{bmatrix}$ & ... \\
... & \\
\end{tabular}
\end{center}
\caption{\label{tab:derivation} Derivation in $\mathcal{G}$ for the example of Tab.~\ref{tab:osm-breakdown}.}
\end{table}

\begin{align}
\label{eq:set-marker-rule} \mathtt{SET\_MARKER}:\forall i\in \mathcal{N}: \begin{bmatrix}
()\\
(X_i)\\
(P_i)
\end{bmatrix} \rightarrow \begin{bmatrix}
()\\
(X_{i+1}X_i)\\
(P_i)
\end{bmatrix} \\
\label{eq:token-rule} \mathtt{INSERT}:\forall i\in \mathcal{N}, t\in\mathcal{V}_{trg}: \begin{bmatrix}
()\\
(X_i)\\
(P_i)
\end{bmatrix} \rightarrow \begin{bmatrix}
()\\
(tX_i)\\
(P_i)
\end{bmatrix}
\end{align}
where $\mathcal{V}_{trg}$ is the target language vocabulary. The identity mapping $P_i\rightarrow P_i$ in the third component enforces that the write head is at marker $X_i$. We note that $\mathcal{G}$ is not only context-free but also regular in the first and third components (but not in the second component due to Eq.~\ref{eq:set-marker-rule}). Rules of the form in Eq.~\ref{eq:token-rule} are directly related to alignment links (cf.\ Fig.~\ref{fig:alignment}) as they represent the fact that target token $t$ is aligned to the last terminal symbol in the first stream. We formalize removing markers/nonterminals at the end by introducing a special nonterminal $T$ which is eventually mapped to the end-of-sentence symbol EOS:

\begin{align}
[(S),(),()]^\mathsf{T}\rightarrow [(T), (),()]^\mathsf{T} \\
[(T),(),()]^\mathsf{T}\rightarrow [(\text{EOS}), (),()]^\mathsf{T}\\
\forall i\in \mathcal{N}: [(T),  (X_i), ()]^\mathsf{T}\rightarrow[(T), (\epsilon), ()]^\mathsf{T} \\
\forall i\in \mathcal{N}: [(T),  (), (P_i)]^\mathsf{T}\rightarrow[(T), (), (\epsilon)]^\mathsf{T}
\end{align}

Tab.~\ref{tab:derivation} illustrates that there is a 1:1 correspondence between a derivation in $\mathcal{G}$ and an OSNMT operation sequence. The target-side derivation (the second component in $\mathcal{G}$) is structurally similar to a binarized version of the tree in Fig.~\ref{fig:tree}. However, we assign scores to the structure via the corresponding OSNMT sequence which does not need to obey the usual conditional independence assumptions in hierarchical SMT. Therefore, even though $\mathcal{G}$ is context-free in the second component, our scoring model for $\mathcal{G}$ is more powerful as it conditions on the OSNMT history which potentially contains context information. Note that OSNMT is deficient~\citep{ibm} as it assigns non-zero probability mass to any operation sequence, not only those with derivation in $G$.

We further note that subword-based OSNMT can potentially represent any alignment to any target sentence as long as the alignment does not violate the 1:$n$ restriction. This is in contrast to  phrase-based SMT where reference translations often do not have a derivation in the SMT system due to coverage problems~\citep{smt-searchspace}.

\subsection{Comparison to the OSM for SMT}
\label{sec:osnmt-osm-diff}

Our OSNMT set of operations (\texttt{POP\_SRC}, \texttt{SET\_MARKER}, \texttt{JMP\_FWD}, \texttt{JMP\_BWD}, and \texttt{INSERT}($t$)) is inspired by the original OSM for SMT~\citep{osm} as it also represents the translation process as linear sequence of operations. However, there are significant differences which make OSNMT more suitable for neural models. First, OSNMT is monotone on the source side, and allows jumps on the target side. SMT-OSM operations jump in the source sentence. We argue that source side monotonicity potentially mitigates coverage issues of neural models (over- and under-translation~\citep{nmt-coverage}) as the attention can learn to scan the source sentence from left to right. Another major difference is that we use {\em markers} rather than {\em gaps}, and do not close a gap/marker after jumping to it. This is an implication of OSNMT jumps being defined on the target side since the size of a span is unknown at inference time.

\section{Training}
\label{sec:training}

\begin{algorithm}[t!]
\caption{Align2OSNMT($a$, $\mathbf{x}$, $\mathbf{y}$)}
\label{alg:align2osm}
\begin{algorithmic}[1]
\STATE{$holes\gets \{(0, \infty)\}$}
\STATE{$ops\gets\langle\rangle$}
\COMMENT{Initialize with empty list}
\STATE{$head\gets 0$}
\FOR{$i\gets 1$ \TO $|\mathbf{x}|$}
  \FORALL{$j \in \{j|a_j=i\}$}
    \STATE{$hole\_idx\gets holes.\text{find}(j)$}
    \STATE{$d\gets  hole\_idx - head$}
    \IF{$d < 0$}
      \STATE{$ops.\text{extend}(\mathtt{JMP\_BWD}.\text{repeat}(-d))$}
    \ENDIF
    \IF{$d > 0$}
      \STATE{$ops.\text{extend}(\mathtt{JMP\_FWD}.\text{repeat}(d))$}
    \ENDIF
    \STATE{$head \gets hole\_idx$}
    \STATE{$(s, t) \gets holes[head]$}
    \IF{$s \neq j$}
      \STATE{$holes.\text{append}((s, j-1))$}
      \STATE{$head\gets head + 1$}
      \STATE{$ops.\text{append}(\mathtt{SET\_MARKER})$}
    \ENDIF
    \STATE{$ops.\text{append}(\mathtt{y_j})$}
    \STATE{$holes[head]\gets (j + 1, t)$}
  \ENDFOR
  \STATE{$ops.\text{append}(\mathtt{SRC\_POP})$}
\ENDFOR
\RETURN{ops}
\end{algorithmic}
\end{algorithm}

We train our Transformer model as usual by minimising the negative log-likelihood of the target sequence. However, in contrast to plain text NMT, the target sequence is not a plain sequence of subword or word tokens but a sequence of operations. Consequently, we need to map the target sentences in the training corpus to OSNMT representations. We first run a statistical word aligner like Giza++~\citep{giza} to obtain an aligned training corpus. We delete all alignment links which violate the 1:$n$ restriction of OSNMT (cf.\ Sec.~\ref{sec:osnmt}). The alignments together with the target sentences are then used to generate the reference operation sequences for training. The algorithm for this conversion is shown in Alg.~\ref{alg:align2osm}.\footnote{A Python implementation is available at \url{https://github.com/fstahlberg/ucam-scripts/blob/master/t2t/align2osm.py}.} Note that an operation sequence represents one specific alignment, which means that the only way for an OSNMT sequence to be generated correctly is if both the word alignment and the target sentence are also correct. Thereby, the neural model learns to align and translate at the same time. However, there is spurious ambiguity as one alignment can be represented by different OSNMT sequences. For instance, simply adding a \texttt{SET\_MARKER} operation at the end of an OSNMT sequence does not change the alignment represented by it.

\section{Results}

\begin{table}[t!]
\begin{center}
\small
\begin{tabular}{l|l|r}
\textbf{Corpus} & \textbf{Language pair} & \textbf{\# Sentences} \\
\hline
Scielo & Spanish-English & 587K \\
Scielo & Portuguese-English & 513K \\
WAT & Japanese-English & 1M \\
\end{tabular}
\end{center}
\caption{\label{tab:train-set-sizes} Training set sizes.}
\end{table}

We evaluate on three language pairs: Japanese-English (ja-en), Spanish-English (es-en), and Portuguese-English (pt-en). We use the ASPEC corpus~\citep{aspec} for ja-en and the health science portion of the Scielo corpus~\citep{scielo} for es-en and pt-en. Training set sizes are summarized in Tab.~\ref{tab:train-set-sizes}. We use byte pair encoding~\citep{bpe} with 32K merge operations for all systems (joint encoding models for es-en and pt-en and separate source/target models for ja-en). We trained Transformer models~\citep{transformer}\footnote{We follow the \texttt{transformer\_base} configuration and use 6 layers, 512 hidden units, and 8 attention heads in both the encoder and decoder.} until convergence (250K steps for plain text, 350K steps for OSNMT) on a single GPU using Tensor2Tensor~\citep{t2t} after removing sentences with more than 250 tokens. Batches contain around 4K source and 4K target tokens. Transformer training is very sensitive to the batch size and the number of GPUs~\citep{t2t-training}. Therefore, we delay SGD updates~\citep{SaundersACL18} to every 8 steps to simulate 8 GPU training as recommended by~\citet{transformer}. Based on the performance on the ja-en dev set we decode the plain text systems with a beam size of 4 and OSNMT with a beam size of 8 using our SGNMT decoder~\citep{sgnmt}. We use length normalization for ja-en but not for es-en or pt-en. We report cased \texttt{multi-bleu.pl} BLEU scores on the tokenized text to be comparable with the WAT evaluation campaign on ja-en.\footnote{\url{http://lotus.kuee.kyoto-u.ac.jp/WAT/evaluation/list.php?t=2&o=4}}.

\paragraph{Generating training alignments}

\begin{table}[t!]
\begin{center}
\small
\begin{tabular}{l|r|r|}
 & \multicolumn{2}{c|}{\textbf{BLEU}}  \\
\textbf{Method} & \textbf{es-en} & \textbf{pt-en} \\
\hline
Align on subword level & 36.7 & 38.1 \\
Convert word level alignments & 37.1 & 38.4 \\
\end{tabular}
\end{center}
\caption{\label{tab:how-to-align} Generating training alignments on the subword level.}
\end{table}

\begin{table}[t!]
\begin{center}
\small
\begin{tabular}{l|r}
\textbf{Type} & \textbf{Frequency} \\
\hline
\textbf{Valid} & \textbf{92.49\%} \\
Not enough \texttt{SRC\_POP} & 7.28\% \\
Too many \texttt{SRC\_POP} & 0.22\% \\
Write head out of range & 0.06\% \\
\end{tabular}
\end{center}
\caption{\label{tab:constraints} Frequency of invalid OSNMT sequences produced by an unconstrained decoder on the ja-en test set.}
\end{table}

As outlined in Sec.~\ref{sec:training} we use Giza++~\citep{giza} to generate alignments for training OSNMT. We experimented with two different methods to obtain alignments on the subword level. First, Giza++ can directly align the source-side subword sequences to target-side subword sequences. Alternatively, we can run Giza++ on the word level, and convert the word alignments to subword alignments in a postprocessing step by linking subwords if the words they belong to are aligned with each other. Tab.~\ref{tab:how-to-align} compares both methods and shows that converting word alignments is marginally better. Thus, we use this method in all other experiments.

\paragraph{Constrained beam search}

Unconstrained neural decoding can yield invalid OSNMT sequences. For example, the \texttt{JMP\_FWD} and \texttt{JMP\_BWD} operations are undefined if the write head is currently at the position of the last or first marker, respectively. The number of \texttt{SRC\_POP} operations must be equal to the number of source tokens in order for the read head to scan the entire source sentence. Therefore, we constrain these operations during decoding. We have implemented the constraints in our publicly available SGNMT decoding platform~\citep{sgnmt}. However, these constraints are only needed for a small fraction of the sentences. Tab.~\ref{tab:constraints} shows that even unconstrained decoding yields valid OSNMT sequences in 92.49\% of the cases.

\begin{table}[t!]
\begin{center}
\small
\begin{tabular}{l|r|r|r|r|}
  & \multicolumn{4}{c|}{\textbf{BLEU}} \\
  & \textbf{es-en} & \textbf{pt-en} & \multicolumn{2}{c|}{\textbf{ja-en}} \\
\textbf{Representation} & & & \textbf{dev} & \textbf{test} \\
\hline
Plain & 37.6 & 37.5 & 28.3 & 28.1 \\
OSNMT & 37.1 & 38.4 & 28.1 & 28.8 \\
\end{tabular}
\end{center}
\caption{\label{tab:plain-vs-osnmt} Comparison between plain text and OSNMT on Spanish-English (es-en), Portuguese-English (pt-en), and Japanese-English (ja-en).}
\end{table}

\begin{table*}[t!]
\begin{center}
\footnotesize
\begin{tabular}{|p{15cm}|}
\hline
\begin{center}\includegraphics[scale=0.75]{pten_alignment3.png}\end{center} \\
\textbf{Operation sequence}: \texttt{SRC\_POP} ab road\_ \texttt{SRC\_POP} as\_ \texttt{SRC\_POP} an\_ indicator\_ \texttt{SRC\_POP} of\_ \texttt{SRC\_POP} performance\_ \texttt{SRC\_POP} and\_ \texttt{SRC\_POP} \texttt{SRC\_POP} temper ament\_ \texttt{SRC\_POP} \\
\textbf{Reference}:  the body shape as an indicative of performance and temperament \\
\hline
\begin{center}\includegraphics[scale=0.75]{pten_alignment4.png}\end{center} \\
\textbf{Operation sequence}: behavior\_ \texttt{SRC\_POP} of\_ \texttt{SRC\_POP} \texttt{SET\_MARKER} clones\_ \texttt{SRC\_POP} \texttt{SRC\_POP} \texttt{SRC\_POP} \texttt{SRC\_POP} \texttt{SRC\_POP} \texttt{JMP\_BWD} pen n is et um\_ \texttt{SRC\_POP} \texttt{JMP\_FWD} subjected\_ \texttt{SRC\_POP} to\_ \texttt{SRC\_POP} \texttt{SET\_MARKER} periods\_ \texttt{SRC\_POP} \texttt{SRC\_POP} \texttt{JMP\_BWD} \texttt{SET\_MARKER} restriction\_ \texttt{SRC\_POP} \texttt{JMP\_BWD} \texttt{SET\_MARKER} water\_ \texttt{SRC\_POP} \texttt{JMP\_BWD} controlled\_ \texttt{SRC\_POP} \\
\textbf{Reference}:  response of pennisetum clons to periods of controlled hidric restriction \\
\hline
\begin{center}\includegraphics[scale=0.75]{pten_alignment2.png}\end{center} \\
\textbf{Operation sequence}: \texttt{SET\_MARKER} to\_ \texttt{SRC\_POP} analyze\_ \texttt{SRC\_POP} these\_ \texttt{SRC\_POP} data\_ \texttt{SRC\_POP} \texttt{JMP\_BWD} \texttt{SET\_MARKER} should\_ be\_ \texttt{SRC\_POP} used\_ \texttt{SRC\_POP} \texttt{JMP\_BWD} \texttt{SET\_MARKER} methodologies\_ \texttt{SRC\_POP}> \texttt{JMP\_BWD} appropriate\_ \texttt{SRC\_POP} \texttt{JMP\_FWD} \texttt{JMP\_FWD} \texttt{JMP\_FWD} .\_ \texttt{SRC\_POP} \\
\textbf{Reference}:  to analyze these data suitable methods should be used . \\
\hline
\end{tabular}
\end{center}
\caption{\label{tab:pten-alignments} Examples of Portuguese-English translations together with their (subword-)alignments induced by the operation sequence. Alignment links from source words consisting of multiple subwords were mapped to the final subword in the training data, visible for `temperamento' and `pennisetum'.}
\end{table*}

\paragraph{Comparison with plain text NMT}

Tab.~\ref{tab:plain-vs-osnmt} compares our OSNMT systems with standard plain text models on all three language pairs. OSNMT performs better on the pt-en and ja-en test sets, but slightly worse on es-en. We think that more engineering work such as optimizing the set of operations or improving the training alignments could lead to more consistent gains from using OSNMT. However, we leave this to future work since the main motivation for this paper is explainable NMT and not primarily improving translation quality.

\paragraph{Alignment quality}
\label{sec:pten_alignment}

Tab.~\ref{tab:pten-alignments} contains example translations and subword-alignments generated from our Portuguese-English OSNMT model. Alignment links from source words consisting of multiple subwords are mapped to the final subword, visible for the words `temperamento' in the first example and `pennisetum' in the second one. The length of the operation sequences increases with alignment complexity as operation sequences for monotone alignments consist only of \texttt{INSERT}($t$) and \texttt{SRC\_POP} operations (example 1). However, even complex mappings are captured very well by OSNMT as demonstrated by the third example. Note that OSNMT can represent long-range reorderings very efficiently: the movement from `para' in the first position to `to' in the tenth position is simply achieved by starting the operation sequence with `\texttt{SET\_MARKER} to' and a \texttt{JMP\_BWD} operation later. The first example in particular demonstrates the usefulness of such alignments as the wrong lexical choice (`abroad' rather than `body shape') can be traced back to the source word `exterior`.

\begin{figure}[!t]
\centering
\includegraphics[width=1\linewidth]{aer-bleu.pdf}
\caption{AER and BLEU training curves for OSNMT on the Japanese-English dev set.}
\label{fig:aer-bleu}
\end{figure}

\begin{table*}[t!]
\begin{center}
\small
\begin{tabular}{l|l|r|r|}
 &  &  \multicolumn{2}{c|}{\textbf{AER (in \%)}} \\
\textbf{Representation} & \textbf{Alignment extraction} & \textbf{dev} & \textbf{test} \\
\hline
Plain & LSTM forced decoding & 63.9 & 63.7 \\
Plain & LSTM forced decoding with supervised attention \citep[{\em Cross Entropy} loss]{supervised-att3} & 54.9 & 54.7 \\
OSNMT & OSNMT & 24.2 & 21.5 \\
\end{tabular}
\end{center}
\caption{\label{tab:rnn-vs-osnmt} Comparison between OSNMT and using the attention matrix from forced decoding with a recurrent model.}
\end{table*}

\begin{figure*}
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth]{encdec1.png}
        \caption{Layer 4, head 1; attending to the source side read head.}
        \label{fig:encdec1}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth]{encdec2.png}
        \caption{Layer 2, head 3; attending to the right trigram context of the read head.}
        \label{fig:encdec2}
    \end{subfigure}
    \caption{Encoder-decoder attention weights.}\label{fig:encdec}
\end{figure*}

For a qualitative assessment of the alignments produced by OSNMT we ran Giza++ to align the generated translations to the source sentences, enforced the $1$:$n$ restriction of OSNMT, and used the resulting alignments as reference for computing the alignment error rate~\citep[AER]{giza}. Fig.~\ref{fig:aer-bleu} shows that as training proceeds, OSNMT learns to both produce high quality translations (increasing BLEU score) and accurate alignments (decreasing AER).

As mentioned in the introduction, a light-weight way to extract $1$:$n$ alignments from a vanilla attentional LSTM-based seq2seq model is to take the maximum over attention weights for each target token. This is possible because, unlike the Transformer, LSTM-based models usually only have a single soft attention matrix. However, in our experiments, LSTM-based NMT was more than 4.5 BLEU points worse than the Transformer on Japanese-English. Therefore, to compare AERs under comparable BLEU scores, we used the LSTM-based models in forced decoding mode on the output of our plain text Transformer model from Tab.~\ref{tab:plain-vs-osnmt}. We trained two different LSTM models: one standard model by optimizing the likelihood of the training set, and a second one with supervised attention following \citet{supervised-att3}. Tab.~\ref{tab:rnn-vs-osnmt} shows that the supervised attention loss of \citet{supervised-att3} improves the AER of the LSTM model. However, OSNMT is able to produce much better alignments since it generates the alignment along with the translation in a single decoding run.

\paragraph{OSNMT sequences contain target words in source sentence order}

An OSNMT sequence can be seen as a sequence of target words in source sentence order, interspersed with instructions on how to put them together to form a fluent target sentence. For example, if we strip out all \texttt{SRC\_POP}, \texttt{SET\_MARKER}, \texttt{JMP\_FWD}, and \texttt{JMP\_BWD} operations in the OSNMT sequence in the second example of Tab.~\ref{tab:pten-alignments} we get:

\begin{quote}
behavior of clones pennisetum subjected to periods restriction water controlled
\end{quote}

The word-by-word translation back to Portugese is:

\begin{quote}
comportamento de clones pennisetum submetidos a períodos restrição hídrica controlada
\end{quote}

This restores the original source sentence (cf.\ Tab.~\ref{tab:pten-alignments}) up to unaligned source words. Therefore, we can view the operations for controlling the write head (\texttt{SET\_MARKER}, \texttt{JMP\_FWD}, and \texttt{JMP\_BWD}) as reordering instructions for the target words which appear in source sentence word order within the OSNMT sequence.

\paragraph{Role of multi-head attention}
\label{sec:att}

In this paper, we use a standard seq2seq model (the Transformer architecture~\citep{transformer}) to generate OSNMT sequences from the source sentence. This means that our neural model is representation-agnostic: we do not explicitly incorporate the notion of read and write heads into the neural architecture. In particular, neither in training nor in decoding do we explicitly bias the Transformer's attention layers towards consistency with the alignment represented by the OSNMT sequence. Our Transformer model has 48 encoder-decoder attention matrices due to multi-head attention (8 heads in each of the 6 layers). We have found that many of these attention matrices have strong and interpretable links to the translation process represented by the OSNMT sequence. For example, Fig.~\ref{fig:encdec1} shows that the first head in layer 4 follows the source-side read head position very closely: at each \texttt{SRC\_POP} operation the attention shifts by one to the next source token. Other attention heads have learned to take other responsibilities. For instance, head 3 in layer 2 (Fig.~\ref{fig:encdec2}) attends to the trigram right of the source head.

\section{Related Work}

Explainable and interpretable machine learning is attracting more and more attention in the research community~\citep{xai-blackbox,xai-survey}, particularly in the context of natural language processing~\citep{visual-rnn,visual-nlp,causal-seq2seq,visual-nmt,seq2seq-pathologies}. These approaches aim to explain (the predictions of) an existing model. In contrast, we change the target representation such that the generated sequences themselves convey important information about the translation process such as the word alignments.

Despite considerable consensus about the importance of word alignments in practice~\citep{nmt-challenges}, e.g.\ to enforce constraints on the output~\citep{terminology-constraints} or to preserve text formatting, introducing explicit alignment information to NMT is still an open research problem. Word alignments have been used as supervision signal for the NMT attention model~\citep{supervised-att1,supervised-att2,supervised-att3,supervised-att-rwth}. \citet{ibm-like-attention} showed how to reintroduce concepts known from traditional statistical alignment models~\citep{ibm} like fertility and agreement over translation direction to NMT. Some approaches to simultaneous translation explicitly control for reading source tokens and writing target tokens and thereby generate monotonic alignments on the segment level~\citep{simtrans-blunsom,simtrans-blunsom-noisy-channel,simtrans-neubig}. \citet{alignment-based-nmt} used separate alignment and lexical models and thus were able to hypothesize explicit alignment links during decoding. While our motivation is very similar to \citet{alignment-based-nmt}, our approach is very different as we represent the alignment as operation sequence, and we do not use separate models for reordering and lexical translation.

The operation sequence model for SMT~\citep{osm,osm-journal} has been used in a number of MT evaluation systems~\citep{eval-uedin-wmt14,eval-rwth-iwslt,eval-qcri-iwslt} and for post-editing~\citep{osm-post-editing}, often in combination with a phrase-based model. The main difference to our OSNMT is that we have adapted the set of operations for neural models and are able to use it as stand-alone system, and not on top of a phrase-based system.

Our operation sequence model has some similarities with transition-based models used in other areas of NLP~\citep{transition-based-parsing-rnn1,transition-based-parsing-rnn2,morph-hard-alignment}. In particular, our \texttt{POP\_SRC} operation is very similar to the {\em step} action of the hard alignment model of \citet{morph-hard-alignment}. However, \citet{morph-hard-alignment} investigated monotonic alignments for morphological inflections whereas we use a larger operation/action set to model complex word reorderings in machine translation.

\section{Conclusion}

We have presented a way to use standard seq2seq models to generate a translation together with an alignment as linear sequence of operations. This greatly improves the interpretability of the model output as it establishes explicit alignment links between source and target tokens. However, the neural architecture we used in this paper is representation-agnostic, i.e.\ we did not explicitly incorporate the alignments induced by an operation sequence into the neural model. For future work we are planning to adapt the Transformer model, for example by using positional embeddings of the source read head and the target write head in the Transformer attention layers.

\section*{Acknowledgments}

This work was supported in part by the U.K. Engineering and Physical Sciences Research Council (EPSRC grant EP/L027623/1). We thank Joanna Stadnik who produced the recurrent translation and alignment models during her 4th year project.

redpajama_set_name: RedPajamaArXiv


Austria's Max Franz was victorious in the weather-plagued men's World Cup super-G in Beaver Creek, Colorado, on Saturday. After heavy snowfall forced the original start to be lowered and the start time pushed back a full hour to 12:00pm MTN, racing action finally got underway with snow and variable visibility factoring in to each racer's run.
Franz was followed in second place by Switzerland's Mauro Caviezel, who captured his second podium in as many days after finishing in third in yesterday's downhill, finishing 0.33 seconds shy of Franz. There was a rare three-way tie for third with a pair of Norwegians, Aleksander Aamodt Kilde and Aksel Lund Swindal, joining Italy's Dominik Paris at 0.41 seconds back.
Travis Ganong led the Americans in 15th place, 0.78 seconds back, and was followed by Ryan Cochran-Siegle in 17th, Steven Nyman in 20th, Ted Ligety in 22nd, and Thomas Biesemeyer in 26th.
The Austrian skied her career-best with two victories in Lake Louise this weekend.
2018 German Skier of the Year crashed hard in Beaver Creek.