muc 7 call participation call participation seventh message understanding system evaluation message understanding conference muc 7 evaluation 2 6 march 1998 conference april 1998 washington c area sponsored human language systems tipster text program defense advanced research projects agency information technology office darpa ito message understanding conferences provided ongoing forum assessing state art practice text analysis technology exchanging information innovative computational techniques context fully implemented systems perform realistic tasks evaluations provided researchers potential sponsors customers quantitative means appreciate strengths weaknesses technologies results reported conferences sparked customer interest potential utility technologies seventh message understanding conference muc 7 provide opportunity new experienced muc participants participate flexible evaluation suited development needs abilities provide opportunity select among variety tasks named entity ne coreference co template element te template relationship tr scenario template st two tasks evaluating component technologies ne co use standard generalized markup language sgml output format redesigned information extraction ie task two domainindependent subtasks te tr separated domaindependent subtask st emphases st task portability minimizing human resources required participate evaluation three experimental tracks explore new data sets tasks participation muc 7 actively sought new veteran organizations new redesigned evaluation tasks muc 7 offers good opportunity organizations try new ideas handling nlp problems scientific practical interest without participate entire range tasks conference consist primarily presentations discussions innovative techniques system design test results also opportunity participants demo evaluation systems attendance conference limited evaluation participants guests invited darpa tipster text program conference proceedings including test results published schedule 1 july 97 application deadline participation 15 july 97 release ne co te tr example st training data scorer 8 september 97 release dry run st task definition training data scorer 29 sept 3 oct 97 muc 7 dry run participants 6 february 98 release formal test st task definition training data scorer 2 6 march 98 muc 7 formal run 7 9 april 98 7th message understanding conference tentative dates data task description texts used system development testing news service articles new york times news service supplied linguistic data consortium ldc ldc ldc upenn edu training dry run test data tasks extracted corpus approximately 158 000 articles sets articles used muc 7 evaluation distributed via ftp upon payment one time fee 100 upon signing user agreement use texts user agreement retrieved ldc catalog evaluation agreements url ldc home page http www ldc upenn edu five separate evaluations conducted part muc 7 definition evaluations worked since late 1996 members muc 7 planning committee evaluations may viewed capturing results text analysis various levels aggregation information named entity ne requires system evaluation identify bit pertinent information isolation others coreference co requires connecting references identical entities template element te requires grouping entity attributes together entity objects template relationship tr requires identifying relationships template elements scenario template st requires identifying instances taskspecific event identifying event attributes including entities fill role event overall information content captured via interlinked objects experimental tracks using new data sets variants ne task task definition basic ne task texts different experimental track involving new task simplified version te task key things note evaluation task ne covers named organizations people locations along date time expressions monetary percentage expressions requires production sgml tags output co covers noun phrases common proper personal pronouns identical reference requires production sgml tags output tags coreferring strings form equivalance classes used scoring te covers organizations persons artifacts captured form template objects consisting predefined set attributes tr covers relationships among template elements including location time relationships captured form template relations consisting relationship template elements participating relationship tr new task muc 7 st covers particular scenario kept secret one month prior testing order focus system portability however generalized structure scenario template predefined example scenarios available participants examine task domain dependent tasks experimental tracks derived ne te world wide web site allows automated testing following rules muc 6 particular value new participants website password protected need licensed access acl dci disk ldc obtain password chinchor gso saic com muc 6 articles taken acl dci disk anonymous ftp site available downloading muc 7 related material cfp muc 7 participant agreement available public ftp site participant signing ldc user agreement muc 7 participation agreement receive password download muc 7 data definitions scoring software release times noted url website http muc saic com ftp site ftp muc saic com test protocol evaluation criteria muc 7 participants may elect one combination tasks experimental tracks participants access shared resources training texts annotations templates task documentation scoring software muc 7 participants encouraged participate dry run take advantage material available formal test conducted first week march carried participants sites accordance prepared test procedure results submitted ftp site official scoring software prepared saic muc 7 test sets used evaluations consist 100 texts subsets tasks different data sets dry run formal test systems evaluated using recall precision metrics tasks f measure tasks errorbased metrics tasks except co computation metrics based scoring categories correct partial incorrect spurious missing noncommittal muc 7 participants able familiarize evaluation criteria usage evaluation software released along training data instructions responding call participation organizations within outside u invited respond call participation time actual testing phase evaluation systems must able accept texts without manual preprocessing process without human intervention output annotations ne co templates te tr st expected format organizations plan allocating approximately two personmonths effort participation evaluation conference understood organizations vary respect experience sgml text annotation information extraction domain expertise engineering resources contractual demands expectations etc recognition factors made analyses results organizations wishing participate evaluation conference must respond july 1 1997 submitting short statement interest via email signed copy muc 7 participation agreement via surface mail 1 statement interest submitted via email marsh aic nrl navy mil include following evaluation task choose one named entity coreference template element template relationship scenario template b primary point contact please include name surface email addresses phone fax numbers c site copy muc 6 proceedings 2 participation agreement downloaded anonymous ftp site ftp muc saic com signed copy sent surface mail elaine marsh nrl code 5512 4555 overlook ave sw washington c 203755337 usa questions deferred deadline responding call participation passed may send email elaine marsh marsh aic nrl navy mil copies ralph grishman grishman cs nyu edu nancy chinchor chinchor gso saic com ensure message receives timely response one us muc 7 planning committee ralph grishman new york university program cochair elaine marsh naval research laboratory program cochair chinatsu aone systems research applications lois childs lockheed martin nancy chinchor science applications international jim cowie new mexico state university rob gaizauskas university sheffield megumi kameyama sri international tom keenan u department defense boyan onyshkevych u department defense martha palmer university pennsylvania beth sundheim nccosc nrad marc vilain mitre ralph weischedel bbn systems technologies