CDX Letters (Headers) Meaning A canonized URL B news group C rulespace category D compressed dat file offset F canonized frame G multi-column language description H canonized host I canonized image J canonized jump point K FBIS L canonized link M meta tags (AIF) N massaged URL P canonized path Q language string R canonized redirect U uniqueness V compressed arc file offset * X canonized URL in the other href tags Y canonized URL in other src tags Z canonized URL found in script a original URL b date c old style checksum d uncompressed dat file offset e IP f frame g filename h original host i image j original jump point k new style checksum l link m mime type of original document n arc document length o port p original path r redirect s response code t title v uncompressed arc file offset x URL in the other href tags y URL in other src tags z URL found in script # comment Table 1. CDX and DAT legend (highlighted are default fields) [10] 4. Environment One of the prevalent features of ArchiveSpark is its flexible cross-platform application. In its nature the framework doesn’t limit its user to any particular environment and only requires installed Apache Spark - version 1.6.1 and higher. Being essentially a plain SBT [11]
CDX Letters (Headers) Meaning A B C D F G H I J K L M N P Q R U V X Y Z a b c d e f g h i j k l m n o p r s t v x y z # canonized URL news group rulespace category compressed dat file offset canonized frame multi-column language description canonized host canonized image canonized jump point FBIS canonized link meta tags (AIF) massaged URL canonized path language string canonized redirect uniqueness compressed arc file offset * canonized URL in the other href tags canonized URL in other src tags canonized URL found in script original URL date old style checksum uncompressed dat file offset IP frame filename original host image original jump point new style checksum link mime type of original document arc document length port original path redirect response code title uncompressed arc file offset URL in the other href tags URL in other src tags URL found in script comment Table 1. CDX and DAT legend (highlighted are default fields) [10] 4. Environment One of the prevalent features of ArchiveSpark is its flexible cross-platform application. In its nature the framework doesn’t limit its user to any particular environment and only requires installed Apache Spark - version 1.6.1 and higher. Being essentially a plain SBT [11]
ArchiveSpark-FINAL.docx