Abstract Predicting the approval odds of a patent application is a challenging problem involving multiple factors. The most important factor is arguably the novelty  35 U.S. Code  102 rejects applications that are not sufficiently differentiated from prior art. Novelty evaluation distinguishes the patent approval prediction from conventional document classification  too-similar newer submissions are considered as not novel and would receive the opposite label, thus confusing standard document classifiers (e.g., BERT). To address this issue, we propose a novel framework AISeer that unifies the document classifier with handcrafted features, particularly time-dependent novelty scores. Specifically, we formulate the novelty scores by comparing each application with millions of prior art using a hybrid of efficient filters and a neural bi-encoder. Moreover, we impose a new regularization term into the classification objective to enforce the monotonic change of approval prediction w.r.t. novelty scores, From extensive experiments on a large-scale USPTO dataset, we find that standard BERT fine-tuning can partially learn the correct relationship between novelty and approvals from inconsistent data. However, our time-dependent novelty feature and other handcrafted features offer a significant boost on top of it. Also, our monotonic regularization, while shrinking the search space, can drive the optimizer to better local optima, yielding a further small performance gain. Intellectual property (IP) is an important and integral to the economy. IP-intensive industries directly accounted for 27.9 million jobs in the U.S. (USPTO, 2016) Theoretical and empirical evidence shows that patents are effective in fostering technological progress. (Gallini, 2002; Hu and Png, 2013; Hall and Harhoff, 2012) Securing patent approvals offers a major shot in the arm to inventors  Jingbo Shang is the corresponding author. and innovators, increasing the chances of obtaining angel and venture capital investments. However, the process of getting a patent approved can cost applicants tens of thousands of dollars in payments to law firms who claim to be helpful in understanding what gets approved and improving the odds of success of a patent application. Thus, algorithmic approaches to aid in the patent evaluation process can potentially save precious time and resources for applicants during the patent application phase, as well as benefit patent examiners in government patent offices around the world, accelerating and improving the review process (Ebrahim, 2018). The approval of a patent application , according to U.S. patent laws, is determined necessarily and sufficiently by the approval of application claims . Patent laws define individual claims as the subject matter of inventions ( 35 U.S. Code 112), on which patentability is defined ( 35 U.S. Code  101, 102, and 103) (refer to Appendix B). No overall assessment of a patent application is provisioned. In practice, application claims demarcate the scope of legal protection that an applicant is seeking and are the eventual objects for investigation under legal disputes or transfer of commercial rights. Patent examiners from the U.S. Patent and Trademark Office (USPTO) make decisions on each application claim individually and independently with other sections as supporting materials. Therefore we focus on claim texts and use the term  patent approval  informally and interchangeably referring to  claims approval . In particular, we primarily consider 35 U.S. Code  102 , assessing the novelty of application claims. To the best of our knowledge, we are the first to try to predict patent (claim) approval, which is as an extremely challenging problem for multiple reasons. First, patent documents comprise of technically nuanced and challenging to parse language (intricate legalese). Patent texts are usually legal and technical descriptions of objects or pro-349 cesses, which tend to be complex in vocabulary and grammatical structures (Singer and Smith, 1967). Claims are examined not only literally, but also for their legal implications. Appendix A provides a few example application claims. Second, the patent examination process tends to suffer from subjectivity and inconsistencies (O'Neill, 2018a), exemplified by variance across offices and groups, (O'Neill, 2018b) and across human examiners. In FY17, only 66% of primary examiners are within a 12.5% delta off the average allowance rate (USPTO, 2017). Third, at the core of patent examination, evaluation of novelty is time-dependent. Rejections of claims by 35 U.S. Code  102 require examiners to cite prior approved patent claims, prior art , as evidence. More details about the examination process can be found in USPTO (2020). The United States Patent and Trademark Office (USPTO) receives thousands of applications a week; thus a novel application at one time may be dramatically different in the assessment of novelty after a short time period. This means that a classifier can pick up a positive label from an earlier approved application but receive a negative label from an application sometime later with similar technical content, which is deemed no longer novel. Such conflicting information can confuse the classifier and undermine its performance. In other words, the data labels are intrinsically noisy and inconsistent due to the nature of the domain problem. Although AI/ML approaches are often discussed in the patent domain (Aristodemou and Tietze, 2018) such as in the area of information retrieval (Kang et al., 2007; Fujii, 2007; Shalaby and Zadrozny, 2019), applications of deep NLP methods are mostly concerned with classifying the content domains of patents (Verberne et al., 2010; D'hondt et al., 2013; Hu et al., 2016; Lee and Hsiang, 2019). In addition, the extant literature usually explores approved patents rather than applications (Balsmeier et al., 2018). Even to simply classify the topics of approved patents, state-of-the-art document classifiers can only achieve an accuracy of about 69.3% only 2.2% over RoBerta (Zaheer et al., 2020). Due to these issues, patent approval prediction task is much more challenging than topic classification for document classifiers. To mitigate the issues, we first develop several handcrafted features based on domain knowledge for use alongside the language model for context (cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:38)(cid:79)(cid:68)(cid:76)(cid:80)(cid:3)(cid:55)(cid:72)(cid:91)(cid:87)(cid:86) (cid:3)(cid:3)(cid:3)(cid:3)(cid:37)(cid:40)(cid:53)(cid:55) (cid:3)(cid:3)(cid:3)(cid:3)(cid:53)(cid:72)(cid:83)(cid:85)(cid:72)(cid:86)(cid:72)(cid:81)(cid:87)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81) (cid:3)(cid:3)(cid:3)(cid:3)(cid:44)(cid:81)(cid:87)(cid:72)(cid:74)(cid:85)(cid:68)(cid:87)(cid:76)(cid:81)(cid:74)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:90)(cid:76)(cid:87)(cid:75)(cid:3)(cid:37)(cid:40)(cid:53)(cid:55) (cid:3)(cid:3)(cid:3)(cid:3)(cid:51)(cid:85)(cid:72)(cid:71)(cid:76)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81) (cid:48)(cid:82)(cid:81)(cid:82)(cid:87)(cid:82)(cid:81)(cid:76)(cid:70)(cid:3)(cid:53)(cid:72)(cid:74)(cid:88)(cid:79)(cid:68)(cid:85)(cid:76)(cid:93)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81) (cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:43)(cid:68)(cid:81)(cid:71)(cid:16)(cid:70)(cid:85)(cid:68)(cid:73)(cid:87)(cid:72)(cid:71)(cid:3)(cid:3)(cid:3)(cid:41)(cid:72)(cid:68)(cid:87)(cid:88)(cid:85)(cid:72)(cid:86) (cid:38)(cid:79)(cid:68)(cid:76)(cid:80)(cid:16)(cid:47)(cid:72)(cid:89)(cid:72)(cid:79)(cid:3)(cid:49)(cid:82)(cid:89)(cid:72)(cid:79)(cid:87)(cid:92) (cid:39)(cid:82)(cid:70)(cid:17)(cid:16)(cid:47)(cid:72)(cid:89)(cid:72)(cid:79)(cid:3)(cid:48)(cid:68)(cid:91)(cid:17)(cid:3)(cid:54)(cid:76)(cid:80)(cid:76)(cid:79)(cid:68)(cid:85)(cid:76)(cid:87)(cid:92) (cid:38)(cid:79)(cid:68)(cid:76)(cid:80)(cid:16)(cid:47)(cid:72)(cid:89)(cid:72)(cid:79)(cid:3)(cid:54)(cid:87)(cid:85)(cid:88)(cid:70)(cid:87)(cid:88)(cid:85)(cid:68)(cid:79) (cid:50)(cid:87)(cid:75)(cid:72)(cid:85)(cid:3)(cid:36)(cid:83)(cid:83)(cid:79)(cid:76)(cid:70)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81)(cid:41)(cid:72)(cid:68)(cid:87)(cid:88)(cid:85)(cid:72)(cid:86) Figure 1: An overview of our proposed AISeer. and control. The time-dependent nature of the novelty also makes traditional document classifiers not suitable here, because they typically assume that similar instances belong to the same label. To address this challenge, we propose a novel framework AISeer as shown in Figure 1. We formulate a time-dependent novelty score for each patent claim with its semantic similarity against prior approved claims from patent grants , which are final versions of approved patents. Specifically, inside a comprehensive pool comprising millions of grants, we consider those approved before the filing date of the focal application and then measure the maximum semantic similarity score of the focal patent claim matched with all approved claims in the time-dependent sub-pool. To improve computing efficiency, we apply document-level filters to narrow the sub-pool for each claim. Integrating such similarity scores with handcrafted features and BERT, we conduct experiments on the large-scale USPTO dataset and find significant performance gains over fine-tuning a standard BERT alone. All else equal, a patent claim with a higher similarity score, i.e., semantically more similar to prior approved claims, should be less likely to be approved. Hence we propose to impose monotonic regularization on the novelty score so that the loss function has an additional term of the hinge loss to further penalize non-decreasing predictions in the similarity. This effectively restricts the search space for the optimizer to prediction mechanisms that are reasonably consistent with the novelty measure. From our experiments, this regularization significantly impacts the model outputs. Although performance improvements are limited, it can help the optimizer steer away from unfavorable local optima and further improve AUROC. We further discuss the experimental findings in depth to illustrate how BERT and handcrafted features contribute in overcoming the unconventional data issues. metadata, office actions, rejections and citations data into a massive dataset;  We develop a series of handcrafted features to aid the prediction of 35 U.S. Code  102 approval decisions. In particular, we design and analyze a time-dependent feature that measures the novelty of patent applications at the time of filing;  We incorporate the handcrafted features and impose monotonic regularization on the novelty features to shed light on how the intrinsic data inconsistency issues in the domain problem can be mitigated. Reproducibility. We will release the benchmark dataset and our code on GitHub: https://github.com/ acl-2022-towards-comprehensive/ acl-2022-camera-ready . In this section, we formally formulate the novelty-based patent approval problem. We describe the experiment setup, the dataset, and baseline results with common document classifiers. We follow legal definitions under 35 U.S. Code  102 . Despite the popular notions of patent approval or issues, what is actually being approved/rejected are individual claims. Each patent applications A k , k  { 1    M } , sorted by filing dates, comprises of a number of application claims. Given text representation X i , i  { 1    N } , of each application claim, there exist { i k } , k  { 0    M } such that claim representations { X i k  1    X i k } belong to patent application A k . Binary labels y i indicate approval decisions derived from patent rejections and office actions data where y i = 1 indicates claim approvals. We would like to classify application claims according to approval labels. and grant full texts, application metadata, citations, office actions, and rejections (USPTO, d,e). Patent grants are final versions of approved patent applications. Later we will utilize grants for constructing the application novelty feature. To extract labels and create handcrafted features, we utilize both the legacy data system for office actions, rejections and citations made between 2008 and mid-2017 (Lu et al., 2017), and newer v2 APIs that cover mid-2018 onward. For application metadata, we obtain bulk data from PEDS (Patent Examination Data System) (USPTO, b). In order to match all the available labels, we obtain weekly bulk releases for of both utility patent applications and utility patent grants in XML format ranging between 2005 and 2019. In total, we extract 8.8 million patent applications and 3.7 million patent grants during the same time period whose texts are around 730 GB. Dataset Processing. According to patent laws, only one version among possibly a number of revisions is published and available as full-text data. Meanwhile, for a considerate amount of applications, the entire history of office action data and rejection data are available, where allowances or rejections for each individual claim under each legal clause are formally made. Hence we ought to identify the labels associated with the published version among patent examination rejection data and office action data. We take a "snapshot" approach. Given the available publication version of each application as snapshots, the examination decisions of each claim particularly with respect to the snapshot version are processed and attached as classification labels. Therefore, with the huge number of snapshots, regardless of the subsequent actions of the applicant, e.g. abandonment, the model can be kept agnostic of the status in the application pipeline. This way, we allow the model to predict for any version of a patent application so that the attorneys and applicants can evaluate their chances for decision making. Technical preparations for publication of an application generally begin 4 months prior to the projected date of publication. Hence we match the closest office action dates with publication dates minus 4 months which is supposed to be the benchmark date for the available version, so that correct labels can be obtained. Please refer to the essential publication regulations in Appendix D. Data are merged by the application number and ingested into a DBMS. plications under which all corresponding sections of data are available. Because of the data size and to control for computation times, we choose the most recent, around 500K applications and effectively around 9.5 million claims for experiments. Dataset Splits. We split the data into training, validation, and testing sets by their filing dates. The more recent ones are chosen for testing. The size for final experimental data, including the abstract, claim texts, labels, and handcrafted features, is around 15 GB. The dataset is highly imbalanced towards positive labels (see Table 1 Approval %). evaluate the following common document classifiers. Log. Reg. refers to logistics regression using TF-TDF features. Text-CNN (Kim, 2014) with GloVe (Pennington et al., 2014) embeddings as the input. Adam optimizer with learning rate 0.001. 10 epochs' run; batch size as 1024;  LSTM (Hochreiter and Schmidhuber, 1997) with GloVe embeddings as the input. AdamW optimizer with learning rate 0.005 and 10 epochs' run; batch size as 1024;  BERT (Devlin et al., 2018) fine-tuning. AdamW optimizer with learning rate 5e-5 as the optimizer. The number of fine-tuning epochs as 5; batch size as 256. This is the the same model as in the state-of-the-art model, PatentBERT , in patent content classification (Lee and Hsiang, 2019) with a different set of hyper-parameters and balanced class weights. The original PatentBERT model is designed for a different task, and the experimental setting is not suitable for predicting patent approvals, hence we make the tweaks. In all of the models, we impose class weights in the loss functions inversely proportional to the number of class instances, such that two classes are treated equally by the optimizer. For the details, please refer to Section 3.1. The neural models are trained with text inputs processed at a maximum length of 128 tokens per claim and on a single GPU. Evaluation Metrics. Given the imbalanced nature of our dataset, we adopt both the Area Under the Curve for the ROC plot (Fawcett, 2004) (AUROC) and macro F1 score as our evaluation metrics. With AUROC, the predicting performance of the minority class could be taken into consideration with a similar weight as for the majority class (in our Table 2: Benchmarking Common Document Classifiers. case, positive class). Moreover, the probability-based metric can provide more detailed insights into model performances. Therefore, we choose AUROC as our main metric . The macro F1 score is a direct average of F1 scores of both the positive class and the negative class and provides an alternative balanced view of both classes' performances. We treat it as a secondary metric . We compute the maximum macro F1 score (Lipton et al., 2014) by varying the decision threshold for each model. Other traditional measures focused on the positive class performance such as accuracy and recall have little practical implications due to data imbalance. Benchmark Results. Table 2 shows common document classifiers' performance with some naive predictions as references. Results of neural models are reported with the median metrics among several runs with different optimizer random states. Figure 4 in Appendix F further visualizes more details of the ROC curves of these models. One can find that BERT and LSTM are arguably the most effective ones. Therefore, we will focus on BERT and LSTM for further comparisons. Our AISeer framework unifies the document classifier, handcrafted features and monotonic regularization, as shown in Figure 1. It is compatible with almost all document classifiers. In this paper, we choose BERT as the base document classifier to demonstrate the effects as it is widely adopted and also performs well in our benchmark evaluations. After each application claim text is run through the BERT model, the output representation is concatenated with the corresponding handcrafted features. Our handcrafted features include a time-dependent claim-level novelty score, claim-level structural features, document-level similarity scores, and other application metadata features. We further impose a monotonic regularization on the impact of the claim-level novelty score so that the loss function has an additional term of the hinge loss. For self-containedess, we briefly introduce how we use BERT in AISeer. We first utilize BERT to transform the i -th application claim to a text representation X i in batches of a size N b , which is then passed to a linear layer to obtain the prediction through a softmax layer. Approvals (i.e., y i = 1 ) are much more popular than rejections (refer to Table 1), so the vanilla training will bias the model towards approvals. Therefore, we adopt a weighted loss for training: L = (cid:80) i  w y i ( y i log  y i + (1  y i ) log(1   y i )) where w y i denotes the fixed weights of the two classes, which is inversely proportional to the number of instances from the corresponding class, balancing the training weights of the two classes. The backbone of the novelty feature is the time-dependent claim-level maximum similarity score. We first index all patent grants with ElasticSearch (NV). Given a patent application and a claim under it, we first take advantage of its fast BM25-based document-level fuzzy matches to obtain the 5 most similar grant documents to the focal application document as a first-stage pre-filter. To account for time-dependence, each focal application is matched against a sub-pool of patent grants which are time-stamped to be approved strictly before the filing date of the focal application. In application level matching, all document sections are considered, including the abstract, summary of invention, and details of invention of all claims. Among all claims under the top-5 matched grants, we then find the most similar one to the focal claim using sentence-transformer (Reimers and Gurevych, 2019) with stsb-roberta-large pre-trained bi-encoder model. Base cross-encoder transformers such as BERT can lack in performance for pure semantic similarity tasks. Although certain cross-encoders have excellent semantic similarity performance, it can be computationally too demanding for our purpose since the scale of the claims in all patent grants is more than 100 million, and since each grant claim can be required to be paired many times with a focal application claim. The Elasticsearch-based pre-filter process also helps manage the computational need. Figure 2 demonstrates how the time-dependent novelty feature is generated  the application that the red-highlighted focal claim belongs to is first matched with 5 patent grants on the application level; then the focal claim is matched against every claim under the 5 matched grants to compute the semantic similarity score, before the most similar grant claim is identified. Our experiments confirm that the claim-level maximum similarity score, as expected, is negatively correlated with 35 U.S. Code  102 labels, as shown in Figure 3. 3.3 Application-Level Handcrafted Features Application-Level Similarity. We consider the application-level maximum similarity score, denoted as N s,doc , and mean similarity score generated by ElasticSearch (NV) as handcrafted features. These document-level scores measure how similar overall are the applications to the approved grants. The document-level similarity scores are positively correlated with 35 U.S. Code  102 labels. We believe that they primarily capture the overall writing quality and the common language patterns of approvable applications. offers a rich collection of metadata about each patent application. We use the following two of them:  Patent Classification : the USPC class designated for the applications. USPC (USPTO, a) is a system of classifying the subject matter of each patent application for recording, publication, and assignment purposes. Different classes of patents tend to have varying approval rates (see Table 6 in Appendix C). Number of Applicant Cited References : the number of citations of other patents or articles initiated by the applicant herself. In the patent domain, most citations are initiated by the examiners as prior arts to reject application claims. However, they can also be made by the applicant to demonstrate understanding of related work and claim contributions. The number of applicant-initiated citations is a signal of the effort and 353 200 250 300 350 400 450 Max Similarity Score (Document) 0.78 0.80 0.82 0.84 0.86 P o s i t i v e I n s t a n c e P r o p o r t i o n Max Citation : based on ElasticSearch pre-filter, the maximum number of total citations among the top 5 most similar patent grant documents to the focal patent application. Max Article Citation refers to the maximum number of citations which are research articles (not other patents) in top matched grants. Lexical Diversity : the richness in the vocabulary of the abstract of the patent application. We consider two indicators for each claim. Component refers to indicator on whether the application claim is describing the components of a system (e.g., a machine, a process, a compound). Other claims may describe the properties or utility of particular components. This is identifiable by the transitional phrases used in the claim. Transitional Phrase refers to indicator on whether a component claim is open , closed , or half-open , which is determined by which transitional phrase is used. Openness or closedness regulates the scope of legal IP protection the applicant enjoys once the patent is approved. Often it is a strategic choice by the applicant and the attorney. If a claim is open , indicated by transitional phrases compris-ing and legal synonyms, any additional components later added to the system are also protected, in contrast to closed claims . Open claims are more difficult to be approved. Other examples of transitional phrases include "consisting essentially of" and "consisting of". These particular language phenomena are well-known in the IP communities and sometimes referred to as patentese (Singer and Smith, 1967). The patent examination manual explicitly discusses these phrases with case law (USPTO, c; Silverman and Stacey, 1996). Now let H i denote other handcrafted features in addition to N s,claim and N s,doc . Figure 3 demonstrates the correlations between some representative handcrafted features and the positive label. Let Z i = X i  H i  N s,claim  N s,doc  { 1 } ,  i  { 1 , ..., N b } . Note that X i is the representation for the claim and that the document or application-level handcrafted features will be augmented to each claim. The concatenated Z i will pass through the linear and the softmax layer. Mathematically, we restrict the search space upon N s,claim , regularizing predictions to be decreasing in it. The optimizer will potentially be able to find alternative paths to avoid undesirable local minima. Let Z i denote all other inputs except N s,claim . We manipulate the input such that inconsistency with the monotonicity in N s,claim is represented. The novelty scores need to be manipulated and multiplied by For a positive constant C (0 < C < 1) let N  s,claim = CN s,claim , let Z  i = Z i  N  s,claim . Applying such a manual constraint on the input novelty representation completes such a monotonically decreasing relationship between input N i,k and output Given log-likelihood with respect to Z i , F ( Z i ) = y i log  y i ( Z i ) + (1  y i ) log(1   y i ( Z i )) , we shall constrain F ( Z i ) < F ( Z  i ) . To implement it, we shall impose a hinge loss penalty whenever F ( Z i ) > F ( Z  i ) and return 0 when otherwise. Therefore, the final objective function becomes: O = L +  (cid:88) i max (cid:8) 0 , F ( Z i ) > F ( Z  i ) (cid:9) , where  determines the regularization strength. In the experiments, we seek to answer a number of questions. To begin with, we evaluate how hand-354 Table 3: Evaluation Results of AISeer, Compared Methods, and Ablations. crafted features can help the deep language model, i.e. BERT, adapt to a complex domain that differs from typical NLP use cases. In particular we focus on the novelty feature critical to patent approvals. We are interested in the extent to which a standard BERT application can learn from highly noisy labels and inconsistent data and find out the novelty pattern, i.e. the significance of novelty in determining patent approval outcomes. In addition, we study if the combination of BERT and handcrafted features serves to be adequate in capturing the novelty pattern. We also examine if monotonic regularization boosts the learning process to further overcome the intrinsic data inconsistencies. We mainly compare AISeer with two baseline models, BERT and LSTM , as they are the best common document classifiers from our benchmark results. For ablation study purpose, we also compare with Log. Reg. Feat. Only , a logistics regression model with handcrafted features only, and AISeer w/o Regu. , which is a BERT model integrated with our handcrafted features but not regularized by our monotonic constraints. AISeer is trained with the same set of hyper-parameters as BERT: maximum token length as 128, fine-tuning for 5 epochs; batch size as 256; AdamW with learning rate being 5e-5 as the optimizer. The monotonic regularization parameter C is 12 and  is 5e-4. The models are trained on a single Nvidia Quadro RTX 8000 GPU. The baseline BERT model gives decent AUC (ROC) and macro F1. The full-fledged AISeer, combining handcrafted novelty feature along with other computed ones and motonic regularization, helps with both the metric dimensions: AISeer boosts AUROC by around 2.5% percent and macro F1 by around 1% compared to the best common document classifiers. Figure 5 in Appendix F shows the AUROC improvement originates consistently from the entire spectrum of prediction scores. Aforementioned in the introduction, when simply classifying the topics of approved patents, state-of-the-art document classifiers can only achieve an accuracy of about 69.3% (only 2.2% over RoBerta) (Zaheer et al., 2020). Given the difficulty level and subjective nature of the patent approval task, the performance improvement is nontrivial and practically impactful. Standard BERT fine-tuning realizes an AUROC increase of only 11.79% over completely random or naive predictions, which also exemplifies the problem's difficulty. Our approach achieves an additional performance of 2.35%, which is equivalent to 20% of the total benefits of the original BERT model. Given that BERT remains one of the most effective models in varieties of NLP tasks, and especially that handcrafted features have relatively low dimensionality compared to BERT, we believe that the performance gain equivalent to 20% of the performance gain of BERT is substantial for this completely new application domain. The lower half of Table 3 shows the result of Log. Reg. Feat. Only, indicating the necessity of a language model. Neither a language model only nor handcrafted features only can yield satisfactory performance. Comparing AISeer w/o Regu. result, also in the lower half of Table 3, and the standard BERT and LSTM results, it is shown that handcrafted features improve on best common document classifiers by about 2%. We believe that the handcrafted features combined, in particular, the novelty feature, helps in resolving label contradictions and data inconsistency. We believe the novelty feature should be only considered under contexts and will not perform well on its own. First, novelty can be a subjective concept and may vary according to different types of claims, openness of claims, the department (cate-gory), etc. Second, novelty as practically measured by dis-similarity, can be easily achieved by poorly written random content, thus structural or overall similarity is also important. However, the observations indicate that there are potential conflicts between the novelty feature and other handcrafted features. While the latter helps with prediction performance on their own and provide contexts for the 355 novelty feature thus imperative, it will also attenuate the effects of the regularized novelty feature. We leave this challenge for future work. One may also ask whether the handcrafted features have contributed significantly given the moderate improvement. Granted, application full texts may also contain signals for the patent class and applicant efforts that may partially reflect handcrafted features and the document classifier such as BERT may pick them up. To shed light on how AISeer learns from handcrafted features, we run linear regressions for the model prediction scores on handcrafted features for interpretable insights and present statistical results, as shown in Table 4. In the table, even prediction scores under BERT are significant in all handcrafted features, showing that BERT does learn knowledge overlapping with the handcrafted features to some extent.Overall, low R 2 's indicate that knowledge from the deep neural model and knowledge from handcrafted features are quite distinct. Comparing BERT and AISeer w/o. Regu., the significant R 2 increase from 0.085 to 0.125 shows that AISeer captures handcrafted features much more effectively than BERT. The prediction scores of AISeer w/o. Regu. have an additional about 4% increase in explanability by the handcrafted features. In Table 3, comparing AISeer and AISeer w/o Regu., the median run result indicates that adding monotonic regularization produces a small magnitude of improvement. Table 4 also provides insights with respect to the monotonic regularization. According to Table 4, our claim-level novelty feature N s,claim has the most significant impact, i.e. the coefficients are much larger in every column. The use of monotonic regularization alone boosts the R 2 significantly, indicating that the approach also helps the model learn from handcrafted features overall. About 19% of the knowledge of AISeer corresponds to handcrafted features, a 10% increase over BERT. Also, AISeer corrects incorrect coefficient signs from BERT. Intuitively, the approval chance shall increase with the number of applicant cited references. However, BERT prediction scores are negatively correlated with it statistically significantly. Under AISeer, this direction is reversed to match intuitions. We also evaluate the Spearman correlation coefficients of the probability prediction scores produced by the models with the claim-level novelty feature Pearson correlations with the document-level similarity score. Spearman correlations measure the strength and direction of monotonic association between two variables. According to Table 5, first we can confirm that applying monotonic regularization significantly pushes the prediction scores to be more monotonically decreasing in the core novelty feature  the Spearman correlation shifts from -0.0230 to -0.103. However, compared to the BERT, the regularization effect is less prominent. Observe that adding handcrafted features will actually steer the monotonicity into the opposite direction. Our regularized AISeer model manages to both benefit from the novelty feature and incorporate knowledge from other handcrafted features. While Table 4 illustrates the significant effects of applying the monotonic regularization on the prediction scores, we acknowledge that the observed main performance improvement is not very significant. In fact, although monotonic regularization raise the performance on average, it does not always yield desirable improvements depending on the random seed and the hyperparameter setup. The BERT model may already have a decent learning power to mine the novelty measurement despite the noisy data. We observe that in Table 4, the BERT prediction scores, learned from texts only, are significant in the novelty feature and are in the correct direction. The relatively small performance gain of using monotonic regularization may also be attributed to the compromised precision of the novelty feature due to the use of the ElasticSearch pre-filter for the sake of computational costs. To our knowledge, our work is the first in predicting patent approvals according to the examination procedures at the government patent office. Few extant researches attempt to predict decisions in office. Winer (2017) studies PTAB (Patent Trial and Appeal Board) hearing decisions at USPTO. Other related work addresses patent quality in a general and broad sense (Wu et al., 2016). More broadly in the IP/patent domain, although AI/ML applications have been often advocated (Ebrahim, 2018), studied (for a review see (Aristodemou and 356 Table 4: Regression Analysis of Prediction Scores on Handcrafted Features. Tietze, 2018)) or implemented in practice (Lu et al., 2017), most work focus on determining patent content classes to save manpower or concern only with patent grants rather than applications (Verberne et al., 2010; D'hondt et al., 2013; Hu et al., 2016; Balsmeier et al., 2018; Lee and Hsiang, 2019). Recent studies (Hsu et al., 2020) emerge aiming at predicting patent transfers and the economic value. Other streams of related work include those exploring patent similarity. Our approach of constructing the novelty feature with a state-of-the-art neural bi-encoder (Reimers and Gurevych, 2019) is significantly more advanced than relatively rudimentary approaches in the extant literature, such as text matching and frequency-based methods (Younge and Kuhn, 2016; Arts et al., 2018; Shah-mirzadi et al., 2019). Studies on semantic analysis and representation of technology (Kim et al., 2016; Strumsky and Lobo, 2015) based on patent data are also related. In this paper, we tackle the challenging problem of predicting patent approval decisions as per 35 U.S. Code  102 , namely the novelty-based decisions. We have prepared a large-scale benchmark dataset by consolidating different data sources from USPTO. From the evaluations of the popular document classifiers, BERT and LSTM are arguably the most effective ones. We identify the time-dependent challenge of the novelty judgement, and therefore propose AISeer, a novel framework going beyond the traditional document classifiers. Specifically, we construct a claim-level core novelty feature along with several other handcrafted features and apply them on top of the pre-trained BERT model. We further propose to add the monotonic regularization on the core novelty feature to resolve the potential label conflicts caused by the mechanism of the patent examination process. Experimental results have verified the superiority of AISeer and also the effectiveness of introducing novelty features and monotonic regularization. We believe that our work is beneficial to various parties, including patent applicants, attorneys, examiners and regulators. While the advantages of our regularization methodology are significant, there is still room for potential metric improvements, thus further developing the work will yield opportunities for promising future research and greater contributions to the communities. In future, it is important to extend the scope from claims to the other sections in the patent applications. Relationships among components and entities described in claims and relations among claims are also critical to investigate. We want to thank the anonymous reviewers for their insightful comments. The research was sponsored in part by National Science Foundation Convergence Accelerator under award OIA-2040727 as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, conclusions, or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright annotation hereon. This paper focuses on the patent approval prediction. The data is publicly available from USPTO and we collected it to form a large-scale dataset. Our architecture is built upon open-source models and all the datasets are available online. Therefore, we do not anticipate any major ethical concerns.
abstain abstain abstain method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain objective abstain abstain objective method method result result abstain objective abstain abstain abstain objective objective method abstain other method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain result abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain objective other other other other other other other method other method abstain abstain objective method objective abstain method objective abstain abstain other other other other method method method abstain