Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state ofknowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the over parameterization issue, and approaches to compression. We then outline directions for future research.

Large language models present privacy risks through memorization of training data though little attention goes to fine-tuning risks or comparing methods. We study memorization of the methods via attacks to show they differ.

LegalAI focuses on applying AI technology to legal tasks. In recent years, LegalAI has drawn increasing attention from AI researchers and legal professionals, as LegalAI is beneficial to the legal system for liberating legal professionals from paperwork. Legal professionals focus on rule-based and symbolic methods, while NLP researchers focus on data-driven and embedding methods. We introduce LegalAI's history, current state, and future directions. We illustrate tasks from legal and NLP perspectives and show applications. We analyze existing works' pros and cons to explore future directions. implementation: https://github.com/thunlp/CLAIM.

Natural Language Processing(NLP)is widely used to support the automation of different Requirements Engineering(RE)tasks.Most of the proposed approaches start with various NLP stepsthat analyze requirements statements, extract their linguistic information, and convert them to easy-to-process representations,such as lists of features or embedding-based vector representations.These NLP-based representations usually used at a later stage as inputs for machine learning techniques or rule-based methods.Thus, requirements representations play a major role determining the accuracy of different approaches. After compiling an initial pool of 2,227 papers, and applying a set of inclusion/exclusion criteria, we obtained a final pool containing 104 relevant papers. Our survey shows that the research direction has changed from the use of lexical and syntactic features to the use of advanced embedding techniques, especially in the last two years. Using embeddingrepresentations has proved its effectiveness in most RE tasks(such as requirement analysis, extracting requirements from reviews and forums, and semantic-level quality tasks).However,representations that are based on lexical and syntactic featuresare still more appropriate for other RE tasks(such as modeling and syntax-level quality tasks) since they provide the required information for the rules and regular expressions used when handling these tasks.In addition, we identify four gaps in the existing literature, why they matter, and how future research can begin to address them.

To explain NLP models a popular approach is to use importance measures, such as attention, which inform input tokens are important for making a prediction. However, an open question is how well these explanations accurately reflect a model's logic, a property called faithfulness. To answer this question, we propose Recursive ROAR, a new faithfulness metric.This works by recursively masking allegedly important tokens and then retraining the model. The result is a performance curve given a masking-ratio. Furthermore, we propose a summarizing metric using relative area-between-curves (RACU), which allows for easy comparison across papers, models, and tasks. We evaluate 4 different importance measures on 8 different datasets, using both LSTM-attention models and RoBERTa models. We find that the faithfulness of importance measures is both model-dependent and task-dependent. This conclusion contradicts previous evaluations in both computer vision and faithfulness of attention literature.

We propose a backdoor scanning technique to detect backdoors in NLP models. It transforms a subject model to an equivalent but differentiable form and uses optimization to invert a distribution of words denoting their likelihood in the trigger. It then uses word analysis to see if the model is discriminative for trigger words. Evaluation on 3839 NLP models shows it is highly effective, achieving over 0.9 detection accuracy and outperforming state-of-the-art scanners.

There are many resources for English NLP but few for Greek. A survey of Greek NLP tools and methods could benefit researchers.

We propose using page layout features to improve text extraction from legal documents. Splitting pages by type improves accuracy, achieving 0.98 for single-column and 0.96 for double-column layouts.

NLMs challenges are explained through machine rationales, less utilizing rationales to improve NLM behavior.Explanation regularization aims to improve NLM behavior by aligning machine and human rationales.Prior works evaluate ER models on in-distribution generalization, little on out-of-distribution.Little is understood how ER model performance is affected by the choice of ER criteria or by the number/choice of training instances with human rationales. We propose ER-TEST, evaluating ER models’ out-of-distribution generalization: (1) unseen datasets, (2) contrast set tests, (3) functional tests.Using ER-TEST, we study: (A) Effective ER criteria for given out-of-distribution setting? (B) How is ER affected by number/choice of instances with human rationales?