Build Links Between Problems and Solutions in the Patent

Abstract

Inventive Design Method mostly relies on the presence of exploitable knowledge. It has been elaborated to formalize some aspects of TRIZ being expert-dependent. Patents are appropriate candidates since they contain problems and their corresponding partial solutions. When associated with patents of different fields, problems and partial solutions constitute a potential inventive solution scheme for a target problem. Nevertheless, our study found that links between these two major components are worth studying further. We postulate that problem-solution effectively matching contains a hidden value to automate the solution retrieval and uncover inventive details in patents in order to facilitate R&D activities. In this paper, we assimilate this challenge to the field of the Question Answering system instead of the traditional syntactic analysis approaches and proposed a model called IDM-Matching. Technically, a state-of-the-art neural network model named XLNet in the Natural Language Processing field is combined into our IDM-Matching to capture the corresponding partial solution for the given query that we masked using the related problem. Then we construct links between these problems and solutions. The final experimental results on the real-world U.S. patent dataset illustrates our model’s ability to effectively match IDM-related knowledge with each other. A detailed case study is demonstrated to prove the usage and latent perspective of our proposal in the TRIZ field.

1 Introduction

The foundations of TRIZ [1] taught us that associating a problem with a piece of information from a domain distant from the domain where the problem occurs was an appropriate inventive scheme. In this scenario, the two fundamental elements “problem” and “solution” appear to be indispensable for the implementation of an approach that would aim at automating their association on the assumption that they must respectively come from different domains. These three elements are all three present in the corpus of patents. The problems are explained under the heading “description” and the solutions under the heading “claim” [6]. Each patent is also classified in a category called IPC. With this database of more than 130 million texts, we are therefore in the presence of the elements essential to the automation of a TRIZ-based process. In our previous research, we have developed a Patextractor API [18], capable of isolating from unstructured texts in the patent corpus, Problems, and Partial Solutions when they respect the formalism of IDM-TRIZ and its problem graph. Although offering precision scores that needed to be improved, these sentences were isolated from the perspective of constructing a problem graph, a state of the art of an initial situation of an inventive problem. Our objective in this new research is to provide a base for the hypothesis that two semantically close problems from two distant domains, logically associated with partial solutions, may present inventive opportunities when the problem of one domain is associated with the solution of the other [10].

In order to ensure the quality of problem-solution associations from different domains, we proposed our IDM-Matching model in this paper. It combines a state-of-the-art natural language processing model called XLNet to capture the partial solutions. The problem-solution links are then proposed to the user through an online software interface. Specially, we treat this task as a question answering system [15] and convert each problem into a query to make full use of XLNet neural networks and avoid the drawbacks of lexico-syntactic pattern matching methods. To summarize, the main contribution of this paper is proposing a novel method for automatic matching problems and corresponding solutions in patent documents. The final experiments on the open-source SQuAD dataset [14] and real-world patent dataset illustrate our model’s performance. Though, the patent base with which we perform our initial tests is for the moment limited to USPTO patents (thus only American) but its scope nevertheless allows us to verify the validity of our hypotheses. Especially, a detailed case study eventually proves our model’s usage and latent perspective on reality.

This article is composed of 5 parts after the introduction, the second part is dedicated to the state of the art on the subject and cites the main contributions that have been useful to our research. It is in the third part that our methodology is presented, as well as the framework of our experiments. The forth part is dedicated to the case study; it constitutes a part both of verification of our hypotheses but also with the didactic aim of presenting the logic of our approach by example. It is followed by discussions, conclusions, acknowledgments, and references.

2 Related Work

Inventive Design Method (IDM) is based on the Theory of Inventive Problem Solving - TRIZ [21]. It represents an extension of TRIZ and is perceived as more guided. Different from other ontologies, IDM ontology is generic and applicable in all fields [3]. In addition, Cavallucci et al. [6] proposed the main concepts of IDM that are problems, partial solutions, and contradictions including element parameters and values. In patents, problems normally describe unsatisfactory features of existing methods or situations. Partial solutions provide improvements or changes to the defined problems. Each problem may cause one or more contradictions the patent solves. Besides, partial solutions must be the simplest possible. The correct pairwise between problems and corresponding solutions has great value for engineers to capture the hidden inventive details in patents.

Recent years, most of researchers focus on making use of images [9, 13], tabulations [13] or novel proposals [19] in patent documents to facilitate TRIZ or R&D activities. Nevertheless, only a few of research works notice the hidden value of relation between problems and corresponding solutions in IDM-related knowledge. Among them, syntactic analysis is used as the main research method. The syntactical structure of subject (noun phrase), action (verb phrase), and object (noun phrase) is leveraged to explicitly represent relationships between the components of a patent by several researchers [4, 5, 16, 18].

Especially, Souili et al. [17] leverage generic linguistic markers to build patent lexicon database for extracting IDM-related knowledge and building links between problems and partial solutions. Our study also leverages this work to extract problems in patents but the links it provides always are weak and not precise. Besides, our study found that, in patent documents, inventors always provide several inventive details for solving different problems in order to construct an entire inventive plan or object. Furthermore, these inventive details are contained in different corresponding solutions of IDM-related knowledge. It makes us cannot ignore the latent value of building precise links between problems and corresponding solutions in patent documents. Therefore, in this work, we proposed a model called IDM-Matching that combined state-of-the-art natural language processing model called XLNet [20] in order to match problems to corresponding solutions in patents. XLNet integrates the segment recurrence mechanism and relative encoding scheme of Transformer-XL [7] into pretraining, which empirically improves the performance especially for tasks involving a longer text sequence like patent text. In order to leverage its specialty, we especially convert our task as a question answering system task [2]. The answers to the queries that are packed by problems become the corresponding solutions. The links between these IDM-knowledge are eventually built by IDM-Matching.

3 Methodology and Experiments

In this section, we introduce our IDM-Matching model for building links between problems and partial solutions. Our work aims to match the corresponding solution with the target problem in each patent document in order to help engineers conveniently find out as many inventive sub-solutions as the inventor provided in the patent. As shown in Fig. 1, IDM-Matching first extracts problems from patent documents by Patentextractor [18] to prepare a list of related problems. Then, we assume these problems as several queries and convert them into related questions. These packed queries are sent to the Question Answering system–pretrained with XLNet neural networks [20] to compute an answer list. After a filtering mechanism, the corresponding solutions are extracted so that we can build links between the corresponding solution and the target problem.

Formally, the extracted IDM-related knowledge set $k_i = \{P_i, PS_i\}$ is from i-th patent document where $P_i$ and $PS_i$ are problems and partial solutions respectively in the i-th patent document. Given the j-th problem $P_i^j = \{(X_i^{j_1},z_1); (X_i^{j_2},z_2); ... \, (X_i^{j\left| x_i \right| },\left| z_t\right| )\}$ in the i-th patent document where $X_i^{j\left| x_i \right| }$ is the $\left| x_i \right|$-th word that is located in the $\left| z_t\right|$-th position.

Fig. 1.

An overview of IDM-Matching

3.1 Pack Problem

We aim to build links between problems P and partial solutions PS via Question Answering system.

Indeed, we first leverage Patentextractor to capture IDM-related knowledge including problems and partial solutions from patent documents. For i-th patent document, we build a problem database without partial solutions since it fails extract precise corresponding solutions for the given problems. For each single problem $P_i^j$ in i-th patent document, we then pack it into a query sentence with a fixed format, for instance “What is the solution for the problem that___?”. With this type of converting, each problem can be seen as a query sentence in the question answering system. Besides, we do not need to make a custom design for several different types of problems in the IDM-related knowledge in patent documents. More importantly, the head of “What is the solution for the problem that” is apparent to let model learn that finding solutions are the target and given problem is the sentence after “problem that”. This type of packing is helpful, especially when the problem sentences are not obvious to be seen as problems or lacking some unique negative words. Besides, “solution” in the design is useful for the model to locate the corresponding solution sentence when the related context information is containing the key word “solution”.

3.2 Define Context Information

Context information plays a significant role in the question answering system. The model captures related answers for the given query. Longer context information normally contains more noisy information. Redundant context information also leads to bigger computational cost.

Through the study over patent documents, we noticed that, due to the natural structure of the patent document, corresponding partial solutions for the target problems always appear near to the paragraph containing the target problem. Furthermore, some obvious partial solutions are located beside the target problem the patent described. Thus, for this situation, we can easily find out corresponding solutions to the target problem in the same paragraph.

Due to these reasons, in this task, we define context information for our IDM-Matching as 3 paragraphs of context, including the paragraph containing the target problem as well as an upper and lower single paragraph.

3.3 XLNet Model

A state-of-the-art natural language model called XLNet [20] is combined into our IDM-Matching model.

As a pre-trained permutation language modeling, XLNet solves the drawback that the traditional autoregressive language modeling like GPT [12], ELMO [11] etc. cannot learn the forward context and backward context information at the same time to predict the target word. Moreover, XLNet is able to solve the drawback that the artificial symbols like [MASK] used by BERT [8] during pretraining are absent from real data at netuning time, resulting in a pretrain-netune discrepancy.

According to XLNet, it captures bidirectional context via the permutation based dependency rule. Indeed, take a problem sentence in patent as an example. For the given length ($T = 3$) problem $P_i^j = \{(X_i^{j_1},z_1); (X_i^{j_2},z_2); (X_i^{j_3},z_3)$ in the i-th patent document, there are T! different orders to perform a valid autoregressive factorization for the tokens that the problem sentence $P_i^j$ contained:

$$\begin{aligned} p(\mathbf{X} ) = p(x_i^{j_1})p(x_i^{j_2}|x_i^{j_1})p(x_i^{j_3}|x_i^{j_1}x_i^{j_2}) \Rightarrow 1 \rightarrow 2 \rightarrow 3 \end{aligned}$$

(1)

$$\begin{aligned} p(\mathbf{X} ) = p(x_i^{j_1})p(x_i^{j_2}|x_i^{j_1}x_i^{j_3})p(x_i^{j_3}|x_i^{j_1}) \Rightarrow 1 \rightarrow 3 \rightarrow 2 \end{aligned}$$

(2)

$$\begin{aligned} p(\mathbf{X} ) = p(x_i^{j_1}|x_i^{j_2})p(x_i^{j_2})p(x_i^{j_3}|x_i^{j_1}x_i^{j_2}) \Rightarrow 2 \rightarrow 1 \rightarrow 3 \end{aligned}$$

(3)

$$\begin{aligned} p(\mathbf{X} ) = p(x_i^{j_1}|x_i^{j_2}x_i^{j_3})p(x_i^{j_2})p(x_i^{j_3}|x_i^{j_2}) \Rightarrow 2 \rightarrow 3 \rightarrow 1 \end{aligned}$$

(4)

$$\begin{aligned} p(\mathbf{X} ) = p(x_i^{j_1}|x_i^{j_3})p(x_i^{j_2}|x_i^{j_1}x_i^{j_3})p(x_i^{j_3}) \Rightarrow 3 \rightarrow 1 \rightarrow 2 \end{aligned}$$

(5)

where $p(x_i^{j_2}|x_i^{j_1}x_i^{j_3})$ denotes the possibility p of second word $x_i^{j_2}$ with the constraint condition that first word is $x_i^{j_1}$ and third word is $x_i^{j_3}$. This mechanism makes XLNet could capture bidirectional context information at the same time.

For predicting target word $x_i^{j|x_i|}$ with its position $z_t$, the function is as follows:

$$\begin{aligned} p_{\theta }\left( X_{z_{t}}=x_i^{j|x_i|} | \mathbf {x}_{{z}_{<t}}\right) =\frac{\exp \left( e(x_i^{j|x_i|})^{T} g_{\theta }\left( \mathbf {x}_{z_{<t}}, z_{t}\right) \right) }{\sum _{x^{\prime }} \exp \left( e\left( {x_i^{j|x_i|}}^{\prime }\right) ^{T} g_{\theta }\left( \mathbf {x}_{z_{<t}}, z_{t}\right) \right) } \end{aligned}$$

(6)

where $\mathbf {x}_{{z}_{<t}}$ denotes previous words for the target word $x_i^{j|x_i|}$.

Additionally, as an unsupervised language representation learning method, XLNet also profits the pros of Transformer-XL so that it exhibits better performance on language tasks involving long context. We thus combine it into our IDM-matching model to be as a question answering system to capture corresponding solutions.

3.4 Experimental Settings

We detail the experimental settings in this section.

Datasets and Evaluation Metrics: In this work, as a notable benchmark in question answering, we leverage open-source Stanford Question Answering Dataset (SQuAD 2.0)¹ to train and evaluate our model. SQuAD is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage. It contains 100,000 questions with corresponding labelled answers. Furthermore, due to lacking labelled patent datasets, we choose 50 U.S. patent documents that are issued on 03, January 2017 by the United States Patent and Trademark Office (USPTO)² to evaluate IDM-Matching’s performance on our task by manual work. The final EM (Exact Match) and F1 score are used as evaluation metrics.

Parameter and Computer Settings: In this work, we tune our IDM-Matching on SQuAD dataset and use the grid search to determine the optimal parameters. We use Huggingface’s pretrained XLNet-base-cased model³ as our question answering system. It contains 12-layer, 768-hidden, 12-heads, 110M parameters. For training the model, we select the epochs among {3, 4, 5, 6}, batch size among {5, 10, 15}, learning rate among {$2e^{-5}, \mathbf{3} e^{-5}, 4e^{-5}$}, and others are as default. The optimal parameters are highlighted with bold faces. Besides, the model is trained on 1 T P100 GPU for around 15 h.

Overall Results: We first evaluate our model on the evaluation dataset and achieves 70.99% EM and 72% F1. On the 50 real-world U.S. patent datasets, IDM-Matching achieves average 72.43% accuracy.

4 Case Study

In this section, we demonstrate a real-world case study on an U.S. patent document in order to show our model’s practical performance.

US8847930B2: “Electrically conductive touch pen” is an U.S. patent from physics field. As shown in Fig. 2, it presents a multi-function writing devices that can physically mark on traditional writing surfaces and can also digitally mark on, or be used as other input means in association with, computerized digital displays. This invention has an internal ink cartridge deployable through a hole in the stylus tip. The stylus tip extends from a sleeve that is formed of a conductive elastomeric material. The sleeve extends up a rigid shaft of the device such that it contacts a sufficient ground. The stylus tip is coated with a protective material that adjusts the coefficient of friction and prevents carbon deposits on the touch screen. A sufficient contact patch is achieved to simulate a human finger so as to overcome false positives from common touch screen logic.

Fig. 2.

An overview of the invention

Patentextractor extracted, as shown in Fig. 3, several problems (red circles) from the patent. We picked up 7 correct problems (red circles with yellow edges) as inputs for the IDM-Matching and convert them into 7 queries. The questions, answers that our model extracted, correct answers, and the related context information are as follows:

Fig. 3.

The extracted problems (Color figure online)

• 1. [Question]: What is the solution for the problem that this would hamper a user’s ability to operate the touch pen 10 with gloves?

  —Answer List—

  • 1) The inner molding 29 is replaced by a former 39 that is ideally metallic. This alternative embodiment is designed to address the aforementioned problems attendant to a user wearing gloves, having very dry skin, or situations in which the user does not make good conductive contact with the touch pen 10. In such cases the conductive cover 28 needs to be in good electrical contact with a volume of metal V (m3) of conductivity.

  • 2) conductive cover 28 needs to be in good electrical contact with a volume of metal V (m3) of conductivity a (Siemens per meter S/m).

• 2. [Question]: What is the solution for the problem that too much flexibility can also lead to false positives?

  —Answer List—

  • 1) The larger the air cavity 32, the more flexible the stylus tip 22 will become. However, too much flexibility can also lead to false positives. As shown, the former 39 comprises an extension 41 of various sizes.

  • 2) The larger the air cavity 32, the more flexible the stylus tip 22 will become. However, too much flexibility can also lead to false positives. As shown, the former 39 comprises an extension 41 of various sizes. The size of this extension directly controls the size of the air cavity 32. In some embodiments, this extension may be a controllable feature of the touch pen 10.

• 3. [Question]: What is the solution for the problem that one disadvantage to stylus use is that it necessitates carrying an additional personal item?

  —Answer List—

  • 1) combination pen and stylus.

  • 2) combination pen and stylus. A touch screen.

  • 3) the combination pen and stylus. A touch screen is, generally speaking, a combination touchpad and computer display.

  • 4) pen and stylus. A touch screen is, generally speaking, a combination touchpad.

  • 5) A touch screen is, generally speaking, a combination touchpad.

  • 6) touch screen is, generally speaking, a combination touchpad and computer display that can detect the presence and location of a touch within the display area. Context: One solution to this problem is the combination pen and stylus.

• 4. [Question]: What is the solution for the problem that a problem with rubber containing carbon sufficient for conductivity is that it may leave black marks on substrates to which it comes into contact?

  —Answer List—

  • 1) coating the rubber, or selectively the rubber tip, with a very thin layer of Parylene.

  • 2) selectively the rubber tip, with a very thin layer of Parylene. This conformal coating.

  • 3) very thin layer.

    Context:

    These problems can be solved by coating the rubber, or selectively the rubber tip, with a very thin layer of Parylene.

• 5. [Question]: What is the solution for the problem that the problem is exacerbated if the user of the stylus is wearing gloves or has extremely dry skin?

  —Answer List—

  • 1) films made from graphene (carbon nanotubes), or other suitable material. Conductive materials that touch or are in very close proximity to this type of touch screen alter the electrostatic field of the screen, thereby creating a registerable change in capacitance. Context: Traditional plastic or polymer-based styli are not effective in marking on capacitive touch screens because they are not sufficiently conductive. The problem is exacerbated if the user of the stylus is wearing gloves or has extremely dry skin.

    Correct Answer:

    Capacitive touch screens generally comprise a flat insulative transparent sheet such as glass having an inside portion coated with a transparent conductor such as indium tin oxide (ITO), films made from graphene (carbon nanotubes), or other suitable material.

• 6. [Question]: What is the solution for the problem that this is common in colder environments, where people may often need to mark on handheld devices while outside?

  —Answer List—

  • 1) Capacitive touch screens are quickly replacing resistive touch screens.

  • 2) sink or source of electrons.

  • 3) films made from graphene (carbon nanotubes), or other suitable material. Conductive materials that touch or are in very close proximity to this type of touch screen alter the electrostatic field of the screen, thereby creating a registerable change in capacitance.

  • 4) sink or source of electrons, sometimes called a “ground.

    Correct Answer:

    One solution that enables a stylus to be used with a capacitive touch screen is the use of conductive rubber or a similar conductive elastomeric material.

    Context:

    One solution that enables a stylus to be used with a capacitive touch screen is the use of conductive rubber or a similar conductive elastomeric material. Conductive rubber is a rarer and more expensive form of rubber that contains suspended graphite carbon, carbon nanotubes, nickel or silver particles.

• 7. [Question]: What is the solution for the problem that other materials providing better conductivity could be used, such as aluminum or other metals, they would likely scratch or otherwise damage the touch screen?

  —Answer List—

  • 1) Conductive materials that touch or are in very close proximity to this type of touch screen.

  • 2) films made from graphene (carbon nanotubes), or other suitable material. textbfConductive materials that touch or are in very close proximity.

  • 3) films made from graphene.

  • 4) ions–cations.

  • 5) conductive materials such as biological tissue, these charged carriers could be predominantly ions–cations and/or anions.

    Correct Answer:

    One solution that enables a stylus to be used with a capacitive touch screen is the use of conductive rubber or a similar conductive elastomeric material.

The correct predictive answers have been labelled with bold faces. Instead of some obvious correct answers, we especially mention 2 answers that represent the latent perspective that IDM-Matching in the automatic of IDM-related knowledge. For instance, Question 5 presents the problem that the finger with gloves or dry skin cannot use the stylus well. Actually, the plastic or polymer-based stlyli contributes to this problem but this important information does not appear in the question. IDM-Matching still successfully learned the related information and achieve the correct answer of films made from graphene (carbon nanotubes), or other suitable material. Besides, Question 6 provides a problem that people need to mark on handheld devices while outside in colder environments. This situation leads to insulation from the stylus with the human body. The answer list that our model proposed does not illustrate a precise answer. The correct answer is the use of a conductive rubber or a similar conductive elastomeric material. However, we see the carbon nanotubes appeared in our answer list and conductive rubber exactly contains carbon nanotubes. It means our model still learned some significant information in order to build the link between the problem and the corresponding solution.

In the conclusion of this detailed case study, we note that the final corresponding solutions that IDM-Matching extracted are precise and have significant practical value for automatic matching the target problems. It illustrates that the built links between problems and partial solutions can facilitate engineers to face a large number of patent documents to extract problems the target patent faced and corresponding inventive solutions it provides.

5 Conclusion and Future Work

In this paper, we proposed an IDM-related knowledge association model called IDM-Matching for matching problems and corresponding solutions in patent documents. Our approach can automate the solution retrieval and match with corresponding problems in patents. This work will facilitate engineers to find out inventive details hidden in patent documents in order to speed the R&D activities. More importantly, this work can further improve inventive solutions retrieval for the target problem by associating with different domains’ similar problems [10] in patent documents. By that time, engineers without a broad understanding of the different domains’ knowledge to make full use of inventive knowledge from a wide range of patent documents to facilitate their inventive manufacturing inspirations. Final experimental results on the real-world patent dataset illustrate the performance of our model. In particular, a detailed case study demonstrates the usage of our model in reality and shows its latent perspective on TRIZ field.

In the future, we will explore the following directions:

• (1) Fine-tune our IDM-Matching model in order to further improve its final accuracy in patent datasets.

• (2) Combine it with similarity computation approaches of different domains’ IDM-related knowledge in order to leverage different domains’ inventive solutions to facilitate R&D activities.