Integrating Knowledge Graphs with Neural Networks: Generative Pre‑Training, Differentiable Reasoning, and Fuzzy Logic Query Embedding

The talk introduces two research directions pursued over the past year: (1) using structured knowledge from knowledge graphs to pre‑train neural network models, and (2) applying differentiable neural networks for complex knowledge‑graph reasoning.

It first explains what a knowledge graph is—a multi‑relational graph where nodes represent entities and edges represent relations, typically expressed as triples. Combining knowledge graphs with neural networks allows the latter to learn high‑level semantic representations from structured data.

The speaker describes a generative self‑supervised pre‑training approach for graph neural networks (GNNs), including the Heterogeneous Graph Transformer (HGT) that parameterises meta‑relations and leverages attention mechanisms to propagate information without manual meta‑path design.

Challenges such as label sparsity in downstream QA tasks are addressed by using graph structure as a self‑supervised signal, enabling models to transfer with few annotated examples.

Next, the presentation covers open‑domain QA pre‑training using a Wiki‑graph that fuses Wikidata triples with Wikipedia page hyperlinks. By generating relation‑guided QA pairs from this graph, the model learns to predict relations, perform dense retrieval with hard negatives sampled via random walks, and conduct reading comprehension.

Experimental results show significant gains over baselines and prior pre‑training methods, especially for infrequent relations, reducing the need for large labelled datasets.

The final part introduces differentiable logical reasoning on knowledge graphs via a fuzzy‑logic based query‑embedding model (FuzzQE). It explains how first‑order logic queries are embedded as computation graphs, how fuzzy t‑norm operations (product logic) implement AND, OR, NOT without extra parameters, and how the model satisfies fundamental logical laws.

Empirical evaluation on standard complex FOL query benchmarks demonstrates that FuzzQE outperforms previous query‑embedding methods and matches more computationally intensive approaches, even when trained only on edge information.

In conclusion, the work presents a unified framework that injects structured graph knowledge into neural models through generative pre‑training and fuzzy‑logic reasoning, improving both general QA performance and the ability to handle rare relational queries.