Papers
arxiv:2410.09870

ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains

Published on Oct 13
· Submitted by Minbyul on Oct 17
Authors:
,
,
,

Abstract

Large language models (LLMs) have significantly impacted many aspects of our lives. However, assessing and ensuring their chronological knowledge remains challenging. Existing approaches fall short in addressing the accumulative nature of knowledge, often relying on a single time stamp. To overcome this, we introduce ChroKnowBench, a benchmark dataset designed to evaluate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, temporal state. Our benchmark distinguishes between knowledge that evolves (e.g., scientific discoveries, amended laws) and knowledge that remain constant (e.g., mathematical truths, commonsense facts). Building on this benchmark, we present ChroKnowledge (Chronological Categorization of Knowledge), a novel sampling-based framework for evaluating and updating LLMs' non-parametric chronological knowledge. Our evaluation shows: (1) The ability of eliciting temporal knowledge varies depending on the data format that model was trained on. (2) LLMs partially recall knowledge or show a cut-off at temporal boundaries rather than recalling all aspects of knowledge correctly. Thus, we apply our ChroKnowPrompt, an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans. We observe that our framework successfully updates the overall knowledge across the entire timeline in both the biomedical domain (+11.9%) and the general domain (+2.8%), demonstrating its effectiveness in refining temporal knowledge. This non-parametric approach also enables knowledge updates not only in open-source models but also in proprietary LLMs, ensuring comprehensive applicability across model types. We perform a comprehensive analysis based on temporal characteristics of ChroKnowPrompt and validate the potential of various models to elicit intrinsic temporal knowledge through our method.

Community

Paper author Paper submitter

ChroKnowledge is a novel sampling-based framework for evaluating and updating LLM's non-parametric chronological knowledge, in five different domains: general, biomedical, legal, commonsense and mathematics.

(1) Background: As current methods for evaluating and updating LLMs tends to focus on specific single time stamp, it is neglected to analyze the accumulative and evolving nature of knowledge, though it is important especially in scientific work and legal regulations.
(2) Motivation: To address this, we deal with the temporal and accumulative nature of knowledge in various domains, and finally, suggest a ChroKnowBench, evaluating knowledge across three key aspects: time dependency, domain diversity, and temporal state.
(3) Method: We devise a ChroKnowPrompt, offering a non-parametric method to directly update LLMs by traversing time spans. This yields substantial improvements without retraining effort.

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Paper author Paper submitter

https://p-yi.github.io/ChroKnowledge/

More details could be found in the above url!

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2410.09870 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2410.09870 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.