this post was submitted on 29 Nov 2023
1 points (100.0% liked)

LocalLLaMA

1 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] Inkbot_dev@alien.top 1 points 9 months ago

It was not an insignificant amount of work to get it working as well as it is tbh.

For example, one of the tweaks I did that had the most impact...you'll notice the node IDs are all greek letters. They were originally contextually-relevant IDs, like the name of the entity in the graph.

```

- id: Eta

event: Construction of the Eiffel Tower

date: 1889

```

would have been

```

- id: eiffel

event: Construction of the Eiffel Tower

date: 1889

```

But that lead to the model relying on context clues from that piece of text, rather than being forced to actually look up the data in the knowledge graph during training. So switching that out to use a symbol approach worked much better for relying on data in the graph, rather than model built-in knowledge.

I was planning on testing that out on my own, but then I ran into this paper: https://arxiv.org/abs/2305.08298, which made me pull the trigger and convert my whole dataset and creation process to support symbolic identifiers.