this post was submitted on 29 Nov 2023
1 points (100.0% liked)
LocalLLaMA
3 readers
1 users here now
Community to discuss about Llama, the family of large language models created by Meta AI.
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
If you are interested in knowledge graphs, I did a whole bunch of research and work on fine-tuning Inkbot to create knowledge graphs. The structure returned is proper YAML, and I got much better results with my fine-tune than using GPT4.
https://huggingface.co/Tostino/Inkbot-13B-8k-0.2
Here is an example knowledge graph generated from an article about the Ukraine conflict: https://gist.github.com/Tostino/f6f19e88e39176452c1a765cb7c2caff
Great work! Would you mind sharing the datasets you used and/or how you augmented the data for training?
Alright, here are two full logs, Inkbot generated everything below the <#bot#> response.
Simple prompt: https://gist.github.com/Tostino/c3541f3a01d420e771f66c62014e6a24
Complex prompt: https://gist.github.com/Tostino/44bbc6a6321df5df23ba5b400a01e37d
So in this case, the complex prompt did perform better.
Great work, this is impressive, especially for a 13B model!
It was not an insignificant amount of work to get it working as well as it is tbh.
For example, one of the tweaks I did that had the most impact...you'll notice the node IDs are all greek letters. They were originally contextually-relevant IDs, like the name of the entity in the graph.
```
- id: Eta
event: Construction of the Eiffel Tower
date: 1889
```
would have been
```
- id: eiffel
event: Construction of the Eiffel Tower
date: 1889
```
But that lead to the model relying on context clues from that piece of text, rather than being forced to actually look up the data in the knowledge graph during training. So switching that out to use a symbol approach worked much better for relying on data in the graph, rather than model built-in knowledge.
I was planning on testing that out on my own, but then I ran into this paper: https://arxiv.org/abs/2305.08298, which made me pull the trigger and convert my whole dataset and creation process to support symbolic identifiers.