LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

Using Mistral Openorca to create a knowledge graph from a text document (towardsdatascience.com)

submitted 9 months ago by WaterdanceAC@alien.top to c/localllama@poweruser.forum

23 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] Inkbot_dev@alien.top 1 points 9 months ago (1 children)

Alright, here are two full logs, Inkbot generated everything below the <#bot#> response.

Simple prompt: https://gist.github.com/Tostino/c3541f3a01d420e771f66c62014e6a24

Complex prompt: https://gist.github.com/Tostino/44bbc6a6321df5df23ba5b400a01e37d

So in this case, the complex prompt did perform better.

[–] andrewlapp@alien.top 1 points 9 months ago (1 children)

Great work, this is impressive, especially for a 13B model!

[–] Inkbot_dev@alien.top 1 points 9 months ago

It was not an insignificant amount of work to get it working as well as it is tbh.

For example, one of the tweaks I did that had the most impact...you'll notice the node IDs are all greek letters. They were originally contextually-relevant IDs, like the name of the entity in the graph.

```

- id: Eta

event: Construction of the Eiffel Tower

date: 1889

```

would have been

```

- id: eiffel

event: Construction of the Eiffel Tower

date: 1889

```

But that lead to the model relying on context clues from that piece of text, rather than being forced to actually look up the data in the knowledge graph during training. So switching that out to use a symbol approach worked much better for relying on data in the graph, rather than model built-in knowledge.

I was planning on testing that out on my own, but then I ran into this paper: https://arxiv.org/abs/2305.08298, which made me pull the trigger and convert my whole dataset and creation process to support symbolic identifiers.