So I have an experimental build of Koboldcpp that allows for Dynamic Temperature sampling. Some people tell me that my dynamic temp has become a mainstay of their configurations. Now this poses an obvious question:
Why would you need a Dynamic Temperature?
- Typical attempts to make language models more varied and creative through higher temperature values might not work as you'd expect, due to the fact that higher temperatures disproportionately impact high confidence token generations. This is especially a problem for weaker language models that have less of an innate ability to 'course correct' when a 'bad' token is chosen, or for instances where 'course correction' just isn't good enough (like in programming languages)
- As a consequence, higher temperature values (past ~1.2) are rather difficult to use if you want your language model to output coherent and creative generations. A specific example of how higher temperature can introduce difficulties is in the case of adhering to programming language syntax, as programming languages all have strictly defined rules. This can be an issue if you want an LLM to try a more 'creative' solution to a specific programming problem while still consistently adhering to the rules of the language; a static temperature, therefore, wouldn't be the most effective way to scale the language model's creativity.
For an example, here's how the Dynamic Temperature mapping looks, assuming you use the "HHI" dynamic temp method (which measures how concentrated the model's probabilities are at any given point in time.)
Red = Closer to maximum temperature, Grey = Closer to minimum temperature
The idea is, we turn temperature into a range, where only the highly randomizable tokens get mapped a high temperature, and a non-randomizable token stays near-deterministic.
This sounds great on paper. Except, there's 3 different versions of it that measure different metrics in an attempt to create a better sampler, and not just the HHI version of it. As they say, perfect is the enemy of good... because of this, it's hard to create a 'standard' that I can propose to any of these LLM model hosting backends, and therefore, Dynamic Temperature hasn't been implemented where people can use it beyond my test builds.
This, of course, has made it difficult for me to settle on the 'best method'.
So! To determine the most effective method, I need the community's help in testing and documenting the effects of this experimental sampler on various models. The lack of a standardized approach has hindered widespread implementation, so your feedback on the best method or even just the 'best values' for each method is crucial.
How to Test: I've provided a custom build of Koboldcpp for testing: Link to the experimental build. You can modify the values in the generated .txt file for quick testing. There are also overrides for different dynamic temperature sampling methods.
These overrides include:
- 1.84 Temp
This value overrides to Entropy Sampling, which uses a power function & SamplerTemp.txt file to control the values.
It measures the entropy (uncertainty) of the probability distribution before sampling. This means, if it is highly certain for a certain token, it will use values closer to the minimum temperature. If it is highly uncertain, it will increase the temperature (to avoid repetition / determinism issues in a more natural fashion).
This is probably really difficult for this sub to understand but maybe it makes sense.
It has minTemp (minimum temperature), maxTemp (maximum temperature), and the exponent value (which controls how aggressively it scales the mapping of temperature.)
UNIQUE OBSERVATIONS ABOUT THIS SAMPLER:
- I'm able to turn off all truncation samplers (Min P, Top P, etc) and it still functions coherently within the default range of values (from 0.0 minTemp to 2.0 maxTemp).
- I'm guessing the reason why that happens is because it's really difficult to achieve maximum entropy on a 32,000 token vocabulary model. However, you can turn up the maxTemp to even 5.0 and get some really weird but still pretty coherent results.
- 2.0 Temp
This form of DynaTemp is HHI Sampling, uses a power function & SamplerTemp.txt file to control the values. I misnamed this as Gini sampling before, but it is measuring HHI.
The 'HHI' value it measures is how concentrated the probabilities are. If it is highly concentrated on just one token, then it reduces the temperature to a strong degree. It is more spread out or evenly divided, the temperature is increased towards the maxTemp.
It has minTemp (minimum temperature), maxTemp (maximum temperature), and the exponent value (which controls how aggressively it scales).
UNIQUE OBSERVATIONS ABOUT THIS SAMPLER:
- The measurements of concentration (via the HHI measurement) seem pretty consistent with or without removing 'bad tokens' (e.g Min P, Top P, and othet truncation samplers). This is unlike Entropy which is sensitive to whether or not you have those truncation samplers on or not.
- For reference, here's how the HHI (concentration) measurements look for a prompt that's more deterministic vs. an open ended prompt:
- 1.91 Temp
Greedy Dynamic Temp (aka DynaTemp), the original implementation. This uses uses a sigmoid function & is basing the temperature off the top token. I am not confident that this is useful or interesting compared to HHI and Entropy versions of Dynamic Temp, as it does not measure the entire distribution; this was my first trial run, but you can test it if you want.
What preset settings do you recommend to pair with this?
Here is my two results with the preset set to minimum values in ST, with Kobold sampling order...I think T-2.00 is better, it had less repetition than 1.84.
Use the premise below, and create a story based on it. Aim for at least 5,000 words for your first chapter.
Goddess of Reincarnation
OOC: As a middling goddess, your task is to bring useful souls onto the fantastical world that you look after. You can see humans on Earth, and have assorted means for freeing and collecting their souls. How do you go about this? As a foreign deity, you do not have to adhere to common sense or morality on Earth. /OOC.
Yi-34b Dolphin 2.2, T-1.84
In the vast expanse of the cosmos, there existed a realm of gods and goddesses, each with their own unique powers and dominions. Among them was a goddess known as the ""Goddess of Reincarnation"". She was a middling goddess, tasked with the responsibility of bringing useful souls onto the fantastical world that she looked after.
The goddess had been given a great power, the ability to see humans on Earth and free their souls from their mortal bodies. She could then guide these souls to her world, where they would be reincarnated into new bodies and given a second chance at life.
However, as a foreign deity, the goddess did not have to adhere to common sense or morality on Earth. She was free to do as she pleased, and she often used her powers to cause chaos and destruction among the humans.
One day, the goddess decided that it was time to visit Earth and begin her work. She opened a portal to the human world and stepped through, finding herself in the middle of a bustling city.
The goddess was amazed by the sights and sounds of the city. She had never seen anything like it before, and she was eager to explore and see what kind of souls she could find.
As she walked through the city, the goddess began to notice that there were many people who were suffering and in need of help. She saw homeless people living on the streets, children crying for food and shelter, and people suffering from terrible illnesses.
The goddess knew that she had the power to help these people, and she decided that she would use her powers to bring them to her world and give them a second chance at life.
She began by finding a homeless man who was lying on the ground, cold and hungry. The goddess approached him and offered him a warm meal and a place to stay. The man was grateful and agreed to come with her.
The goddess then opened a portal to her world and led the man through it. Once they arrived, she showed him the beauty and wonders of her realm, and he was amazed by what he saw.
The goddess then took the man to a place where he could be reincarnated into a new body. She helped him choose a new form, and he was soon reborn as a strong and healthy young man.
The goddess was pleased with her work, and she knew that she had made a difference in the life of this man. She decided to continue her mission and find more souls to bring to her world.
As she walked through the city, the goddess began to notice that there were many other people who were in need of her help. She saw children who were being abused and neglected, people who were trapped in terrible jobs and unable to escape, and individuals who were suffering from mental and physical illnesses.
The goddess knew that she had the power to help these people, and she decided that she would use her powers to bring them to her world and give them a second chance at life.
She began by finding a young girl who was being abused by her parents. The goddess approached her and offered her a place of safety and comfort. The girl was grateful and agreed to come with her.
The goddess then opened a portal to her world and led the girl through it. Once they arrived, she showed her the beauty and wonders of her realm, and the girl was amazed by what she saw.
The goddess then took the girl to a place where she could be reincarnated into a new body. She helped her choose a new form, and she was soon reborn as a beautiful and healthy young woman.
The goddess was pleased with her work, and she knew that she had made a difference in the life of this girl. She decided to continue her mission and find more souls to bring to her world.
As she walked through the city, the goddess began to notice that there were many other people who were in need of her help. She saw men and women who were trapped in terrible relationships, individuals who were struggling with addiction, and people who were suffering from poverty and desperation.
Yi-34b Dolphin 2.2 T-2.00
In the vast expanse of the cosmos, there existed a realm of gods and goddesses, each with their own dominion and purpose. Among them was a goddess known as the ""Goddess of Reincarnation"". She was a middling goddess, tasked with the responsibility of bringing useful souls onto the fantastical world that she looked after.
The goddess had been given a great power, the ability to see humans on Earth and free their souls from their mortal bodies. This power allowed her to select the most suitable souls for her world, where they would be reborn into new lives filled with adventure and opportunity.
However, as a foreign deity, the goddess did not have to adhere to common sense or morality on Earth. She was free to use her powers as she saw fit, and she would stop at nothing to ensure that her world was filled with the most exceptional souls.
The goddess had been watching Earth for many years, carefully selecting the most promising souls to bring back to her world. She had seen empires rise and fall, witnessed the birth and death of countless civilizations, and had even intervened in the lives of some of the most powerful humans in history.
But now, the goddess had come to a decision. She had chosen a new world to bring her souls to, a world filled with magic and adventure. This world was unlike any other, and she knew that the souls she collected would thrive in its unique environment.
The goddess began her journey, traveling through the cosmos to the edge of the universe. There, she found a portal that led to her new world. She stepped through the portal, her eyes filled with determination and excitement.
As she emerged on the other side, she found herself in a breathtaking landscape of mountains, forests, and rivers. The air was filled with the sweet scent of flowers, and the sun shone brightly overhead. She knew that this was the perfect place for her souls to thrive.
The goddess began her work, using her powers to bring the souls of the most exceptional humans to her new world. She carefully selected each soul, ensuring that they would be a good fit for her world and that they would contribute to its growth and prosperity.
As the days turned into weeks, the goddess watched as her new world began to take shape. The souls she had brought with her were thriving, and they were quickly becoming the leaders and innovators of their new home.
But the goddess knew that her work was far from over. She had to continue to bring new souls to her world, and she had to ensure that they were the best of the best. She would stop at nothing to ensure that her world was filled with the most exceptional souls, and she would use her powers to shape the lives of these souls in any way she saw fit.
As the years passed, the goddess continued to bring new souls to her world. She watched as her world grew and prospered, and she knew that she had made the right decision in choosing this new world for her souls.
But the goddess also knew that her work was far from done. She would continue to bring new souls to her world, and she would continue to shape their lives in any way she saw fit. She was the Goddess of Reincarnation, and she would ensure that her world was filled with the most exceptional souls that the universe had to offer.