So I have an experimental build of Koboldcpp that allows for Dynamic Temperature sampling. Some people tell me that my dynamic temp has become a mainstay of their configurations. Now this poses an obvious question:
Why would you need a Dynamic Temperature?
- Typical attempts to make language models more varied and creative through higher temperature values might not work as you'd expect, due to the fact that higher temperatures disproportionately impact high confidence token generations. This is especially a problem for weaker language models that have less of an innate ability to 'course correct' when a 'bad' token is chosen, or for instances where 'course correction' just isn't good enough (like in programming languages)
- As a consequence, higher temperature values (past ~1.2) are rather difficult to use if you want your language model to output coherent and creative generations. A specific example of how higher temperature can introduce difficulties is in the case of adhering to programming language syntax, as programming languages all have strictly defined rules. This can be an issue if you want an LLM to try a more 'creative' solution to a specific programming problem while still consistently adhering to the rules of the language; a static temperature, therefore, wouldn't be the most effective way to scale the language model's creativity.
For an example, here's how the Dynamic Temperature mapping looks, assuming you use the "HHI" dynamic temp method (which measures how concentrated the model's probabilities are at any given point in time.)
Red = Closer to maximum temperature, Grey = Closer to minimum temperature
The idea is, we turn temperature into a range, where only the highly randomizable tokens get mapped a high temperature, and a non-randomizable token stays near-deterministic.
This sounds great on paper. Except, there's 3 different versions of it that measure different metrics in an attempt to create a better sampler, and not just the HHI version of it. As they say, perfect is the enemy of good... because of this, it's hard to create a 'standard' that I can propose to any of these LLM model hosting backends, and therefore, Dynamic Temperature hasn't been implemented where people can use it beyond my test builds.
This, of course, has made it difficult for me to settle on the 'best method'.
So! To determine the most effective method, I need the community's help in testing and documenting the effects of this experimental sampler on various models. The lack of a standardized approach has hindered widespread implementation, so your feedback on the best method or even just the 'best values' for each method is crucial.
How to Test: I've provided a custom build of Koboldcpp for testing: Link to the experimental build. You can modify the values in the generated .txt file for quick testing. There are also overrides for different dynamic temperature sampling methods.
These overrides include:
- 1.84 Temp
This value overrides to Entropy Sampling, which uses a power function & SamplerTemp.txt file to control the values.
It measures the entropy (uncertainty) of the probability distribution before sampling. This means, if it is highly certain for a certain token, it will use values closer to the minimum temperature. If it is highly uncertain, it will increase the temperature (to avoid repetition / determinism issues in a more natural fashion).
This is probably really difficult for this sub to understand but maybe it makes sense.
It has minTemp (minimum temperature), maxTemp (maximum temperature), and the exponent value (which controls how aggressively it scales the mapping of temperature.)
UNIQUE OBSERVATIONS ABOUT THIS SAMPLER:
- I'm able to turn off all truncation samplers (Min P, Top P, etc) and it still functions coherently within the default range of values (from 0.0 minTemp to 2.0 maxTemp).
- I'm guessing the reason why that happens is because it's really difficult to achieve maximum entropy on a 32,000 token vocabulary model. However, you can turn up the maxTemp to even 5.0 and get some really weird but still pretty coherent results.
- 2.0 Temp
This form of DynaTemp is HHI Sampling, uses a power function & SamplerTemp.txt file to control the values. I misnamed this as Gini sampling before, but it is measuring HHI.
The 'HHI' value it measures is how concentrated the probabilities are. If it is highly concentrated on just one token, then it reduces the temperature to a strong degree. It is more spread out or evenly divided, the temperature is increased towards the maxTemp.
It has minTemp (minimum temperature), maxTemp (maximum temperature), and the exponent value (which controls how aggressively it scales).
UNIQUE OBSERVATIONS ABOUT THIS SAMPLER:
- The measurements of concentration (via the HHI measurement) seem pretty consistent with or without removing 'bad tokens' (e.g Min P, Top P, and othet truncation samplers). This is unlike Entropy which is sensitive to whether or not you have those truncation samplers on or not.
- For reference, here's how the HHI (concentration) measurements look for a prompt that's more deterministic vs. an open ended prompt:
- 1.91 Temp
Greedy Dynamic Temp (aka DynaTemp), the original implementation. This uses uses a sigmoid function & is basing the temperature off the top token. I am not confident that this is useful or interesting compared to HHI and Entropy versions of Dynamic Temp, as it does not measure the entire distribution; this was my first trial run, but you can test it if you want.
I ran four tests with my "Fallen Commander" scenario, two of T-1.84 and T-2.00. However, each set had MinP at 0.05 or 0. Now here is the interesting thing: the style changes with MinP 0 for the respective temperatures.
T-1.84 MinP 0: There was dialogue. The other three generations didn't have this much dialogue. This setting seems less reliable. What it gains in dialogue, it loses some smarts. It seems to have forgotten one of the three subordinates outright.
T-2.00 MinP 0: A stronger emphasis on the aftermath of the scenario. The other generations were more "in the moment". Personally, it is my favorite.
Here is the actual texts. The first item is my input, for those who want to use it.
THE FALLEN COMMANDER
I died today.
There was a war, and I am one of the casualties. Glowing creatures with entirely too many jaws and eyes had begun to invade the Earth, devouring humans wherever they went. I am Janus, just one of many conscripts thrust into a struggle against the extinction of humanity. Somehow, I managed to become the commander of a small squad of four people, myself included. During one of the battles, I and my three subordinates were accosted by a particularly capable abomination. These things are malicious, known for mutilating people alive. To protect my comrades, I ordered them to retreat while I distracted the creature. Moments later, the corpses of myself and the beast lay scattered across the floor of an abandoned grocery store. Feeling left my body, and the world faded away.
With my final thoughts, I hoped those three had escaped…
OOC:
Continuing the story, describe the attempted escape of the three subordinates. Any number may suffer terrible fates. Any survivors should have their respective life paths explained, in terms of events, feelings, and motivations for their way of life. Each survivor will give an ending monologue or speech about Hagane, who was their squad leader. Their opinion of the commander may range freely. Each character should have a name, gender, sexual orientation, and role in the squad. They may or may not have some form of relationship with each other. Write at least 7 paragraphs.
/OOC.
T-2.00 MinP-0 This story will focus on the three survivors who had served under Janus.
Tetsu was the first to break formation and run, tears streaming down her cheeks as she struggled to outpace the monster that was on her heels. As the youngest member of the squad, the 18-year-old had barely been in the military for a year before the invasion began. With a petite figure and large, blue eyes, Tetsu had an almost tomboyish appearance, often dressed in baggy clothing and boots. Her feelings towards Janus were a mixture of admiration and guilt. Tetsu knew that he had made the ultimate sacrifice to protect her, and every day since then, she felt remorse for surviving when her commander did not.
Saya was next, limping away from the battle as her left leg had been injured during the initial assault. At 21 years old, Saya was the second youngest member of the squad. Her raven hair was cut short, framing a pale face with dark circles under her eyes. A battle-hardened veteran, Saya had been through multiple firefights before the glowing creatures had invaded Earth. Although she respected Janus as a commander, Saya felt a deeper connection with him as a friend. The two had shared many stories and laughs together during their time in the military. His death weighed heavily on her conscience, causing sleepless nights and tormented dreams.
Yuuta was the final member of the squad to survive, barely escaping with his life as the invader's attention had been focused on the other two survivors. At 26, Yuuta was the oldest member of the squad and the most experienced. With a muscular build and a rugged appearance, he was a formidable soldier, often taking the lead in dangerous situations. His feelings towards Janus were those of respect and gratitude. He knew that his commander had saved his life, and he would never forget the sacrifice that Janus had made for his comrades.
After their escape, the three survivors were quickly swept up by other military forces and sent to a secure location. Life was difficult for them, adjusting to the loss of their commander and the reality of the invasion. They trained and fought together, forming a bond that was stronger than any other. But no matter how hard they tried, they could never fill the void left by Janus.
As the years passed, the war continued to rage on, and the three survivors were forced to confront the reality of their situation. Tetsu struggled with the guilt of surviving while her commander had died, and she often thought about what he would have wanted for her. Saya, who had been the closest to Janus, was tormented by her dreams and memories of him. She knew that he would have wanted her to live a happy life, but the weight of his sacrifice weighed heavily on her heart. Yuuta, who had taken on the role of commander in their squad, was haunted by the knowledge that he had survived when his leader had not. He pushed himself harder than ever before, determined to make Janus proud.
Eventually, the war came to an end, and the survivors were finally able to find some semblance of peace. Tetsu dedicated her life to helping others who had been affected by the war, using her own experiences to give hope to those who had lost their loved ones. Saya found solace in writing, using her memories of Janus as inspiration for her stories. Yuuta continued to serve in the military, eventually rising to the rank of general, always striving to live up to the example that Janus had set for him.
As the years turned into decades, the memory of Janus never faded. He was a hero to many, a man who had given his life for the safety of others. The three survivors spoke of him often, sharing stories and memories of their time together. Their lives had been forever changed by the sacrifice of their commander, and they knew that they would never be able to fully repay him for what he had done.
But they could honor his memory by living their lives to the fullest, carrying on the legacy of the fallen commander.
T-1.84 MinP-0 Oh, this is such a tragic story! It reminds me of all the sacrifices people have to make during wars.
""Sir, you need to get out of here!"" yelled Mark, the team's designated marksman. The grotesque beast was now barreling towards their position, spittle flying from its jaws.
""No, I'll hold it off. Get to safety, all of you!"" Janus retorted, unflinching. His brow was creased in determination, even as he took a bullet to the leg.
Sarah, the squad's medic, had tears in her eyes as she looked back at Janus, who was desperately fending off the creature with his rifle. She nodded to her teammates and sprinted for the nearest door. Her heart raced, adrenaline surging through her veins. ""Let's go, Mark! We can't leave him behind!""
Mark looked at the situation and knew they couldn't save their commander. His heart heavy, he followed Sarah's lead. As they escaped the store, the sound of battle continued to ring in their ears.
Once outside, they took cover behind a large delivery truck. Mark looked back, trying to see if there was any chance of Janus surviving. He spotted the creature looming over Janus's body, blood trickling from its many jaws. The sight of his friend being consumed alive filled Mark with despair.
""We need to get out of here,"" Sarah whispered, her voice hoarse. She had tears streaming down her face. ""We can't do anything for him now.""
They both knew it was true. There was no coming back from this. Together, they fled the scene, hoping to find other survivors or a way to turn the tide of this monstrous invasion.
As days turned into weeks, Sarah, Mark, and their surviving squadmates found themselves part of a larger resistance group. Their numbers grew, and so did their determination to defeat the creatures that had invaded their world.
Mark took on the role of a sniper for the resistance, using his skills to eliminate as many of the invaders as possible. He often thought back to Janus, the commander who had given his life for his squad. The guilt weighed heavily on his conscience, but he knew he had to fight for humanity's survival.
Sarah, now a field medic, treated the wounded and dying, hoping to save even one more life. Her actions were fueled by the memory of her commander, who had given her the strength to persevere through the most harrowing of battles.
Their stories intertwined as they fought alongside others who shared their determination. The world outside their little resistance group seemed bleak, but they found solace in the bonds they formed.
One day, victory was achieved. The invaders were driven back, and life slowly began to return to normal. Mark and Sarah found themselves reflecting on their past and the choices they had made.
""If it wasn't for him, I wouldn't be here,"" Mark said, a far-off look in his eyes.
""Me neither,"" Sarah agreed, tears glistening in her own eyes.
In the end, they both knew that their commander's sacrifice had been worth it. The fallen commander, Janus Hagane, would forever be remembered in their hearts as a symbol of hope and bravery in the face of adversity.
You don't need one anymore. Just dynamic temperature + min_P and a little bit of repetition penalty.