Technology

I really wish all of these companies racing to replace their existing software features and employees with LLMs understood this.

They totally understand it. And OpenAI has solved it. For example while researching The Ultimate Answer to Life the Universe and Everything, I asked it to calculate 6 by 9 in base 13 and got the correct answer - 42.

ChatGPT didn't use the LLM to calculate that. It only used the LLM understand an obscure and deliberately confusing chapter of the Hitchhiker's Guide book, to write and execute this python script.

# To calculate six by nine in base 13, we multiply the numbers in our standard decimal system and then convert the result to base 13.

# Calculate 6 * 9 in decimal
result_decimal = 6 * 9

# Convert the result to base 13
# The easiest approach is to use the divmod() function repeatedly to get the remainder (which corresponds to the base 13 digit) 
# and update the quotient for the next iteration until the quotient is 0.

def decimal_to_base_n(num, base):
    if num == 0:
        return "0"
    digits = []
    while num:
        num, remainder = divmod(num, base)
        digits.append(str(remainder))
    return ''.join(digits[::-1])

# Convert the decimal result to base 13
result_base_13 = decimal_to_base_n(result_decimal, 13)

result_base_13

Future systems could for example start asking questions more often

Current systems already do that. But they're expensive and it might be cheaper to have a human do it. Prompt engineering is very much a thing if you're working with high performance low memory consumption language models.

We're a long way from having smartphones with a couple terabytes of RAM and a few thousand GPU cores... but our phones can run basic models and they do. Some phones use a basic LLM for keyboard auto correct for example.

Human experts often say things like "customers say X, they probably mean they want Y and Z" purely based on their experience of dealing with people in some field for a long time.

That is something that can be learned. Follow-up questions can be asked to clarify (or even doubts - "are you sure you don't mean Y instead?"). Etc. Not that complicated.

(Could be why OpenAI chooses to degrade the experience so much when you disable chat history and training in ChatGPT 😀)

Today's LLMs have other quirks, like adding certain words can help even if they don't change the meaning that much, etc., but that's not some magic either.

Ah so you've seen the maintenance cycle on my open-source stuff.

Usually it all springs to life when the cart comes around and the man shouts "bring out yer dead!"