I am not sure I "solved" this but when I add this to my startup script for my terminal (~/.zshrc):
SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket
export SSH_AUTH_SOCK
it works then. I am not sure I'm still using the ssh agent, but at least it also does not cache my passphrase.
I am not sure I "solved" this but when I add this to my startup script for my terminal (~/.zshrc):
it works then. I am not sure I'm still using the ssh agent, but at least it also does not cache my passphrase/private key