OS X seems to cache ssh-key passwords by default. This is rather useless. You can disable this with
launchctl unload /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist

Source: http://www.reddit.com/r/apple/comments/9fnnc/ask_rapple_how_do_i_disable_sshagent/

If you launch Keychain Access (in /Applications/Utilities/), and remove the entry for your ssh keys, you should be prompted for your password, and have the option to not save it.

To ~remove ssh-agent you can could run..

launchctl unload /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist

..or use http://lingon.sourceforge.net/

Or unset the SSH_AUTH_SOCK env variable

Or, look at the ssh-add command - ssh-add -D forgets all identities (including removing relevant entries from Keychain), ssh-agent -t 123 sets it to remember identities for 123 seconds

No new comments allowed (anymore) on this post.