When recording a URL from which a key may be fetched, add an option to store that URL in encrypted form (encrypted to a given public key).
E.g. record the URL as encryptedurl://key_id/[base64-encoded encryption of the original URL].
Many cloud services let you create a pre-signed URL for a non-public file. Anyone with the URL can get the file, so the URL is "public" in that sense;
but you only share the URL with intended recipient(s), not the public. Or you might store files in a bucket that can be publicly read but not listed, and
If the URLs could be stored encrypted in the git-annex branch, one could track such files using the ordinary web remote. One could use an S3 export-tree
remote to share a directory with specific recipient(s), without them needing either AWS credentials or git-annex.