How to skip existing files when copying with scp

Suppose you have a list of files on a remote host, some of which already exist locally. What you want is to transfer only those files that are not found locally. If you blindly run scp with wildcard, it would fetch all remote files (existing as well as non-existing files), and overwrite existing local files. You want to avoid this. In another similar situation, you may want to upload local files to a remote site, but without replacing any remote files.