Neat VNC: Buffer overflow due to oversized RSA public keys

Neat VNC is a VNC server library. Prior to 0.9.6, a pre-authentication stack buffer overflow exists in neatvnc in the RSA-AES security type handler. An unauthenticated remote attacker who can reach the VNC listening socket can send a crafted security type 5 (RSA-AES) or security type 129 (RSA-AES-256) handshake with an oversized client RSA public key, causing rsa_aes_send_challenge in src/auth/rsa-aes.c to overflow a 1024-byte on-stack buffer when encrypting the server challenge. This results in at least a denial of service via server crash. This vulnerability is fixed in 0.9.6.

server.c in Neat VNC (aka neatvnc) before 0.8.1 does not properly validate the security type, a related issue to CVE-2006-2369.