How does WinRAR Check a Password?
WinRAR does not check a password at all. Instead, it passes the password through the hash function and sets an AES-128/256 Bit encryption key; then uses this key to encrypt the file data. (only valid until RAR 4.x format).
Since the development of the RAR 5.0 format, WinRAR detects wrong passwords before starting extraction and does not extract garbage. RAR 5.0 stores a special password hash generated by a one-way hash function.
When a password is entered, RAR compares its hash to the stored hash; in case of no match, it rejects the wrong password early. This one-way hash function is intentionally slow and based on PBKDF2. This noticeably decreases the chances of a successful brute force attack.
