When sudo or sshd or the login prompt checks a password, none of them implement password checking themselves - they call into PAM, which runs a configured stack of modules and reports back success or failure. Understanding PAM explains where to actually change password policy, lockout behavior, or authentication logging, since it's almost never in the application itself.

The four management groups

Every PAM-aware service's config is organized into four kinds of checks, each independently configurable:

Group Purpose
auth Verifies identity - password check, key check, etc.
account Non-authentication checks - is the account expired, locked, allowed to log in at this time
password Handles updating a password (complexity rules apply here)
session Setup/teardown for a logged-in session - mounting a home directory, setting resource limits, logging

Where the config lives

ls /etc/pam.d/
sshd  sudo  su  login  common-auth  common-account  common-password  common-session

One file per PAM-aware service, named after it. Debian/Ubuntu factor shared rules into common-* files that most service configs simply include, so a policy change (like password complexity) is made once and applies everywhere; RHEL/Fedora tend to duplicate more per-service instead. A typical line:

auth    required    pam_unix.so

Reads as: for the auth group, run the pam_unix.so module (standard Unix password checking against /etc/shadow), and this module's result is required to succeed.

Control flags

Flag Behavior on failure
required Records failure, but keeps evaluating the rest of the stack before ultimately denying
requisite Fails immediately, stopping the stack right there
sufficient If this module succeeds, skip the rest and allow immediately (unless an earlier required module already failed)
optional Result mostly ignored unless it's the only module in the stack

Modules in a stack are evaluated in order, and this is exactly how multi-factor setups get built - a sufficient public-key check followed by a required password fallback, for instance.

A practical example: password complexity

password    requisite    pam_pwquality.so retry=3 minlen=12
password    required     pam_unix.so use_authtok sha512 shadow

pam_pwquality (or the older pam_cracklib) enforces rules - minimum length, character variety, rejecting dictionary words - before pam_unix.so actually commits the new password. This is the real mechanism behind "your password must contain..." prompts; the complexity rule lives here, not in any individual application.

Locking accounts after failed attempts

auth    required    pam_faillock.so preauth silent deny=5 unlock_time=900
auth    [success=1 default=bad]    pam_unix.so
auth    [default=die]    pam_faillock.so authfail deny=5 unlock_time=900

pam_faillock (RHEL/Fedora; pam_tally2 is the older equivalent seen on some Debian-based systems) locks an account for a configured period after too many consecutive failed attempts - the underlying mechanism behind account lockout policies, independent of anything fail2ban does at the network/firewall level (see Hardening Basics, which covers fail2ban specifically).

Why this matters for sudo and SSH

Sudo and Privilege and SSH both authenticate through PAM under the hood - /etc/pam.d/sudo and /etc/pam.d/sshd respectively. This is also why disabling password authentication in sshd_config isn't the only place password login could still sneak in: PAM stacks for other services (console login, su) are configured separately and aren't affected by an SSH-specific setting.

Editing PAM config safely

Same caution as /etc/sudoers: a broken PAM config for sudo or login can lock out authentication entirely, including the ability to fix it via normal means. Keep a root shell open (or a second working login session) while testing changes, and confirm a new config works before closing it.