Too easy to log in as the "postgres" user?

I've noticed that if I just log in to my server, I don't su to root,
or become the postgres user, I can get straight into the database as
the postgres user merely with "psql -U postgres -h localhost".  My
user account isn't a member of the postgres group.

It appears I've not applied my security settings correctly.  What can
I do to prevent access this way?  I'd still want to be able to su to
the postgres user and log in that way, but not with the -U parameter
allowing access.

The pg_hba.conf is probably relevant here, so this is the setup:

# TYPE DATABASE USER CIDR-ADDRESS METHOD

# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust

Thanks

Thom Brown
Crawley, UK