Ok, here is the current plan.
Change the spamassassin API to pass a hash of tokens into the storage
module, pass the tokens to the proc as an array, start a transaction,
load the tokens into a temp table using copy, select the tokens distinct
into the token table for new tokens, update the token table for known
tokens, then commit.
This solves the following problems:
1. Each email is a transaction instead of each token.
2. The update statement is only called when we really need an update
which avoids all of those searches.
3. The looping work is done inside the proc instead of perl calling a
method a zillion times per email.
I'm not sure how vacuuming will be done yet, if we vacuum once per email
that may be too often, so I may do that every 5 mins in cron.
schu