Downsides of liberally using CREATE TEMP TABLE ... ON COMMIT DROP

Hello,

I heard at a PostgreSQL talk that you should not liberally create temp
tables in the course of frequently-used functions etc, because (roughly)
you're using up some of the same resources that you for your regular tables.

Is this true? Is there an important reason not to have e.g. a plpgsql
function that uses a temp table? What are the specific problems if I do
this? Is the problem ameliorated if I add ON COMMIT DROP?

Best,
Ryan