Postgresql 12, 512 partition by hash. Slow select

I have a table with 120 million rows of data spread among 512
partitioned by hash table. The id column of the table is a uuid, which
is what is being used for the partition hash and it's also the PK for
the table.

The table has a text column, which also has a btree index on it. A
select query on an identical non-partitioned table takes 0.144
seconds, but on the partitioned table it takes 5.689 seconds.

Am I missing something in my setup? Or is this expected? I do know
having more than 100 partitions in prior versions of PostgreSQL 12
would cause a major slow down, but from what I read PostgreSQL 12
addresses that now?

https://www.2ndquadrant.com/en/blog/postgresql-12-partitioning/