This is just a heads-up for anyone using Ruby on Rails (with
ActiveRecord) on JRuby who sees performance degradation over time. Each
Ruby runtime instance will degrade query performance slightly. You can
see this if the minimum and maximum number of active runtimes are not
configured to the same value, or (at a potentially extreme rate) if you
have Ruby class caching turned off. Class caching should only be off
for development, but we saw this issue in production when someone copied
the development directory to production without changing this setting.
The cause is a bug in the RoR activerecord-jdbc-adapter gem. It
creates an instance of the JDBC Driver class and explicitly registers it
with Java's DriverManager. This DriverManager.registerDriver method is
only supposed to be invoked from a static initializer in the driver
class itself, so this is just plain wrong on the part of the gem. The
registerDriver method just adds the Driver instance (along with a bit of
related information) in a java.util.Vector list.
The performance hit comes when you try to connect through
DriverManager. It sequentially scans through the list from the front,
checking whether each driver instance was loaded with the same class
loader as the requester. The newer driver instances are at the end, so
once you accumulate enough driver instances, this search can take quite
a while. (With class caching off in production, we accumulated tens of
thousands of driver instances, each keeping a zombie class loader alive,
before it started causing failures from resource exhaustion. At that
point, the time to find the right driver instance was up to about 15
seconds.)
It strikes me that not only should the gem not be explicitly
registering the driver instances, but it should be finding and
*deregistering* any drivers before its Ruby runtime is torn down;
otherwise there is a memory leak as runtimes are created and destroyed,
even with proper start-up behavior -- just not as fast as with the
explicit register (which creates a second driver instance for each
runtime).
This problem is not PostgreSQL specific and does *not* reflect any bug
or flaw in PostgreSQL; but PostgreSQL was initially blamed by our
programmers. I'm sharing the information to save time for any other DBA
who may be faced with similar deterioration in performance for a RoR
environment. We will be discussing the issue on the JRuby list and will
be attempting to work with them on a proper fix. In the meantime you
can work around the issue by making sure that Ruby class caching is on
and minimum and maximum active runtimes match.
-Kevin