pg_upgrade with C-language extensions?

I can't figure out how to get pg_upgrade to work when I have my own
C-language extension.

Say I have this:

CREATE OR REPLACE FUNCTION my_foo() RETURNS integer
AS '/usr/local/mycompany/lib/libmystuff_pg.so', 'my_foo'
LANGUAGE c VOLATILE;

When I run "pg_upgrade ... --check", it complains that it won't be able to
do the update because libmystuff_pg.so isn't available.

OK, fair enough. But ... pg_upgrade has to be able to run both 8.4.17 and
9.5.3 at the same time. So if I compile libmystuff_pg.so against 9.3.5, it
won't work for 8.4.17, and vice versa. On the other hand, if I compile two
versions of libmystuff_pg.so, one for 8.4.17 and one for 9.3.5, then the
"CREATE OR REPLACE" will have the wrong path for one or the other.

How is this supposed to be done? The only thing I can think of is maybe I
should put libmystuff_pg.so into the Postgres lib directory and then leave
the path off of the "CREATE OR REPLACE" statement. I don't like mixing my
code with Postgres code, but is that the only way?

Thanks,
Craig