function performs differently with different values

I've got a sql language function which does a fairly simple select from a table. If I give it one value, it performs quickly (half a ms). If I give it another value, it does not (1.1 seconds). When I run the equivalent select outside of the function, both values perform roughly the same (even though one value returns 140k more rows, as expected).

My understanding is that this generally happens because the plan should be different for the different values, but the first time the function is run it caches the plan for one of the values and will never use the appropriate plan for the second value. However, when I do an explain analyze of the raw sql for both values, I get the same plan. So my understanding must be wrong?

I suppose the other possibility is that the slower value is slower in a function because it's returning 140k more rows and the function has to deal with that additional data..... but that seems far-fetched, given that each row is just an int.