Re: Printing backtrace of postgres processes

On Wed, Feb 3, 2021 at 1:49 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
>
> On Wed, Feb 3, 2021 at 1:00 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >
> > vignesh C <vignesh21(at)gmail(dot)com> writes:
> > > On Mon, Feb 1, 2021 at 11:04 AM Bharath Rupireddy
> > > <bharath(dot)rupireddyforpostgres(at)gmail(dot)com> wrote:
> > >> Are these superuser and permission checks enough from a security
> > >> standpoint that we don't expose some sensitive information to the
> > >> user?
> >
> > > This will just print the backtrace of the current backend. Users
> > > cannot get password information from this.
> >
> > Really?
> >
> > A backtrace normally exposes the text of the current query, for
> > instance, which could contain very sensitive data (passwords in ALTER
> > USER, customer credit card numbers in ordinary data, etc etc). We
> > don't allow the postmaster log to be seen by any but very privileged
> > users; it's not sane to think that this data is any less
> > security-critical than the postmaster log.
> >
> > This point is entirely separate from the question of whether
> > triggering stack traces at inopportune moments could cause system
> > malfunctions, but that question is also not to be ignored.
> >
> > TBH, I'm leaning to the position that this should be superuser
> > only. I do NOT agree with the idea that ordinary users should
> > be able to trigger it, even against backends theoretically
> > belonging to their own userid. (Do I need to point out that
> > some levels of the call stack might be from security-definer
> > functions with more privilege than the session's nominal user?)
> >
>
> I had seen that the log that will be logged will be something like:
> postgres: test postgres [local]
> idle(ProcessClientReadInterrupt+0x3a) [0x9500ec]
> postgres: test postgres [local] idle(secure_read+0x183) [0x787f43]
> postgres: test postgres [local] idle() [0x7919de]
> postgres: test postgres [local] idle(pq_getbyte+0x32) [0x791a8e]
> postgres: test postgres [local] idle() [0x94fc16]
> postgres: test postgres [local] idle() [0x950099]
> postgres: test postgres [local] idle(PostgresMain+0x6d5) [0x954bd5]
> postgres: test postgres [local] idle() [0x898a09]
> postgres: test postgres [local] idle() [0x89838f]
> postgres: test postgres [local] idle() [0x894953]
> postgres: test postgres [local] idle(PostmasterMain+0x116b) [0x89422a]
> postgres: test postgres [local] idle() [0x79725b]
> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f6e68d75555]
> postgres: test postgres [local] idle() [0x484249]
> I was not sure if we would be able to get any secure information from
> this. I did not notice the function arguments being printed. I felt
> the function name, offset and the return address will be logged. I
> might be missing something here.
> Thoughts?

First of all, we need to see if the output of pg_print_backtrace shows
up function parameter addresses or only function start addresses along
with line and file information when attached to gdb. In either case,
IMO, it will be easy for experienced hackers(I'm not one though) to
calculate and fetch the query string or other function parameters or
the variables inside the functions from the stack by looking at the
code (which is available openly, of course).

Say, if a backend is in a long running scan or insert operation, then
pg_print_backtrace is issued from another session, the
exec_simple_query function shows up query_string. Below is captured
from attached gdb though, I'm not sure whether the logged backtrace
will have function address or the function parameters addresses, I
think we can check that by having a long running query which
frequently checks interrupts and issue pg_print_backtrace from another
session to that backend. Now, attach gdb to the backend in which the
query is running, then take bt, see if the logged backtrace and the
gdb bt have the same or closer addresses.

#13 0x00005644f4320729 in exec_simple_query (
query_string=0x5644f6771bf0 "select pg_backend_pid();") at postgres.c:1240
#14 0x00005644f4324ff4 in PostgresMain (argc=1, argv=0x7ffd819bd5e0,
dbname=0x5644f679d2b8 "postgres", username=0x5644f679d298 "bharath")
at postgres.c:4394
#15 0x00005644f4256f9d in BackendRun (port=0x5644f67935c0) at postmaster.c:4484
#16 0x00005644f4256856 in BackendStartup (port=0x5644f67935c0) at
postmaster.c:4206
#17 0x00005644f4252a11 in ServerLoop () at postmaster.c:1730
#18 0x00005644f42521aa in PostmasterMain (argc=3, argv=0x5644f676b1f0)
at postmaster.c:1402
#19 0x00005644f4148789 in main (argc=3, argv=0x5644f676b1f0) at main.c:209

As suggested by Tom, I'm okay if this function is callable only by the
superusers. In that case, the superusers can fetch the backtrace and
send it for further analysis in case of any hangs or issues.

Others may have better thoughts.

With Regards,
Bharath Rupireddy.
EnterpriseDB: http://www.enterprisedb.com