codekingpro/portable-devtools

1#! /usr/bin/env python32 3"""4combinerefs path5 6A helper for analyzing PYTHONDUMPREFS output.7 8When the PYTHONDUMPREFS envar is set in a debug build, at Python shutdown9time Py_FinalizeEx() prints the list of all live objects twice:  first it10prints the repr() of each object while the interpreter is still fully intact.11After cleaning up everything it can, it prints all remaining live objects12again, but the second time just prints their addresses, refcounts, and type13names (because the interpreter has been torn down, calling repr methods at14this point can get into infinite loops or blow up).15 16Save all this output into a file, then run this script passing the path to17that file.  The script finds both output chunks, combines them, then prints18a line of output for each object still alive at the end:19 20    address refcnt typename repr21 22address is the address of the object, in whatever format the platform C23produces for a %p format code.24 25refcnt is of the form26 27    "[" ref "]"28 29when the object's refcount is the same in both PYTHONDUMPREFS output blocks,30or31 32    "[" ref_before "->" ref_after "]"33 34if the refcount changed.35 36typename is Py_TYPE(object)->tp_name, extracted from the second PYTHONDUMPREFS37output block.38 39repr is repr(object), extracted from the first PYTHONDUMPREFS output block.40CAUTION:  If object is a container type, it may not actually contain all the41objects shown in the repr:  the repr was captured from the first output block,42and some of the containees may have been released since then.  For example,43it's common for the line showing the dict of interned strings to display44strings that no longer exist at the end of Py_FinalizeEx; this can be recognized45(albeit painfully) because such containees don't have a line of their own.46 47The objects are listed in allocation order, with most-recently allocated48printed first, and the first object allocated printed last.49 50 51Simple examples:52 53    00857060 [14] str '__len__'54 55The str object '__len__' is alive at shutdown time, and both PYTHONDUMPREFS56output blocks said there were 14 references to it.  This is probably due to57C modules that intern the string "__len__" and keep a reference to it in a58file static.59 60    00857038 [46->5] tuple ()61 6246-5 = 41 references to the empty tuple were removed by the cleanup actions63between the times PYTHONDUMPREFS produced output.64 65    00858028 [1025->1456] str '<dummy key>'66 67The string '<dummy key>', which is used in dictobject.c to overwrite a real68key that gets deleted, grew several hundred references during cleanup.  It69suggests that stuff did get removed from dicts by cleanup, but that the dicts70themselves are staying alive for some reason. """71 72import re73import sys74 75# Generate lines from fileiter.  If whilematch is true, continue reading76# while the regexp object pat matches line.  If whilematch is false, lines77# are read so long as pat doesn't match them.  In any case, the first line78# that doesn't match pat (when whilematch is true), or that does match pat79# (when whilematch is false), is lost, and fileiter will resume at the line80# following it.81def read(fileiter, pat, whilematch):82    for line in fileiter:83        if bool(pat.match(line)) == whilematch:84            yield line85        else:86            break87 88def combinefile(f):89    fi = iter(f)90 91    for line in read(fi, re.compile(r'^Remaining objects:$'), False):92        pass93 94    crack = re.compile(r'([a-zA-Z\d]+) \[(\d+)\] (.*)')95    addr2rc = {}96    addr2guts = {}97    before = 098    for line in read(fi, re.compile(r'^Remaining object addresses:$'), False):99        m = crack.match(line)100        if m:101            addr, addr2rc[addr], addr2guts[addr] = m.groups()102            before += 1103        else:104            print('??? skipped:', line)105 106    after = 0107    for line in read(fi, crack, True):108        after += 1109        m = crack.match(line)110        assert m111        addr, rc, guts = m.groups() # guts is type name here112        if addr not in addr2rc:113            print('??? new object created while tearing down:', line.rstrip())114            continue115        print(addr, end=' ')116        if rc == addr2rc[addr]:117            print('[%s]' % rc, end=' ')118        else:119            print('[%s->%s]' % (addr2rc[addr], rc), end=' ')120        print(guts, addr2guts[addr])121 122    print("%d objects before, %d after" % (before, after))123 124def combine(fname):125    with open(fname) as f:126        combinefile(f)127 128if __name__ == '__main__':129    combine(sys.argv[1])130