Coverage for local/lib/python2.7/site-packages/sage/misc/explain_pickle.py : 69%

""" 

A tool for inspecting Python pickles 

AUTHORS: 

- Carl Witty (2009-03) 

The explain_pickle function takes a pickle and produces Sage code that 

will evaluate to the contents of the pickle. Ideally, the combination 

of explain_pickle to produce Sage code and sage_eval to evaluate the code 

would be a 100% compatible implementation of cPickle's unpickler; this 

is almost the case now. 

EXAMPLES:: 

sage: explain_pickle(dumps(12345)) 

pg_make_integer = unpickle_global('sage.rings.integer', 'make_integer') 

pg_make_integer('c1p') 

sage: explain_pickle(dumps(polygen(QQ))) 

pg_Polynomial_rational_flint = unpickle_global('sage.rings.polynomial.polynomial_rational_flint', 'Polynomial_rational_flint') 

pg_unpickle_PolynomialRing = unpickle_global('sage.rings.polynomial.polynomial_ring_constructor', 'unpickle_PolynomialRing') 

pg_RationalField = unpickle_global('sage.rings.rational_field', 'RationalField') 

pg = unpickle_instantiate(pg_RationalField, ()) 

pg_make_rational = unpickle_global('sage.rings.rational', 'make_rational') 

pg_Polynomial_rational_flint(pg_unpickle_PolynomialRing(pg, ('x',), None, False), [pg_make_rational('0'), pg_make_rational('1')], False, True) 

sage: sage_eval(explain_pickle(dumps(polygen(QQ)))) == polygen(QQ) 

True 

By default (as above) the code produced contains calls to several 

utility functions (unpickle_global, etc.); this is done so that the 

code is truly equivalent to the pickle. If the pickle can be loaded 

into a future version of Sage, then the code that explain_pickle 

produces today should work in that future Sage as well. 

It is also possible to produce simpler code, that is tied to the current 

version of Sage; here are the above two examples again:: 

sage: explain_pickle(dumps(12345), in_current_sage=True) 

from sage.rings.integer import make_integer 

make_integer('c1p') 

sage: explain_pickle(dumps(polygen(QQ)), in_current_sage=True) 

from sage.rings.polynomial.polynomial_rational_flint import Polynomial_rational_flint 

from sage.rings.polynomial.polynomial_ring_constructor import unpickle_PolynomialRing 

from sage.rings.rational import make_rational 

Polynomial_rational_flint(unpickle_PolynomialRing(RationalField(), ('x',), None, False), [make_rational('0'), make_rational('1')], False, True) 

The explain_pickle function has several use cases. 

- Write pickling support for your classes 

You can use explain_pickle to see what will happen when a pickle 

is unpickled. Consider: is this sequence of commands something 

that can be easily supported in all future Sage versions, or does 

it expose internal design decisions that are subject to change? 

- Debug old pickles 

If you have a pickle from an old version of Sage that no longer 

unpickles, you can use explain_pickle to see what it is trying to 

do, to figure out how to fix it. 

- Use explain_pickle in doctests to help maintenance 

If you have a ``loads(dumps(S))`` doctest, you could also add an 

``explain_pickle(dumps(S))`` doctest. Then if something changes 

in a way that would invalidate old pickles, the output of 

``explain_pickle`` will also change. At that point, you can add 

the previous output of :obj:`explain_pickle` as a new set of 

doctests (and then update the :obj:`explain_pickle` doctest to use 

the new output), to ensure that old pickles will continue to work. 

As mentioned above, there are several output modes for :obj:`explain_pickle`, 

that control fidelity versus simplicity of the output. For example, 

the GLOBAL instruction takes a module name and a class name and 

produces the corresponding class. So GLOBAL of ``sage.rings.integer``, 

``Integer`` is approximately equivalent to ``sage.rings.integer.Integer``. 

However, this class lookup process can be customized (using 

sage.structure.sage_object.register_unpickle_override). For instance, 

if some future version of Sage renamed ``sage/rings/integer.pyx`` to 

``sage/rings/knuth_was_here.pyx``, old pickles would no longer work unless 

register_unpickle_override was used; in that case, GLOBAL of 

'sage.rings.integer', 'integer' would mean 

``sage.rings.knuth_was_here.integer``. 

By default, ``explain_pickle`` will map this GLOBAL instruction to 

``unpickle_global('sage.rings.integer', 'integer')``. Then when this code 

is evaluated, unpickle_global will look up the current mapping in the 

register_unpickle_override table, so the generated code will continue 

to work even in hypothetical future versions of Sage where integer.pyx 

has been renamed. 

If you pass the flag ``in_current_sage=True``, then 

:obj:`explain_pickle` will generate code that may only work in the 

current version of Sage, not in future versions. In this case, it 

would generate:: 

from sage.rings.integer import integer 

and if you ran explain_pickle in hypothetical future sage, it would generate: 

from sage.rings.knuth_was_here import integer 

but the current code wouldn't work in the future sage. 

If you pass the flag ``default_assumptions=True``, then 

:obj:`explain_pickle` will generate code that would work in the 

absence of any special unpickling information. That is, in either 

current Sage or hypothetical future Sage, it would generate:: 

from sage.rings.integer import integer 

The intention is that ``default_assumptions`` output is prettier (more 

human-readable), but may not actually work; so it is only intended for 

human reading. 

There are several functions used in the output of :obj:`explain_pickle`. 

Here I give a brief description of what they usually do, as well as 

how to modify their operation (for instance, if you're trying to get 

old pickles to work). 

- ``unpickle_global(module, classname)``: 

unpickle_global('sage.foo.bar', 'baz') is usually equivalent to 

sage.foo.bar.baz, but this can be customized with 

register_unpickle_override. 

- ``unpickle_newobj(klass, args)``: 

Usually equivalent to ``klass.__new__(klass, *args)``. If 

``klass`` is a Python class, then you can define :meth:`__new__` 

to control the result (this result actually need not be an 

instance of klass). (This doesn't work for Cython classes.) 

- ``unpickle_build(obj, state)``: 

If ``obj`` has a :meth:`__setstate__` method, then this is equivalent to 

``obj.__setstate__(state)``. Otherwise uses state to set the attributes 

of ``obj``. Customize by defining :meth:`__setstate__`. 

- ``unpickle_instantiate(klass, args)``: 

Usually equivalent to ``klass(*args)``. Cannot be customized. 

- unpickle_appends(lst, vals): 

Appends the values in vals to lst. If not ``isinstance(lst, list)``, 

can be customized by defining a :meth:`append` method. 

""" 

#***************************************************************************** 

# Copyright (C) 2009 Carl Witty <Carl.Witty@gmail.com> 

# 

# This program is free software: you can redistribute it and/or modify 

# it under the terms of the GNU General Public License as published by 

# the Free Software Foundation, either version 2 of the License, or 

# (at your option) any later version. 

# http://www.gnu.org/licenses/ 

#***************************************************************************** 

from __future__ import absolute_import, print_function 

import sys 

import re 

import types 

from six import iteritems 

from six.moves import cStringIO as StringIO 

from six.moves import cPickle 

import pickletools 

from pickletools import genops 

import zlib as comp 

import bz2 as comp_other 

import sage.all 

from sage.misc.sage_input import SageInputBuilder, SageInputExpression 

from sage.misc.sage_eval import sage_eval 

from sage.structure.sage_object import unpickle_override, unpickle_global, dumps, register_unpickle_override 

try: 

from types import ClassType 

except ImportError: 

# Python 3 does not have a "ClassType". Instead, we ensure that 

# isinstance(foo, ClassType) will always return False. 

ClassType = () 

def explain_pickle(pickle=None, file=None, compress=True, **kwargs): 

r""" 

Explain a pickle. That is, produce source code such that evaluating 

the code is equivalent to loading the pickle. Feeding the result 

of ``explain_pickle`` to ``sage_eval`` should be totally equivalent to loading 

the ``pickle`` with ``cPickle``. 

INPUT: 

- ``pickle`` -- the pickle to explain, as a string (default: None) 

- ``file`` -- a filename of a pickle (default: None) 

- ``compress`` -- if False, don't attempt to decompress the pickle 

(default: True) 

- ``in_current_sage`` -- if True, produce potentially simpler code that is 

tied to the current version of Sage. (default: False) 

- ``default_assumptions`` -- if True, produce potentially simpler code that 

assumes that generic unpickling code will be 

used. This code may not actually work. 

(default: False) 

- ``eval`` -- if True, then evaluate the resulting code and return the 

evaluated result. (default: False) 

- ``preparse`` -- if True, then produce code to be evaluated with 

Sage's preparser; if False, then produce standard 

Python code; if None, then produce code that will work 

either with or without the preparser. (default: True) 

- ``pedantic`` -- if True, then carefully ensures that the result has 

at least as much sharing as the result of cPickle 

(it may have more, for immutable objects). (default: False) 

Exactly one of ``pickle`` (a string containing a pickle) or 

``file`` (the filename of a pickle) must be provided. 

EXAMPLES:: 

sage: explain_pickle(dumps({('a', 'b'): [1r, 2r]})) 

{('a', 'b'):[1r, 2r]} 

sage: explain_pickle(dumps(RR(pi)), in_current_sage=True) 

from sage.rings.real_mpfr import __create__RealNumber_version0 

from sage.rings.real_mpfr import __create__RealField_version0 

__create__RealNumber_version0(__create__RealField_version0(53r, False, 'RNDN'), '3.4gvml245kc0@0', 32r) 

sage: s = 'hi' 

sage: explain_pickle(dumps((s, s))) 

('hi', 'hi') 

sage: explain_pickle(dumps((s, s)), pedantic=True) 

si = 'hi' 

(si, si) 

sage: explain_pickle(dumps(5r)) 

5r 

sage: explain_pickle(dumps(5r), preparse=False) 

5 

sage: explain_pickle(dumps(5r), preparse=None) 

int(5) 

sage: explain_pickle(dumps(22/7)) 

pg_make_rational = unpickle_global('sage.rings.rational', 'make_rational') 

pg_make_rational('m/7') 

sage: explain_pickle(dumps(22/7), in_current_sage=True) 

from sage.rings.rational import make_rational 

make_rational('m/7') 

sage: explain_pickle(dumps(22/7), default_assumptions=True) 

from sage.rings.rational import make_rational 

make_rational('m/7') 

""" 

if pickle is not None: 

p = pickle 

elif file is not None: 

p = open(file).read() 

else: 

raise ValueError("Either pickle or file must be specified") 

if compress: 

try: 

p = comp.decompress(p) 

except Exception: 

try: 

p = comp_other.decompress(p) 

except Exception: 

# Maybe data is uncompressed? 

pass 

return explain_pickle_string(p, **kwargs) 

def explain_pickle_string(pickle, in_current_sage=False, 

default_assumptions=False, eval=False, preparse=True, 

pedantic=False): 

r""" 

This is a helper function for explain_pickle. It takes a decompressed 

pickle string as input; other than that, its options are all the same 

as explain_pickle. 

EXAMPLES:: 

sage: sage.misc.explain_pickle.explain_pickle_string(dumps("Hello, world", compress=False)) 

'Hello, world' 

(See the documentation for ``explain_pickle`` for many more examples.) 

""" 

sib = SageInputBuilder(preparse=preparse) 

pe = PickleExplainer(sib, in_current_sage=in_current_sage, 

default_assumptions=default_assumptions, 

pedantic=pedantic) 

v = pe.run_pickle(pickle) 

ans = sib.result(sib(v)) 

if eval: 

if default_assumptions: 

raise ValueError("Not safe to evaluate code generated with default_assumptions") 

from sage.misc.sage_eval import sage_eval 

result = sage_eval(ans, preparse=preparse) 

print(ans) 

return result 

else: 

return ans 

valid_name_re = re.compile('^[a-zA-Z_][a-zA-Z0-9_]*$') 

def name_is_valid(name): 

r""" 

Test whether a string is a valid Python identifier. (We use a 

conservative test, that only allows ASCII identifiers.) 

EXAMPLES:: 

sage: from sage.misc.explain_pickle import name_is_valid 

sage: name_is_valid('fred') 

True 

sage: name_is_valid('Yes!ValidName') 

False 

sage: name_is_valid('_happy_1234') 

True 

""" 

# Technically, we also need to reject keywords... 

return bool(valid_name_re.match(name)) 

# The pickle interpreter can push and pop "marks" on the stack. 

# This string is used as the representation of a mark. 

the_mark = 'mark' 

class PickleObject(object): 

r""" 

Pickles have a stack-based virtual machine. The explain_pickle 

pickle interpreter mostly uses SageInputExpressions, from sage_input, 

as the stack values. However, sometimes we want some more information 

about the value on the stack, so that we can generate better 

(prettier, less confusing) code. In such cases, we push 

a PickleObject instead of a SageInputExpression. A PickleObject 

contains a value (which may be a standard Python value, or a 

PickleDict or PickleInstance), an expression (a SageInputExpression), 

and an "immutable" flag (which checks whether this object 

has been converted to a SageInputExpression; if it has, then we 

must not mutate the object, since the SageInputExpression would not 

reflect the changes). 

""" 

def __init__(self, value, expression): 

r""" 

Construct a PickleObject. 

TESTS:: 

sage: from sage.misc.explain_pickle import * 

sage: v = PickleObject(1, 2) 

sage: v.value 

1 

sage: v.expression 

2 

sage: v.immutable 

False 

""" 

self.value = value 

self.expression = expression 

self.immutable = False 

def _sage_input_(self, sib, coerced): 

r""" 

Extracts the expression from a PickleObject, and sets the immutable 

flag. 

TESTS:: 

sage: from sage.misc.explain_pickle import * 

sage: v = PickleObject(1, 2) 

sage: v.immutable 

False 

sage: v._sage_input_('sib', False) 

2 

sage: v.immutable 

True 

""" 

self.immutable = True 

return self.expression 

class PickleDict(object): 

r""" 

An object which can be used as the value of a PickleObject. The items 

is a list of key-value pairs, where the keys and values are 

SageInputExpressions. We use this to help construct dictionary literals, 

instead of always starting with an empty dictionary and assigning to 

it. 

""" 

def __init__(self, items): 

r""" 

Initialize a PickleDict. 

EXAMPLES:: 

sage: from sage.misc.explain_pickle import * 

sage: PickleDict([('a', 1)]).items 

[('a', 1)] 

""" 

self.items = items 

class PickleInstance(object): 

r""" 

An object which can be used as the value of a PickleObject. Unlike 

other possible values of a PickleObject, a PickleInstance doesn't represent 

an exact value; instead, it gives the class (type) of the object. 

""" 

def __init__(self, klass): 

r""" 

Initialize a PickleInstance. 

EXAMPLES:: 

sage: from sage.misc.explain_pickle import * 

sage: PickleInstance(Integer).klass 

<type 'sage.rings.integer.Integer'> 

""" 

self.klass = klass 

class PickleExplainer(object): 

r""" 

An interpreter for the pickle virtual machine, that executes 

symbolically and constructs SageInputExpressions instead of 

directly constructing values. 

""" 

def __init__(self, sib, in_current_sage=False, default_assumptions=False, 

pedantic=False): 

r""" 

Initialize a PickleExplainer interpreter for the pickle virtual machine. 

EXAMPLES:: 

sage: from sage.misc.explain_pickle import * 

sage: from sage.misc.sage_input import SageInputBuilder 

sage: pe = PickleExplainer(SageInputBuilder(), in_current_sage=True, default_assumptions=False, pedantic=True) 

sage: pe.in_current_sage 

True 

sage: pe.pedantic 

True 

""" 

self.sib = sib 

self.in_current_sage = in_current_sage 

self.default_assumptions = default_assumptions 

self.pedantic = pedantic 

self.stopped = False 

self.stack = [] 

self.memo = {} 

if in_current_sage and default_assumptions: 

raise ValueError("in_current_sage and default_assumptions must not both be true") 

self.new_instance = self.sib.import_name('types', 'InstanceType') 

def run_pickle(self, p): 

r""" 

Given an (uncompressed) pickle as a string, run the pickle 

in this virtual machine. Once a STOP has been executed, return 

the result (a SageInputExpression representing code which, when 

evaluated, will give the value of the pickle). 

EXAMPLES:: 

sage: from sage.misc.explain_pickle import * 

sage: from sage.misc.sage_input import SageInputBuilder 

sage: sib = SageInputBuilder() 

sage: pe = PickleExplainer(sib, in_current_sage=True, default_assumptions=False, pedantic=True) 

sage: sib(pe.run_pickle('T\5\0\0\0hello.')) 

{atomic:'hello'} 

""" 

for (op, arg, pos) in genops(p): 

assert(not(self.stopped)) 

try: 

handler = getattr(self, op.name) 

except AttributeError: 

raise NotImplementedError('PickleExplainer does not yet handle opcode %s' % op.name) 

if arg is None: 

handler()