Outdated statement in docs comparing pickle with marshal

[aescarias]

I found this statement in the pickle docs comparing the pickle module with the marshal module:

The pickle module differs from marshal in several significant ways:

• The pickle module keeps track of the objects it has already serialized, so that later references to the same object won’t be serialized again. marshal doesn’t do this.

  This has implications both for recursive objects and object sharing. Recursive objects are objects that contain references to themselves. These are not handled by marshal, and in fact, attempting to marshal recursive objects will crash your Python interpreter. Object sharing happens when there are multiple references to the same object in different places in the object hierarchy being serialized. pickle stores such objects only once, and ensures that all other references point to the master copy. Shared objects remain shared, which can be very important for mutable objects.

However, this appears to no longer be the case as marshal version 3 (added in Python 3.4) supports keeping track of objects via references and as such is capable of serializing recursive objects.

>>> import marshal
>>> xs = [1, 2, 3]
>>> xs.append(xs)
>>> marshal.dumps(xs)
b'\xdb\x04\x00\x00\x00\xe9\x01\x00\x00\x00\xe9\x02\x00\x00\x00\xe9\x03\x00\x00\x00r\x00\x00\x00\x00'

(in Python 3.3, marshal.dumps(xs) fails with ValueError: object too deeply nested to marshal)

So I don't see any reason for this statement to be in the docs in its current state.

[Yashp002]

@aescarias I'd like to take up this PR
However,
Marshal v3 does support recursion via references (as your example shows), so that bullet is outdated.

However sharing question remains—pickle stores shared objects once + rewrites references. Does marshal v3 do object sharing too, or just recursion?

[aescarias]

@aescarias I'd like to take up this PR However, Marshal v3 does support recursion via references (as your example shows), so that bullet is outdated.

However sharing question remains—pickle stores shared objects once + rewrites references. Does marshal v3 do object sharing too, or just recursion?

@Yashp002
Indeed. marshal also supports object sharing.

>>> import marshal
>>> xs = [300, 300, 300]
>>> marshal.dumps(xs)
b'\xdb\x03\x00\x00\x00\xe9,\x01\x00\x00r\x01\x00\x00\x00r\x01\x00\x00\x00'

You can look at marshal.c for the details but, in short, a marshal writer keeps track of objects by storing a map of objects it has already seen. When first writing a "complex object" (anything but singletons and the actual ref objects), it marks it as "referable" by setting the highest bit of its tag (in marshal, an object has a 1-byte "tag" followed by the value). If you now write an object it has already seen, it instead writes a reference object with an index to the already stored object.

So in this example: \xdb\x03\x00\x00\x00 is a list of 3 items (ref. index 0), \xe9,\x01\x00\x00 is the small integer 300 (ref. index 1), and finally, the 2 objects r\x01\x00\x00\x00 are references to ref. index 1.

(before Python 3.4, marshal.dumps(xs) outputs
b'[\x03\x00\x00\x00i,\x01\x00\x00i,\x01\x00\x00i,\x01\x00\x00' which repeats the 3 integers)

[picnixz]

cc @serhiy-storchaka

[serhiy-storchaka]

Yes, this statement is outdated. Marshal protocol 3 and newer supports shared objects and recursive collections. The latter is an unintentional side effect -- it is not needed for .pyc files. Marshal supports even self-recursive tuples which you cannot normally create (pickle don't support them).

If you set pickler.fast = True (you should not), the pickler will also stop supporting shared objects and recursion.

So, I support removing this statement.

On other hand, the second statement can be expanded. Marshal supports only limited set of types, while pickle supports much more builtin and extension types, supports most of user-defined classes by default and allows to add support for classes not supported by default. But this is a different issue.

[Yashp002]

@picnixz @serhiy-storchaka do you think i could push a PR then, I'll remove the part that says marshal doesn't track serialized objects and would you want the expansion saying while marshal handles it too now, pickle is still superior for above said reasons? ⏫