[ZODB-Dev] cPickle: state is not a directory/Upgrade to Zope 2.7 failing

[Andreas Jung]

Post by Andreas Jung
I am trying to move a Zope 2.6.2 site (Python 2.2.3, CMF 1.4) to Zope
2.7 with Python 2.3.2.
----
2003-11-03T14:52:44 INFO(0) ZODB Mounted database
'145966912_1035475459.685228' at /temp_folder
------
2003-11-03T14:52:44 ERROR(200) ZODB Couldn't load state for
File "/develop/sandboxes/tracker/Zope/lib/python/ZODB/Connection.py",

line 564, in setstate

Post by Andreas Jung
self._set_ghost_state(obj, p)
File "/develop/sandboxes/tracker/Zope/lib/python/ZODB/Connection.py",

line 603, in _set_ghost_state state = unpickler.load()

Post by Andreas Jung
UnpicklingError: state is not a dictionary
...

There's not much I can deduce from that, other than that cPickle is trying
to create an instance of some class, where the class doesn't define
__setstate__ itself, but pickling provided *some* state object for use by
the default __setstate__, but provided a bogus state object.

So we need more info. Two things:

1. Fiddle _set_ghost_state to output repr(p) when this fails.
That is, I need to see the actual pickle, in all its gory glory.
All we can deduce from the above is that the state object is
neither None nor a dict (the error message told us it's not a
dict, and it wouldn't have gotten to that check if it were None).

2. You might try fiddling _set_ghost_state to use pickle.py instead
of the cPickle module. Sometimes there's a bug in one that's not
in the other, and even if not pickle.py sometimes produces
error messages that are more informative.

Andreas Jung

2003-11-04 08:22:28 UTC

Post by Tim Peters
1. Fiddle _set_ghost_state to output repr(p) when this fails.
That is, I need to see the actual pickle, in all its gory glory.
All we can deduce from the above is that the state object is
neither None nor a dict (the error message told us it's not a
dict, and it wouldn't have gotten to that check if it were None).

repr(p):

'((U\x0fHelpSys.HelpSysq\x01U\x0bProductHelpq\x02tq\x03Nt.}q\x04(U\x07catal
ogq\x05(U\x08\x00\x00\x00\x00\x00\x01FSq\x06(U\x1aProducts.ZCatalog.ZCatalo
gq\x07U\x08ZCatalogq\x08ttQU\x0elastRegisteredq\t(cDateTime.DateTime\nDateT
ime\nq\nNoq\x0b(GA\xcf\xb7\xf0\xe4\x94\x18\x93U\x05GMT+2q\x0ctbU\x05titleq\
rU\x14CMF CollectorNG
Helpq\x0eU\x08_objectsq\x0f(}q\x10(U\tmeta_typeq\x11U\nHelp
Topicq\x12U\x02idq\x13U\x0fplaceholder.txtq\x14uth\x13U\x04Helpq\x15U\x0fpl
aceholder.txtq\x16(U\x08\x00\x00\x00\x00\x00\x01FTq\x17(U\x11HelpSys.HelpTo
picq\x18U\x08STXTopicq\x19ttQu.'

Post by Tim Peters
2. You might try fiddling _set_ghost_state to use pickle.py instead
of the cPickle module. Sometimes there's a bug in one that's not
in the other, and even if not pickle.py sometimes produces
error messages that are more informative.

Using pickle.py:

------
2003-11-04T08:16:50 INFO(0) ZServer HTTP server started at Tue Nov 4
08:16:50 2003
Hostname: sentinel
Port: 20080
------
2003-11-04T08:16:50 INFO(0) ZServer FTP server started at Tue Nov 4
08:16:50 2003
Hostname: localhost.localdomain
Port: 20021
------
2003-11-04T08:16:51 DEBUG(-200) Localizer Unicode patching for Zope 2.6b1+
------
2003-11-04T08:16:51 DEBUG(-200) FileStorage create storage
/develop/sandboxes/tracker/instance/var/Data.fs
------
2003-11-04T08:16:51 INFO(0) ZODB Opening database for mounting:
'145966912_1035475459.685228'
------
2003-11-04T08:16:51 DEBUG(-200) TemporaryStorage create storage
TemporaryStorage
------
2003-11-04T08:16:51 INFO(0) ZODB Mounted database
'145966912_1035475459.685228' at /temp_folder
------
2003-11-04T08:16:51 ERROR(200) ZODB Couldn't load state for 0000000000014652
Traceback (most recent call last):
File "/develop/sandboxes/tracker/Zope/lib/python/ZODB/Connection.py",
line 564, in setstate
self._set_ghost_state(obj, p)
File "/develop/sandboxes/tracker/Zope/lib/python/ZODB/Connection.py",
line 606, in _set_ghost_state
state = unpickler.load()
File "/opt/python-2.3.2/lib/python2.3/pickle.py", line 872, in load
dispatch[key](self)
File "/opt/python-2.3.2/lib/python2.3/pickle.py", line 1244, in load_build
inst.__dict__.update(state)
AttributeError: keys
------
2003-11-04T08:16:51 ERROR(200) Zope Couldn't install CMFCollectorNG
Traceback (most recent call last):
File "/develop/sandboxes/tracker/Zope/lib/python/OFS/Application.py",
line 642, in install_product
initmethod(context)
File
"/develop/sandboxes/tracker/instance/Products/CMFCollectorNG/__init__.py",
line 52, in initialize
context.registerHelp(directory='help')
File "/develop/sandboxes/tracker/Zope/lib/python/App/ProductContext.py",
line 302, in registerHelp
if help.lastRegistered is not None and \
AttributeError: lastRegistered
Traceback (most recent call last):
File "/develop/sandboxes/tracker/Zope/lib/python/Zope/Startup/run.py",
line 27, in ?
run()
File "/develop/sandboxes/tracker/Zope/lib/python/Zope/Startup/run.py",
line 24, in run
start_zope(opts.configroot)
File
"/develop/sandboxes/tracker/Zope/lib/python/Zope/Startup/__init__.py", line
50, in start_zope
starter.startZope()
File
"/develop/sandboxes/tracker/Zope/lib/python/Zope/Startup/__init__.py", line
221, in startZope
Zope.startup()
File "/develop/sandboxes/tracker/Zope/lib/python/Zope/__init__.py", line
46, in startup
_startup()
File "/develop/sandboxes/tracker/Zope/lib/python/Zope/App/startup.py",
line 95, in startup
OFS.Application.initialize(application)
File "/develop/sandboxes/tracker/Zope/lib/python/OFS/Application.py",
line 421, in initialize
install_products(app)
File "/develop/sandboxes/tracker/Zope/lib/python/OFS/Application.py",
line 580, in install_products
folder_permissions, raise_exc=debug_mode)
File "/develop/sandboxes/tracker/Zope/lib/python/OFS/Application.py",
line 642, in install_product
initmethod(context)
File
"/develop/sandboxes/tracker/instance/Products/CMFCollectorNG/__init__.py",
line 52, in initialize
context.registerHelp(directory='help')
File "/develop/sandboxes/tracker/Zope/lib/python/App/ProductContext.py",
line 302, in registerHelp
if help.lastRegistered is not None and \
AttributeError: lastRegistered

COuld be something with the HelpSys but I have no idea at this time what
the problem could be...doing
some further investigations.

-aj

Andreas Jung

2003-11-04 09:11:17 UTC

--On Dienstag, 4. November 2003 8:22 Uhr +0100 Andreas Jung

Post by Andreas Jung
COuld be something with the HelpSys but I have no idea at this time what
the problem could be...doing
some further investigations.

The site now starts up after adding a "return" statement inside
registerHelp() of the HelpSys and inside
the ZopeTutorial (using pickle.py, cPickle still fails). Ok, I can get to
the ZMI but my CMF site show up
as Folder and when I try to open the folder it won't open and spits at me
:-/ ...very evil....

Andreas

Andreas Jung

2003-11-04 09:49:01 UTC

--On Dienstag, 4. November 2003 9:11 Uhr +0100 Andreas Jung

Post by Andreas Jung
The site now starts up after adding a "return" statement inside
registerHelp() of the HelpSys and inside
the ZopeTutorial (using pickle.py, cPickle still fails). Ok, I can get to
the ZMI but my CMF site show up
as Folder and when I try to open the folder it won't open and spits at me
:-/ ...very evil....

All these error happened with Zope 2.7 (2.7 branch) and Python 2.3.2.
After switching back to Zope 2.7 + Python 2.2.3 everything is working fine.
So there seems to be some compatibility problem in the pickle implementation
of 2.3.2 vs 2.2.3?! Who is in charge for further investigation?!

Andreas

Jeremy Hylton

2003-11-04 16:03:17 UTC

You're doing great so far! <wink>

Jeremy

Andreas Jung

2003-11-04 16:13:34 UTC

--On Dienstag, 4. November 2003 9:59 Uhr -0500 Jeremy Hylton

Post by Jeremy Hylton

Post by Andreas Jung
All these error happened with Zope 2.7 (2.7 branch) and Python 2.3.2.
After switching back to Zope 2.7 + Python 2.2.3 everything is working
fine. So there seems to be some compatibility problem in the pickle
implementation of 2.3.2 vs 2.2.3?! Who is in charge for further
investigation?!

You're doing great so far! <wink>

I knowledge in pickles is very limited...so I am really not the right
person.
SInce this seems to be an important issue someone with pickle-zen
should have a look at this.

-aj

Jeremy Hylton

2003-11-04 16:22:18 UTC

Post by Jeremy Hylton

You're doing great so far! <wink>

I knowledge in pickles is very limited...so I am really not the right
person.
SInce this seems to be an important issue someone with pickle-zen
should have a look at this.

That's right. Sorry that the wink didn't convey that clearly. I'm
looking at it now, but it seems clear that Tim will need to take a look.
As far as I can see, there is definitely something wrong with the
pickle. I can't unpickle it with either Python 2.3 or 2.2; they both
fail the same way. I don't know why Zope is behaving differently for
you with different versions.

Since I can't unpickle the pickle, I expect there's something wrong with
the code that wrote it.

Jeremy

Tim Peters

2003-11-04 17:18:04 UTC

Recap: Andreas has a Zope 2.6.2 site (Python 2.2.3, CMF 1.4), and is trying to
migrate it to Zope 2.7 with Python 2.3.2. He gets exceptions upon startup,
coming out of _set_ghost_state, with cPickle complaining:

cPickle.UnpicklingError: state is not a dictionary

[Tim]

[Andreas Jung]

"""
repr(p):

'((U\x0fHelpSys.HelpSysq\x01U\x0bProductHelpq\x02tq\x03Nt.}q\x04(U\x07catal
ogq\x05(U\x08\x00\x00\x00\x00\x00\x01FSq\x06(U\x1aProducts.ZCatalog.ZCatalo
gq\x07U\x08ZCatalogq\x08ttQU\x0elastRegisteredq\t(cDateTime.DateTime\nDateT
ime\nq\nNoq\x0b(GA\xcf\xb7\xf0\xe4\x94\x18\x93U\x05GMT+2q\x0ctbU\x05titleq\
rU\x14CMF CollectorNG
Helpq\x0eU\x08_objectsq\x0f(}q\x10(U\tmeta_typeq\x11U\nHelp
Topicq\x12U\x02idq\x13U\x0fplaceholder.txtq\x14uth\x13U\x04Helpq\x15U\x0fpl
aceholder.txtq\x16(U\x08\x00\x00\x00\x00\x00\x01FTq\x17(U\x11HelpSys.HelpTo
picq\x18U\x08STXTopicq\x19ttQu.'
"""

That obviously <wink> explains it, but I'm not sure what we can do about it --
it is indeed a bad pickle. The difference is that, before Python 2.3, cPickle
skipped a crucial type check and ended up simply ignoring the offending part of
the pickle; several holes "like that" were fixed in 2.3 cPickle, and it
properly complains. So that's the first point to take away: this pickle is
bad, and never should have worked.

2.3 has a new pickletools module, with a handy dis() function for doing a
symbolic dump of pickles. That pickle contains two things, and
_set_ghost_state complains during the second. Here's a dump of the first
thing:

0: ( MARK
1: ( MARK
2: U SHORT_BINSTRING 'HelpSys.HelpSys'
19: q BINPUT 1
21: U SHORT_BINSTRING 'ProductHelp'
34: q BINPUT 2
36: t TUPLE (MARK at 1)
37: q BINPUT 3
39: N NONE
40: t TUPLE (MARK at 0)
41: . STOP

That's identifying the module and class of an object. It looks fine. The
second thing in the pickle is trying to record that object's state:

0: } EMPTY_DICT
1: q BINPUT 4
3: ( MARK
4: U SHORT_BINSTRING 'catalog'
13: q BINPUT 5
15: ( MARK
16: U SHORT_BINSTRING '\x00\x00\x00\x00\x00\x01FS'
26: q BINPUT 6
28: ( MARK
29: U SHORT_BINSTRING 'Products.ZCatalog.ZCatalog'
57: q BINPUT 7
59: U SHORT_BINSTRING 'ZCatalog'
69: q BINPUT 8
71: t TUPLE (MARK at 28)
72: t TUPLE (MARK at 15)
73: Q BINPERSID
74: U SHORT_BINSTRING 'lastRegistered'
90: q BINPUT 9
92: ( MARK
93: c GLOBAL 'DateTime.DateTime DateTime'
121: q BINPUT 10
123: N NONE
124: o OBJ (MARK at 92)
125: q BINPUT 11
127: ( MARK
128: G BINFLOAT 1064296905.1569999
137: U SHORT_BINSTRING 'GMT+2'
144: q BINPUT 12
146: t TUPLE (MARK at 127)
147: b BUILD
148: U SHORT_BINSTRING 'title'
155: q BINPUT 13
157: U SHORT_BINSTRING 'CMF CollectorNG Help'
179: q BINPUT 14
181: U SHORT_BINSTRING '_objects'
191: q BINPUT 15
193: ( MARK
194: } EMPTY_DICT
195: q BINPUT 16
197: ( MARK
198: U SHORT_BINSTRING 'meta_type'
209: q BINPUT 17
211: U SHORT_BINSTRING 'Help Topic'
223: q BINPUT 18
225: U SHORT_BINSTRING 'id'
229: q BINPUT 19
231: U SHORT_BINSTRING 'placeholder.txt'
248: q BINPUT 20
250: u SETITEMS (MARK at 197)
251: t TUPLE (MARK at 193)
252: h BINGET 19
254: U SHORT_BINSTRING 'Help'
260: q BINPUT 21
262: U SHORT_BINSTRING 'placeholder.txt'
279: q BINPUT 22
281: ( MARK
282: U SHORT_BINSTRING '\x00\x00\x00\x00\x00\x01FT'
292: q BINPUT 23
294: ( MARK
295: U SHORT_BINSTRING 'HelpSys.HelpTopic'
314: q BINPUT 24
316: U SHORT_BINSTRING 'STXTopic'
326: q BINPUT 25
328: t TUPLE (MARK at 294)
329: t TUPLE (MARK at 281)
330: Q BINPERSID
331: u SETITEMS (MARK at 3)
332: . STOP

The offending "subpickle" has repr

(cDateTime.DateTime\nDateTime\nq\nNoq\x0b(GA\xcf\xb7\xf0\xe4\x94\x18\x93Ux05GMT
+2q\x0ctb.

and corresponds to the second dump above starting with

92: ( MARK

and ending with

147: b BUILD

It's the BUILD opcode that's complaining in 2.3. I created a binary file
"pick" with just this part of the pickle (with a STOP opcode appended), and
I'll attach that. Here's a full dump of the attached pickle:

0: ( MARK
1: c GLOBAL 'DateTime.DateTime DateTime'
29: q BINPUT 10
31: N NONE
32: o OBJ (MARK at 0)
33: q BINPUT 11
35: ( MARK
36: G BINFLOAT 1064296905.1569999
45: U SHORT_BINSTRING 'GMT+2'
52: q BINPUT 12
54: t TUPLE (MARK at 35)
55: b BUILD
56: . STOP

This pickle creates a DateTime object, and then *tries* to set its state from a
tuple. That's not legal -- a tuple makes no sense here. Before Python 2.3,
cPickle silently ignored the tuple, seemingly by accident. 2.2.3's cPickle's
load_build() contains:

i = 0;
while (PyDict_Next(value, &i, &d_key, &d_value)) {
if (PyObject_SetItem(instdict, d_key, d_value) < 0) {
r=-1;
break;
}
}

where "value" is the tuple. Alas, PyDict_Next just returns 0, without setting
an exception, if the thing passed to it isn't an instance of a dict subclass.
So, in 2.2.3, this loop ends unexceptionally on the the first PyDict_Next call,
and the "state tuple" is plain ignored.

It's more complicated in 2.3, because a tuple of (exactly) length 2 actually
*is* legal here in 2.3. It expects the 2-tuple to contain a state dict, and a
slot state dict (before 2.3, objects with slots couldn't be pickled reliably).
So, under 2.3, the float 1064296905.1569999 gets unpacked into "the state
dict", and the string 'GMT+2' gets unpacked into "the slotstate dict". Float
isn't a dict subclass, so 2.3's load_build() properly gripes with "state is not
a dictionary".

[Andreas]
"""
Using pickle.py:

------
2003-11-04T08:16:50 INFO(0) ZServer HTTP server started at Tue Nov 4
08:16:50 2003
Hostname: sentinel
Port: 20080
------
2003-11-04T08:16:50 INFO(0) ZServer FTP server started at Tue Nov 4
08:16:50 2003
Hostname: localhost.localdomain
Port: 20021
------
2003-11-04T08:16:51 DEBUG(-200) Localizer Unicode patching for Zope 2.6b1+
------
2003-11-04T08:16:51 DEBUG(-200) FileStorage create storage
/develop/sandboxes/tracker/instance/var/Data.fs
------
2003-11-04T08:16:51 INFO(0) ZODB Opening database for mounting:
'145966912_1035475459.685228'
------
2003-11-04T08:16:51 DEBUG(-200) TemporaryStorage create storage
TemporaryStorage
------
2003-11-04T08:16:51 INFO(0) ZODB Mounted database
'145966912_1035475459.685228' at /temp_folder
------
2003-11-04T08:16:51 ERROR(200) ZODB Couldn't load state for 0000000000014652
Traceback (most recent call last):
File "/develop/sandboxes/tracker/Zope/lib/python/ZODB/Connection.py",
line 564, in setstate
self._set_ghost_state(obj, p)
File "/develop/sandboxes/tracker/Zope/lib/python/ZODB/Connection.py",
line 606, in _set_ghost_state
state = unpickler.load()
File "/opt/python-2.3.2/lib/python2.3/pickle.py", line 872, in load
dispatch[key](self)
File "/opt/python-2.3.2/lib/python2.3/pickle.py", line 1244, in load_build
inst.__dict__.update(state)
AttributeError: keys
------
2003-11-04T08:16:51 ERROR(200) Zope Couldn't install CMFCollectorNG
Traceback (most recent call last):
File "/develop/sandboxes/tracker/Zope/lib/python/OFS/Application.py",
line 642, in install_product
initmethod(context)
File
"/develop/sandboxes/tracker/instance/Products/CMFCollectorNG/__init__.py",
line 52, in initialize
context.registerHelp(directory='help')
File "/develop/sandboxes/tracker/Zope/lib/python/App/ProductContext.py",
line 302, in registerHelp
if help.lastRegistered is not None and \
AttributeError: lastRegistered
Traceback (most recent call last):
File "/develop/sandboxes/tracker/Zope/lib/python/Zope/Startup/run.py",
line 27, in ?
run()
File "/develop/sandboxes/tracker/Zope/lib/python/Zope/Startup/run.py",
line 24, in run
start_zope(opts.configroot)
File
"/develop/sandboxes/tracker/Zope/lib/python/Zope/Startup/__init__.py", line
50, in start_zope
starter.startZope()
File
"/develop/sandboxes/tracker/Zope/lib/python/Zope/Startup/__init__.py", line
221, in startZope
Zope.startup()
File "/develop/sandboxes/tracker/Zope/lib/python/Zope/__init__.py", line
46, in startup
_startup()
File "/develop/sandboxes/tracker/Zope/lib/python/Zope/App/startup.py",
line 95, in startup
OFS.Application.initialize(application)
File "/develop/sandboxes/tracker/Zope/lib/python/OFS/Application.py",
line 421, in initialize
install_products(app)
File "/develop/sandboxes/tracker/Zope/lib/python/OFS/Application.py",
line 580, in install_products
folder_permissions, raise_exc=debug_mode)
File "/develop/sandboxes/tracker/Zope/lib/python/OFS/Application.py",
line 642, in install_product
initmethod(context)
File
"/develop/sandboxes/tracker/instance/Products/CMFCollectorNG/__init__.py",
line 52, in initialize
context.registerHelp(directory='help')
File "/develop/sandboxes/tracker/Zope/lib/python/App/ProductContext.py",
line 302, in registerHelp
if help.lastRegistered is not None and \
AttributeError: lastRegistered
"""

I expect you'd get much the same if you tried this using pickle.py under 2.2.3
too. The

AttributeError: keys

you're getting from pickle.py's load_build() is its way of spelling "hey, the
state isn't a dict or subclass of dict".

f = open('/code/pick', 'rb')
guts = f.read()
import cPickle
cPickle.loads(guts) # this is the same error you originally reported

Traceback (most recent call last):
File "<stdin>", line 1, in ?
cPickle.UnpicklingError: state is not a dictionary

import pickle
pickle.loads(guts) # this is the same error you showed from pickle.py

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\PYTHON23\lib\pickle.py", line 1394, in loads
return Unpickler(file).load()
File "C:\PYTHON23\lib\pickle.py", line 872, in load
dispatch[key](self)
File "C:\PYTHON23\lib\pickle.py", line 1244, in load_build
inst.__dict__.update(state)
AttributeError: keys

f = open('/code/pick', 'rb')
guts = f.read()
import cPickle
cPickle.loads(guts)

DateTime('2003/11/04 10:22:44.270 US/Eastern')

Note that while the loads() didn't complain, it didn't create the intended
object either: that's *current* date and time, as of the time I ran that
example. The time recorded in the bogus "state tuple" was ignored.

So the crucial question is who produces DateTime pickles of this form? All
such pickles will silently do a wrong thing when loaded with a Python before
2.3 (they'll all load as current date and time), and will raise an exception
when loaded under 2.3+.

import pickle
pickle.loads(guts)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\PYTHON22\lib\pickle.py", line 986, in loads
return Unpickler(file).load()
File "C:\PYTHON22\lib\pickle.py", line 597, in load
dispatch[key](self)
File "C:\PYTHON22\lib\pickle.py", line 939, in load_build
inst.__dict__.update(value)
AttributeError: keys
So pickle.py dies the same way under 2.2.3 and 2.3.2.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pick
Type: application/octet-stream
Size: 57 bytes
Desc: not available
Url : http://lists.zope.org/pipermail/zodb-dev/attachments/20031104/a67e8df9/pick-0001.obj

Jim Fulton

2003-11-04 17:56:09 UTC

Post by Tim Peters
Recap: Andreas has a Zope 2.6.2 site (Python 2.2.3, CMF 1.4), and is trying to
migrate it to Zope 2.7 with Python 2.3.2. He gets exceptions upon startup,
cPickle.UnpicklingError: state is not a dictionary
[Tim]

[Andreas Jung]

...

Post by Tim Peters
This pickle creates a DateTime object, and then *tries* to set its state from a
tuple. That's not legal -- a tuple makes no sense here.

That's a little missleading. The state can be anything. as long as the class
provides __setstate__ method that handles it.

It looks as though this pickle was written from a DateTime class with
custom __getstate__ and __setstate__. This would be useful to save space.
I can find no evidence of this in CVS though. Is it possible that sombody
had installed such an optimization in Adreas' Zope 2.6 installation?
If so, somebody needs to migrate that forward to the 2.7 installation.

Jim

--
Jim Fulton mailto:***@zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org

Andreas Jung

2003-11-04 18:10:45 UTC

Post by Jim Fulton
It looks as though this pickle was written from a DateTime class with
custom __getstate__ and __setstate__. This would be useful to save space.
I can find no evidence of this in CVS though. Is it possible that sombody
had installed such an optimization in Adreas' Zope 2.6 installation?
If so, somebody needs to migrate that forward to the 2.7 installation.

*sigh* Jim, you're right. I verified with our CVS. Dieter added __getstate__
and __setstate__ to provide a more efficient pickle format. I was not aware
of this extension and since he is on vacation he could not give me a hint.

Sorry for bringing this up and thanks,
Andreas

Tim Peters

2003-11-04 18:25:08 UTC

[Tim Peters]

Post by Jim Fulton

Post by Tim Peters
This pickle creates a DateTime object, and then *tries* to set its
state from a tuple. That's not legal -- a tuple makes no sense here.

[Jim Fulton]

Post by Jim Fulton
That's a little missleading. The state can be anything. as long as
the class provides __setstate__ method that handles it.

In the full context there is no __setstate__ method for this class (else we
would not have gotten into the code trying to treat the state as a dict,
that being effectively the default __setstate__ implementation).

More, the DateTime instance ends up being attached to attribute
'lastRegistered', and there's no code in Zope or CMF that binds
lastRegistered to anything other than an instance of a DateTime imported via

from DateTime import DateTime

Post by Jim Fulton
Is it possible that sombody had installed such an optimization in Adreas'
Zope 2.6 installation?

Andreas? That's the only plausible explanation I can see -- Zope's native
DateTime class doesn't produce pickles like the one you sent.

Post by Jim Fulton
If so, somebody needs to migrate that forward to the 2.7 installation.

Tim Peters

2003-11-04 19:38:38 UTC

[Jim Fulton]

Post by Jim Fulton
It looks as though this pickle was written from a DateTime class with
custom __getstate__ and __setstate__. This would be useful to save
space. I can find no evidence of this in CVS though. Is it possible
that sombody had installed such an optimization in Adreas' Zope 2.6
installation? If so, somebody needs to migrate that forward to the
2.7 installation.

[Andreas Jung]

Post by Andreas Jung
*sigh* Jim, you're right. I verified with our CVS. Dieter added
__getstate__ and __setstate__ to provide a more efficient pickle
format. I was not aware of this extension and since he is on vacation
he could not give me a hint.
Sorry for bringing this up and thanks,

That's OK. I hope you port Dieter's changes to your 2.7 copy and keep
pursuing this: we're keenly interested to know whether the migration works
for you! If you give the 2.7 DateTime the same custom __setstate__ as you
had in 2.6, the pickle *should* load OK (all the pickle changes in 2.3 were
intended to be backward compatible, excepting cases where 2.3 was changed to
complain about an illegal pickle that 2.2 didn't catch -- like loading this
custom DateTime pickle in a Zope without the fiddled DateClass didn't
complain under 2.2.3, despite that it dropped the state on the floor
unused).

Andreas Jung

2003-11-07 16:15:48 UTC

Post by Tim Peters
[Andreas Jung]

That's OK. I hope you port Dieter's changes to your 2.7 copy and keep
pursuing this: we're keenly interested to know whether the migration
works for you!

No problems so far with Python 2.3.2 and Zope 2.7 with the migrated site.

-aj

Tim Peters

2003-11-07 16:33:40 UTC

[Tim Peters]