[Twisted-Python] Twisted plugins and py2exe

[Gabriel Rossetti]

glyph at divmod.com wrote:
>
> On 04:55 pm, exarkun at divmod.com wrote:
>> Tue, 24 Feb 2009 17:35:14 +0100, Gabriel Rossetti 
>> <gabriel.rossetti at arimaz.com> wrote:
>>> Jean-Paul Calderone wrote:
>>>> On Tue, 24 Feb 2009 11:49:13 +0100, Gabriel Rossetti 
>>>> <gabriel.rossetti at arimaz.com> wrote:
>
>>>>> Has anyone ever had this problem while using plugins with Twisted 
>>>>> and py2exe?
>
>>>> No, but from the traceback, it seems fairly clear that the plugin 
>>>> system
>>>> is trying to write a plugin cache file into your py2exe-created zip 
>>>> file
>>>> of Python source.
>
>>> Ok, I could do this, teach it abot zip files and have it update it's 
>>> cache in the zip file, but in certain cases, like mine actually, if 
>>> you tell py2exe to include the zip file in the exec, then it won't 
>>> work. How about adding the possibility to specify where the cache is 
>>> to be written?
>
> This is almost certainly the wrong solution to your problem, but if 
> you disagree with this assessment please feel free to update this ticket:
>
>    http://twistedmatrix.com/trac/ticket/3348
>
> Several people have requested this feature and nobody has really made 
> a good case for why it should be implemented.  (I don't think it 
> should be, I just think we should record all the reasons why not on 
> that ticket :)).
>> Updating the cache in the zip file probably isn't the right thing to do.
>> Just skipping cache generation would probably make more sense.  It 
>> should
>> be easier to implement, anyway.
>
> Skipping the cache generation is like skipping bytecode compilation. 
> It's not a catastrophic error, everything should still work, but it 
> will result in more unnecessary work being done at runtime.
>
> IMHO the right solution in this particular case would be to get py2exe 
> (or some part of the setup process in py2exe, which I believe is 
> implemented using distutils) to generate a cache file as part of the 
> built zip file.  Presumably .pyc files are also created and included 
> in the zip file.  The appropriate procedure for forcibly generating 
> the cache is described here:
>
> http://twistedmatrix.com/projects/core/documentation/howto/plugin.html#auto3 
>
>
> A cache generated in this manner will be suitable for packaging into a 
> zip file.
Sorry for reviving this old thread now, but I found a way of doing it 
with almost no modifications to the way things are currently done. I 
just made twisted.python.zippath.ZipPath.open have the same signature as 
twisted.python.filepath.FilePath (see my patch) and then added the 
following to my py2exe setup file:

from py2exe.build_exe import py2exe as BuildExe
from twisted.plugin import getCache

class PluginCacheCollector(BuildExe):
    def copy_extensions(self, extensions):
        BuildExe.copy_extensions(self, extensions)
       
        # Import the plugin packages
        from mypackage.plugins.io import myioplugins
        from mypackage.plugins.misc import myotherplugins
        mods = [ myioplugins, myotherplugins ]
       
        for m in mods:
           
            # Pre-gen the plugin cache
            getCache(m)
           
            # Build the cache file's path in the build collect dir and 
copy the cache files there
            f = os.path.join(*(m.__name__.split('.') + ["dropin.cache"]))
            full = os.path.join(self.collect_dir, f)
            self.copy_file(f, full)
           
            # Add the cache file path to the list of files to be added 
to the py2exe zip file
            self.compiled_files.append(f)

and add this 'cmdclass={"py2exe": PluginCacheCollector}' to setup() like so:

opts = {
    "py2exe": {
        "packages": [ "mypackage" ],
        "includes": [ "myincludes" ],
        "excludes": [ "curses", "Tkinter", "Tkconstants", "doctest", 
"pdb", "unittest", "difflib", "pyreadline", "optparse", "calendar", 
"tcl", "pywin.debugger", "pywin.debugger.dbgcon",
                      "pywin.dialogs", "_gtkagg", "_tkagg" ],
        "dll_excludes": ["libgdk-win32-2.0-0.dll", 
"libgobject-2.0-0.dll", "tcl84.dll", "tk84.dll"],
        "dist_dir": "dist",
        "optimize": 2, # Use -OO when building (e.g. python -OO setup.py 
py2exe)
        "bundle_files": 1,
        "compressed": True,
    }
}

setup(  
    console=['mymain.py'],
    zipfile="library.zip",
    options=opts,
    data_files=[("icons", glob.glob("icons/*.*"))],
    cmdclass={"py2exe": PluginCacheCollector}, # <----------- add this
    )

And that does the trick, it will generate the cache files and they will 
be copied to the collect dir, then they will get added to the zip. What 
do you think?

Gabriel
>
> Another part of the solution would be to implement setContent on 
> ZipPath.  However, there are still likely to be erroneous deployment 
> scenarios where the ZipPath is not available for writing, just as an 
> installation path is frequently not available for writing now.  So an 
> initial implementation of setContent on ZipPath could just raise the 
> same exception that an unwritable FilePath would, for consistency in 
> error handling.
>
> For the issue of quieting the mostly-harmless error message now 
> produced by a failure to write the cache file, see here:
>
>    http://twistedmatrix.com/trac/ticket/2409
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>