[Twisted-Python] Twisted-Python Digest, Vol 132, Issue 23

Tue May 5 16:17:18 MDT 2015

Glyph I just saw this response in my junk mail.  Thank you for the help, I can work with this.

Tj

-----Original Message-----
From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-bounces at twistedmatrix.com] On Behalf Of twisted-python-request at twistedmatrix.com
Sent: Friday, March 20, 2015 1:26 AM
To: twisted-python at twistedmatrix.com
Subject: Twisted-Python Digest, Vol 132, Issue 23

Send Twisted-Python mailing list submissions to
	twisted-python at twistedmatrix.com

To subscribe or unsubscribe via the World Wide Web, visit
	http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
or, via email, send a message with subject or body 'help' to
	twisted-python-request at twistedmatrix.com

You can reach the person managing the list at
	twisted-python-owner at twistedmatrix.com

When replying, please edit your Subject line so it is more specific than "Re: Contents of Twisted-Python digest..."

Today's Topics:

   1. custom FTP server is blocking (Paul Wiseman)
   2. Re: custom FTP server is blocking (Louis D. Burr)
   3. Re: __nonzero__ for Deferred (Glyph Lefkowitz)
   4. Re: Removing Python 2.6 Support after Twisted 15.1 (David Ripton)
   5. Re: twistechecker and twisted-dev-tools as part of	twisted
      main repo (Glyph)
   6. Re: twisted echo ssl client with .p12 (Glyph)

----------------------------------------------------------------------

Message: 1
Date: Thu, 19 Mar 2015 18:18:27 +0000
From: Paul Wiseman <poalman at gmail.com>
To: Twisted general discussion <twisted-python at twistedmatrix.com>
Subject: [Twisted-Python] custom FTP server is blocking
Message-ID:
	<CACgdh2i4n5CgwiOQ124YeM-Cyqp+3rMwHGFN7cnq5dt=C_Ne0A at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

I have some blocking going on in a custom twisted ftp server and I'm trying to pin point what exactly is doing it.

I think this line in openForReading on my FTPShell object could be the culprit:

_FileReader(urllib2.urlopen(result.link,timeout=settings.GLOBAL_TIMEOUT))

(_FileReader is twisted.protocols.ftp._FileReader and result.link a url string)

I'm guessing urllib2.urlopen blocks resolving the domain and connecting? What would be the twisted equivalent way of providing a file like object of the download stream to the FileReader that wouldn't block?

Regards,

Paul

------------------------------

Message: 2
Date: Thu, 19 Mar 2015 15:38:30 -0500
From: "Louis D. Burr" <ldanielburr at me.com>
To: Twisted general discussion <twisted-python at twistedmatrix.com>
Subject: Re: [Twisted-Python] custom FTP server is blocking
Message-ID: <3E9E76E8-E3D3-4905-86B1-0003192C2834 at me.com>
Content-Type: text/plain; charset=us-ascii

Hi Paul,

> On Mar 19, 2015, at 1:18 PM, Paul Wiseman <poalman at gmail.com> wrote:
> 
> I have some blocking going on in a custom twisted ftp server and I'm 
> trying to pin point what exactly is doing it.
> 
> I think this line in openForReading on my FTPShell object could be the culprit:
> 
> _FileReader(urllib2.urlopen(result.link,timeout=settings.GLOBAL_TIMEOU
> T))
> 

Yeah, urllib2.urlopen() blocks, so this is at least one place in your code that will cause you a problem.

> (_FileReader is twisted.protocols.ftp._FileReader and result.link a 
> url string)
> 
> I'm guessing urllib2.urlopen blocks resolving the domain and 
> connecting? What would be the twisted equivalent way of providing a 
> file like object of the download stream to the FileReader that 
> wouldn't block?
> 

You probably want to use twisted.web.client.Agent for this, as it allows you to perform non-blocking http requests.  If you can live with buffering the http response body into, say, a tempfile or an io.BytesIO instance, then you can just pass that to _FileReader as the fObj argument and call it a day.

Hope this helps,

--
L. Daniel Burr

------------------------------

Message: 3
Date: Thu, 19 Mar 2015 14:13:48 -0700
From: Glyph Lefkowitz <glyph at twistedmatrix.com>
To: Twisted general discussion <twisted-python at twistedmatrix.com>
Subject: Re: [Twisted-Python] __nonzero__ for Deferred
Message-ID: <18084B73-66AE-4F0A-857C-507D27537533 at twistedmatrix.com>
Content-Type: text/plain; charset=us-ascii

> On Mar 18, 2015, at 5:50 PM, exarkun at twistedmatrix.com wrote:
> 
> On 18 Mar, 10:03 pm, glyph at twistedmatrix.com wrote:
>> I'd like to deprecate and remove __nonzero__ on Deferreds. Occasionally I see code where someone (maybe even me) does "if d:" where they should have been doing "@d.addCallback def cb(result): if result: ...".
>> 
>> Anybody else have this issue?  Objections?
> 
> Er.  What?  It's a little early for april fools' day, isn't it?

> Seriously, this is weird.  Am I missing something?

Apparently?

> Can expand on what it would mean to deprecated a special method of a 
> class that doesn't even define it

Add such a method and cause it to emit a warning, then raise an exception.

> and what makes Deferred unique among all user-defined classes in this regard.

As I stated in my original message, there's a class of bug where someone does "if somethingThatReturnsDeferred()".  Particularly, "if checkPassword()" is a dangerous manifestation of this class of bug.  I'd like this to fail noisily instead of appearing to work.

In my opinion, having all user-defined classes implement true truthiness by default is a design error in the language.  It's a usually-harmless one except in a few cases, and this is one of those cases, which is why Deferred is not the same as the average user-defined class.

-glyph

------------------------------

Message: 4
Date: Thu, 19 Mar 2015 20:23:35 -0400
From: David Ripton <dripton at ripton.net>
To: twisted-python at twistedmatrix.com
Subject: Re: [Twisted-Python] Removing Python 2.6 Support after
	Twisted 15.1
Message-ID: <550B6887.4020703 at ripton.net>
Content-Type: text/plain; charset=utf-8

On 03/19/2015 08:05 AM, Phil Mayers wrote:
> On 18/03/15 23:57, Glyph Lefkowitz wrote:
> 
>> Rather than just suggest we preserve the status quo and stay on 2.6 
>> forever to do indefinite free work to support Red Hat's obsolescence
> 
> You *definitely* shouldn't do that. Push back on RedHat, and tell 
> customers to push back on RedHat. I say this to other projects.

Red Hat supports old versions of RHEL forever because they have customers that want that (because they have legacy apps that they want to keep working with minimal maintenance, but who still want security patches), and are willing to pay for it.  It's a good thing that someone provides that option, even if we don't all need it.

That doesn't mean Twisted needs to support old versions of RHEL forever.
 RHEL 7 has been out for a while, and comes with Python 2.7 as the default.  So even if Twisted drops support for Python 2.6, it'll still work on RHEL.  (Just not on old versions of RHEL.)

Furthermore, SCL makes Python 2.7 available (in /opt) on RHEL 5 and 6 and CentOS 6, without users needing to build it themselves.  So even those who are stuck with old versions can install SCL, then use pip to install new Twisted.

IOW, I wouldn't worry about it.  Drop support for Python 2.6 when it makes sense for Twisted.

-- 
David Ripton    dripton at ripton.net

------------------------------

Message: 5
Date: Thu, 19 Mar 2015 20:25:04 -0700
From: Glyph <glyph at twistedmatrix.com>
To: Twisted general discussion <twisted-python at twistedmatrix.com>
Subject: Re: [Twisted-Python] twistechecker and twisted-dev-tools as
	part of	twisted main repo
Message-ID: <790710C6-7461-4E4C-A796-381BD3ED2679 at twistedmatrix.com>
Content-Type: text/plain; charset=utf-8

> On Mar 19, 2015, at 8:41 AM, Christopher Armstrong <radix at twistedmatrix.com> wrote:
> 
> 
>> On Mar 19, 2015, at 4:45 AM, Adi Roiban <adi at roiban.ro> wrote:
>> 
>>> OK, cool.  I didn't realize that was ready for review.  I am liking 
>>> the process we have for Mimic for flagging PRs with keywords - 
>>> https://github.com/rackerlabs/mimic/pulls - but this needs 
>>> automation so as not to be unfriendly to non-committer contributors.
>>> 
>>> I'll try to get to it by the end of the week.
>> 
>> For twistedchecker I start using tags ... but I don't have 
>> permissions for twisted-infra repos.
>> 
>> Yes. GitHub PR are nice but horrible for implementing anything formal 
>> on top of it.
>> 
>> I was preparing to suggest someting similar also for Trac as 
>> sometimes I am not sure if a comment is 'do minor changes and merge' 
>> or 'do minor change and send them for review'
>> 
>> My suggestion is this:
>> 
>> when a PR is ready for review the author should leave a comment 
>> containing the `needs-review` marker. This will send a hook and a 
>> custom hook server will update the tags of the ticket.
>> 
> 
> 
> I suggest making the default, tagless state of a PR mean ?in review?, for a couple of reasons:
> 
> - having labels for both `needs-changes` and `in-review` means you can 
> enter an invalid state where both labels are applied
> - if the automation loses track of a PR, we don?t want people 
> filtering only for PRs with `label:in-review` and missing out on the 
> forgotten PR
> 
> so I propose tagless PRs be considered ?in review?, and a label only for `needs-changes`.

I like seeing the big green "ready for review" on Mimic PRs; it's more eye-catching.  We can have 2 labels though, so that if the automation breaks, we can have a periodic check that adds "in-review" to everything un-labeled.

-g

------------------------------

Message: 6
Date: Thu, 19 Mar 2015 22:25:10 -0700
From: Glyph <glyph at twistedmatrix.com>
To: Twisted general discussion <twisted-python at twistedmatrix.com>
Subject: Re: [Twisted-Python] twisted echo ssl client with .p12
Message-ID: <1AE7D177-6F83-4204-B088-3706F7E1B05F at twistedmatrix.com>
Content-Type: text/plain; charset="us-ascii"

> On Mar 19, 2015, at 8:36 AM, Louis D. Burr <ldanielburr at me.com> wrote:
> 
> Hi Timothy,
> 
>> On Mar 19, 2015, at 9:56 AM, Timothy Gallagher <timothy.gallagher at nuspire.com <mailto:timothy.gallagher at nuspire.com>> wrote:
>> 
>> Hello all,
>> I have a project that requires client server with ssl/tls including client certificate authentication.  Also the a requirement is that the client needs to use a .p12 file to house its keys.  I have the server part and client part down except I cannot find any code examples using a .p12 file to get the certificates.  Can this be done without having to hack into the ssl.ClientConextFactory?
> 
> Maybe http://stackoverflow.com/questions/6345786/python-reading-a-pkcs12-certificate-with-pyopenssl-crypto <http://stackoverflow.com/questions/6345786/python-reading-a-pkcs12-certificate-with-pyopenssl-crypto> will be useful to you.  Twisted uses pyopenssl under the covers, so the solution exarkun posted to StackOverflow should be applicable.
> 
> Hope this helps,
> 
> - L. Daniel Burr

You definitely shouldn't use ssl.ClientContextFactory.  It doesn't verify certificates, or provide any authentication of the server.  We should really remove and deprecate it :-\.

You should use ssl.optionsForClientTLS, and you should build it like this:

import getpass

from OpenSSL.crypto import load_pkcs12
from twisted.internet.ssl import (
    PrivateCertificate, KeyPair, Certificate, optionsForClientTLS
)

from twisted.internet.protocol import Factory, Protocol from twisted.internet.endpoints import SSL4ClientEndpoint from twisted.internet.defer import inlineCallbacks, Deferred from twisted.internet.task import react

@inlineCallbacks
def main(reactor, p12file, host, port=443):
    host = host.decode("utf-8")
    port = int(port)
    with open(p12file) as f:
        pkcs12 = load_pkcs12(f.read(), getpass.getpass())
        publicCertificate = Certificate(pkcs12.get_certificate())
        privateKey = KeyPair(pkcs12.get_privatekey())
        privateCertificate = PrivateCertificate.fromCertificateAndKeyPair(
            publicCertificate, privateKey
        )
    contextFactory = optionsForClientTLS(host,
                                         clientCertificate=privateCertificate)
    endpoint = SSL4ClientEndpoint(reactor, host, port, contextFactory)
    x = Deferred()
    class it(Protocol, object):
        def connectionMade(self):
            self.transport.write(b"GET / HTTP/1.1\r\n\r\n")
        def dataReceived(self, data):
            x.callback(Certificate.peerFromTransport(self.transport))
            self.transport.abortConnection()
    yield endpoint.connect(Factory.forProtocol(it))
    cert = yield x
    print(cert)

from sys import argv
react(main, argv[1:])

Hopefully that's a pretty complete answer :-).

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20150319/5d5edec5/attachment.html>

------------------------------

_______________________________________________
Twisted-Python mailing list
Twisted-Python at twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

End of Twisted-Python Digest, Vol 132, Issue 23
***********************************************