[Twisted-Python] Backup Agent
Frankie Chu
frank at rollingegg.net
Tue Feb 17 06:08:04 MST 2004
On 17 Feb 04, at 10:21 AM, twisted-python-request at twistedmatrix.com
wrote:
> Message: 1
> Date: Mon, 16 Feb 2004 10:03:37 -0500
> To: twisted-python at twistedmatrix.com
> From: "David A. Leedom" <daleedom at hightowergroup.com>
> Subject: [Twisted-Python] Backup Agent
> Reply-To: twisted-python at twistedmatrix.com
>
> --=====================_580045343==.ALT
> Content-Type: text/plain; charset="us-ascii"; format=flowed
>
> OK Guys I Need your help.
>
> I own a small computer consulting firm and am struggling with the basic
> sales and marketing issues. However, I think I have finally stumbled
> onto
> a great "no brainer" idea:
>
> A Remote Backup Service for Small businesses.
>
> Yes I know there are a bunch out their, but I am thinking of the
> Hundreds
> of Businesses around my general geographic area who don't know how to
> find
> them and who don't backup their data. I would charge a monthly fee
> for a
> given amount of space used ( no more than 500 Meg) and provide a
> software
> that would backup and restore their data.
>
> My dilemma is that my margins would be to small to make it affordable
> for
> me to resale someone else's software. So....I would like to roll my
> own. Python makes a lot of sense for many reasons that I don't need
> to get
> into here.
>
> My struggle is that although I have been programming since the
> mid-70s, I
> am new to Python and feel like a man groping around in the dark. Also
> being new to Twisted it is like the room is full of chairs as well. I
> did
> get some sample code to work over the weekend, but I got a serious
> brain
> cramp trying to figure it all out.
>
> The basic architecture that I am thinking of is client/server. There
> would
> be three basic applications running to make things work. First, would
> be
> an Agent program that does most of the work. The Agent would run as a
> service on each computer being backed up. Second would be a GUI
> program
> running locally to control the agents. Finally there would be a "Web
> Service" that received backup sets from each agent or a designated
> master
> agent. The Web Service would handle authentication and other house
> keeping
> things.
>
> Finally the Question!!!!
> There are many other details that I have in mind as well (using
> patches to
> reduce size, compression, encryption...), but right now I am working on
> issues related to the communication between the GUI and Agents.
> Assuming
> that I have one or more number of agents running on a network I want
> to do
> a number of things. 1) Have the GUI request that the agents identify
> themselves so I don't have to tell the GUI where they are. 2)
> Transmit a
> catalog list of the file system to the GUI so I can build a TreeView
> and
> pick which files to backup on each computer. 3) Finally I may want to
> identify a "Master" agent that collects all the backups on the network
> and
> transmits them as one session to the Web Service. This would allow for
> local "On Site" backups for fast restores due to normal computer
> crashes
> and failures.
>
> 1. What would you guys suggest I do to I get the GUI to ask all
> the
> agents to identify themselves?
> 2. What would be the best approach using twisted to transmit a
> file
> system catalog across the local wire. I was playing with some remote
> procedure calls over the weekend that seemed to work. I am thinking I
> could create a class that would gather the local file system and send
> it
> over the wire as one lump.
> 3. If I am going to transmit a backup file (<500meg) over the
> wire to
> another local computer what is the best Twisted approach to take? Due
> to
> WinXP security issues and the desire the run in a cross platform
> environment I don't want to rely on normal file shares.
>
> There are a hundred other questions running though my mind right now,
> but
> this will due for now,
>
>
> Thanks for any help you can give,
>
> David A. Leedom
>
>
>
>
> The Hightower Group, Inc.
> Custom Software Solutions Designed To Fit Your Business Like A Glove.
> 165 West Airport Road/Lititz, PA 17543
> V:717-560-4002, 877-560-4002 x: 114
> F:717-560-2825
> www.hightowergroup.com
> --=====================_580045343==.ALT
> Content-Type: text/html; charset="us-ascii"
>
> <html>
> <body>
> OK Guys I Need your help.<br><br>
> I own a small computer consulting firm and am struggling with the basic
> sales and marketing issues. However, I think I have finally
> stumbled onto a great "no brainer" idea: <br><br>
> <x-tab> </x-tab>A Remote
> Backup Service for Small businesses. <br><br>
> Yes I know there are a bunch out their, but I am thinking of the
> Hundreds
> of Businesses around my general geographic area who don't know how to
> find them and who don't backup their data. I would charge a
> monthly
> fee for a given amount of space used ( no more than 500 Meg) and
> provide
> a software that would backup and restore their data.<br><br>
> My dilemma is that my margins would be to small to make it affordable
> for
> me to resale someone else's software. So....I would like to roll
> my
> own. Python makes a lot of sense for many reasons that I don't
> need
> to get into here.<br><br>
> My struggle is that although I have been programming since the
> mid-70s, I
> am new to Python and feel like a man groping around in the dark.
> Also being new to Twisted it is like the room is full of chairs as
> well. I did get some sample code to work over the weekend, but I
> got a serious brain cramp trying to figure it all out.<br><br>
> The basic architecture that I am thinking of is client/server.
> There would be three basic applications running to make things
> work. First, would be an Agent program that does most of the
> work. The Agent would run as a service on each computer being
> backed up. Second would be a GUI program running locally to
> control
> the agents. Finally there would be a "Web Service" that
> received backup sets from each agent or a designated master agent. The
> Web Service would handle authentication and other house keeping
> things.<br><br>
> <b>Finally the Question!!!!<br>
> </b>There are many other details that I have in mind as well (using
> patches to reduce size, compression, encryption...), but right now I am
> working on issues related to the communication between the GUI and
> Agents. Assuming that I have one or more number of agents running
> on a network I want to do a number of things. 1) Have the
> GUI
> request that the agents identify themselves so I don't have to tell the
> GUI where they are. 2) Transmit a catalog list of the file system
> to the GUI so I can build a TreeView and pick which files to backup on
> each computer. 3) Finally I may want to identify a
> "Master" agent that collects all the backups on the network
> and
> transmits them as one session to the Web Service. This would allow for
> local "On Site" backups for fast restores due to normal
> computer crashes and failures. <br><br>
> 1.<x-tab> </x-tab>What would you
> guys
> suggest I do to I get the GUI to ask all the agents to identify
> themselves?<br>
> 2.<x-tab> </x-tab>What would be the
> best approach using twisted to transmit a file system catalog across
> the
> local wire. I was playing with some remote procedure calls over
> the
> weekend that seemed to work. I am thinking I could create a class
> that would gather the local file system and send it over the wire as
> one
> lump. <br>
> 3.<x-tab> </x-tab>If I am going to
> transmit a backup file (<500meg) over the wire to another local
> computer what is the best Twisted approach to take? Due to WinXP
> security issues and the desire the run in a cross platform environment
> I
> don't want to rely on normal file shares.<br><br>
> There are a hundred other questions running though my mind right now,
> but
> this will due for now,<br><br>
> <br>
> Thanks for any help you can give,<br><br>
> David A. Leedom<br><br>
> <br><br>
> <x-sigsep><p></x-sigsep>
> The Hightower Group, Inc.<br>
> Custom Software Solutions Designed To Fit Your Business Like A
> Glove.<br>
> 165 West Airport Road/Lititz, PA 17543<br>
> V:717-560-4002, 877-560-4002 x: 114<br>
> F:717-560-2825<br>
> <a href="http://www.hightowergroup.com/"
> eudora="autourl">www.hightowergroup.com</a>
> </body>
> </html>
>
> --=====================_580045343==.ALT--
>
>
>
> --__--__--
>
> Message: 2
> Date: Mon, 16 Feb 2004 18:23:03 +0100
> From: Michal Pasternak <michal at pasternak.w.lub.pl>
> To: twisted-python at twistedmatrix.com
> Subject: Re: [Twisted-Python] Backup Agent
> Reply-To: twisted-python at twistedmatrix.com
>
> David A. Leedom [Mon, Feb 16, 2004 at 10:03:37AM -0500]:
>> 1. What would you guys suggest I do to I get the GUI to ask all
>> the
>> agents to identify themselves?
>
> Well, that's a good question. In some app I did, I used machine's IP
> address
> + some operating system info it run (sys.platform, os.uname) to build
> a kind
> of machine identifier. You could as well generate some UID, then save
> it,
> then send it together with machine's IP to identify it on server, by
> hand.
>
> Using OS-specific procedures could help you with this.
>
>> 2. What would be the best approach using twisted to transmit a
>> file
>> system catalog across the local wire. I was playing with some remote
>> procedure calls over the weekend that seemed to work. I am thinking I
>> could create a class that would gather the local file system and send
>> it
>> over the wire as one lump.
>
> Remote procedures have limit of 640 KB for their parameters (because
> "640 KB
> should be enough to anyone"). You should use Pager and Collector.
>
>> 3. If I am going to transmit a backup file (<500meg) over the
>> wire to
>> another local computer what is the best Twisted approach to take?
>
> See above. Also, please remember, that memory issues matter. If you're
> paging the data (send small chunks), you should save them on the backup
> server, as they are received. If you are transfering a big file,
> keeping all
> the data received in some buffer is a bad idea.
>
>
> --__--__--
>
> Message: 3
> Date: Mon, 16 Feb 2004 13:00:47 -0800
> To: Twisted <twisted-python at twistedmatrix.com>
> From: stephan <mailinglists at shechen.at>
> Subject: [Twisted-Python] p2p, gnutella
> Reply-To: twisted-python at twistedmatrix.com
>
>
>
> I am looking forward to add p2p functionality to my app. Does anybody
> know
> what the current status of twisted's gnutella implementation is?
>
> There seems to have been a semi-finished implemention in 1.0.2alpha4
> which
> I can't find in 1.1.1 anymore.
>
> I might also be willing to implement missing parts but I would need to
> get
> a short briefing on how things are.
>
> thanks,
>
> _stephan
>
>
> --__--__--
>
> Message: 4
> Date: Mon, 16 Feb 2004 18:13:13 -0500
> To: peak at eby-sarna.com
> From: "Phillip J. Eby" <pje at telecommunity.com>
> Cc: zope3-dev at zope.org, twisted-python at twistedmatrix.com
> Subject: [Twisted-Python] PEAK 0.5a3 released
> Reply-To: twisted-python at twistedmatrix.com
>
> After several months of feature creep, the third alpha release of PEAK
> 0.5
> is now available for download at:
>
> http://peak.telecommunity.com/dist/
>
> Some highlights of the changes since 0.5a2 include:
>
> * A simplified and consolidated binding API: there are now only four
> kinds
> of bindings (Make, Obtain, Require, and Delegate) that provide a wider
> variety of functionality than the old API. You can also now create
> your
> own IRecipe or IComponentKey implementations for use with Make and
> Obtain.
>
> * The 'n2' (namespace navigator) tool provides a handy, extensible
> shell
> (with completion and history) for exploring PEAK naming systems and SQL
> connections.
>
> * A greatly enhanced configuration file parser, with lots of new
> convenience features such as [Import on Demand], [Named Services], and
> [Component Factories] sections, based on its new extensible parsing
> framework. You can define new section types of your own, even from
> inside
> an .ini file.
>
> * New configuration key features, such as multi-keys (similar to some
> of
> Zope X3's "adaptergeddon" features), the ability to iterate over
> defined
> keys in a hierarchy (useful for finding/registering various kinds of
> "plugins"), new convenience features for defining wildcard property
> rules,
>
> * A new 'peak.events' framework that supports event sources, listeners,
> subscriptions, generator-based pseudothreads ("tasks"), scheduling,
> and I/O
> events. Using 'peak.events', you can write event-driven code in a more
> natural, sequential, "untwisted" style, but without giving up access to
> Twisted's many great features. 'peak.events' can use PEAK's built-in
> event
> loop, or adapt a Twisted reactor for use as an event loop. You can
> also
> use Twisted's "Deferred" objects as event sources, which means you can
> use
> them in your Tasks (pseudothreads).
>
> * A new 'peak.ddt' "document-driven testing" framework, similar in
> form and
> function to the FIT ( http://fit.c2.com/ ) framework, but in Python and
> well-integrated with the rest of PEAK. 'peak.ddt' lets you write test
> data
> as tables in HTML documents (created with Word, Open Office, Mozilla,
> or
> virtually any other tool), and then view the results in your
> browser. Correct results are highlighted in green, incorrect answers
> are
> highlighted in red, and errors are highlighted in yellow, with a
> small-font
> traceback added to the table cell. It's a great way to do data-driven
> acceptance testing, or to manage a project's progress using example
> data as
> part of a requirements document.
>
> * Many, many more features, both large and small, like a pre-forking
> multiprocess supervisor for FastCGI and other forking servers. Quick
> access to documentation via the 'peak help' command, which uses pydoc
> to
> display reference documentation. Early versions of the 'peak.security'
> (access control), 'peak.web' (Zope X3-based web publishing), and
> 'peak.query' (declarative relational and conceptual queries w/SQL
> generation) packages. The 'whenImported()' function that lets you
> call a
> hook when a named module is first used. "Mock" DBAPI and socket
> implementations for testing. Hooks for "thunking" database-specific
> SQL
> and functions. And many, many more examples of unbelievably blatant
> feature creep!
>
> There is also now a nice "Hello World" introduction to PEAK available
> at:
>
> http://peak.telecommunity.com/DevCenter/IntroToPeak
>
> that presents a guided tour of some of PEAK's "vertical" frameworks,
> like
> command-line application support, database access, and so on.
>
> For more information about PEAK, and access to community resources
> including documentation, the Wiki, mailing list archives, IRC channel
> logs,
> CVS access, and more, please visit:
>
> http://peak.telecommunity.com/
>
>
>
> --__--__--
>
> Message: 5
> To: twisted-python at twistedmatrix.com
> From: ralf at brainbot.com
> Date: Tue, 17 Feb 2004 02:05:41 +0100
> Subject: [Twisted-Python] [PATCH] nasty bug in
> twisted.names.client.createResolver
> Reply-To: twisted-python at twistedmatrix.com
>
> The parameter variable 'hosts' gets overwritten by the import statement
> in createResolver. hosts.Resolver constructor is called with a module
> instead of a filename.
>
> - Ralf
>
> ===================================================================
> RCS file: /cvs/Twisted/twisted/names/client.py,v
> retrieving revision 1.45
> diff -u -r1.45 client.py
> --- twisted/names/client.py 4 Feb 2004 19:55:34 -0000 1.45
> +++ twisted/names/client.py 17 Feb 2004 01:01:53 -0000
> @@ -303,20 +303,20 @@
> return p
>
>
> -def createResolver(servers = None, resolvconf = None, hosts = None):
> +def createResolver(servers = None, resolvconf = None, hostsfile =
> None):
> import resolve, cache, hosts, root
> if platform.getType() == 'posix':
> if resolvconf is None:
> resolvconf = '/etc/resolv.conf'
> - if hosts is None:
> - hosts = '/etc/hosts'
> + if hostsfile is None:
> + hostsfile = '/etc/hosts'
> theResolver = Resolver(resolvconf, servers)
> - hostResolver = hosts.Resolver(hosts)
> + hostResolver = hosts.Resolver(hostsfile)
> else:
> - if hosts is None:
> - hosts = r'c:\windows\hosts'
> + if hostsfile is None:
> + hostsfile = r'c:\windows\hosts'
> bootstrap = ThreadedResolver()
> - hostResolver = hosts.Resolver(hosts)
> + hostResolver = hosts.Resolver(hostsfile)
> theResolver = root.bootstrap(bootstrap)
>
> L = [hostResolver, cache.CacheResolver(), theResolver]
>
> --
> brainbot technologies ag
> boppstrasse 64 . 55118 mainz . germany
> fon +49 6131 211639-1 . fax +49 6131 211639-2
> http://brainbot.com/ mailto:ralf at brainbot.com
>
>
> --__--__--
>
> Message: 6
> Subject: Re: [Twisted-Python] [PATCH] nasty bug in
> twisted.names.client.createResolver
> From: Itamar Shtull-Trauring <itamar at itamarst.org>
> To: twisted-python at twistedmatrix.com
> Organization: http://itamarst.org
> Date: Mon, 16 Feb 2004 20:33:55 -0500
> Reply-To: twisted-python at twistedmatrix.com
>
> On Mon, 2004-02-16 at 20:05, ralf at brainbot.com wrote:
>> The parameter variable 'hosts' gets overwritten by the import
>> statement
>> in createResolver. hosts.Resolver constructor is called with a module
>> instead of a filename.
>
> Please add patches and bugs to twistedmatrix.com/bugs/
>
>
>
> --__--__--
>
> Message: 7
> Subject: Re: [Twisted-Python] Advice on doing thousands of
> simultaneous UDP
> queries with Twisted...
> From: Itamar Shtull-Trauring <itamar at itamarst.org>
> To: twisted-python at twistedmatrix.com
> Organization: http://itamarst.org
> Date: Mon, 16 Feb 2004 20:40:06 -0500
> Reply-To: twisted-python at twistedmatrix.com
>
> On Mon, 2004-02-09 at 05:51, Mike C. Fletcher wrote:
>
>> Writing a simple asynchronous loop myself (poll on a simple socket,
>> send
>> message from queue when writable, read one into other queue when
>> readable) allowed for doing a few thousand queries simultaneously),
>> with
>> only a few dozen dropped messages. However, with the Twisted
>> equivalent
>> (UDPTransport with my simple protocol object), I was seeing huge drop
>> rates, so, gathering that Twisted isn't queueing up the UDP requests,
>
> Possibly we need some generic UDP queuing, rather than current system.
> Patches are welcome.
>
> --
> Itamar Shtull-Trauring http://itamarst.org
> Looking for a job: http://itamarst.org/resume.html
>
>
>
> --__--__--
>
> Message: 8
> To: twisted-python at twistedmatrix.com
> From: ralf at brainbot.com
> Date: Tue, 17 Feb 2004 02:47:19 +0100
> Subject: [Twisted-Python] [PATCH] twisted.names.hosts not working
> correct with getHostByName
> Reply-To: twisted-python at twistedmatrix.com
>
> Even with my prior patch I'm not able to use
> twisted.names.client.getHostByName.
> Seems like twisted.names.hosts.Resolver calls itself with bad
> parameters
> and even doesn't use the ip address it extracted from /etc/hosts. Also
> the return
> value could not be handled by the code in
> twisted.names.common.extractRecord.
> The following patch seems to solve those problems.
>
> - Ralf
>
>
>
>
> ===================================================================
> RCS file: /cvs/Twisted/twisted/names/hosts.py,v
> retrieving revision 1.14
> diff -u -r1.14 hosts.py
> --- twisted/names/hosts.py 19 Jul 2003 10:13:50 -0000 1.14
> +++ twisted/names/hosts.py 17 Feb 2004 01:45:44 -0000
> @@ -49,11 +49,11 @@
>
>
> def query(self, query, timeout = None):
> - return self._lookup(str(query.name), query.type, query.type,
> timeout)
> + return self._lookup(str(query.name), query.cls, query.type,
> timeout)
>
>
> def _lookup(self, name, cls, type, timeout):
> - if cls != dns.IN or type != dns.A:
> + if cls != dns.IN or (type != dns.A and type !=
> dns.ALL_RECORDS): # getHostByName passes type==dns.ALL_RECORDS. what
> else do we need here?
> return
> defer.fail(failure.Failure(NotImplementedError(type, cls)))
> return self.lookupAddress(
> name, timeout
> @@ -64,7 +64,7 @@
> res = searchFileFor(self.file, name)
> if res:
> return defer.succeed([
> - [dns.RRHeader(name, dns.A, dns.IN, self.ttl)], (), ()
> + [dns.RRHeader(name, dns.A, dns.IN, self.ttl,
> dns.Record_A(res, self.ttl))], [], []
> ])
> return defer.fail(failure.Failure(dns.DomainError(name)))
>
> --
> brainbot technologies ag
> boppstrasse 64 . 55118 mainz . germany
> fon +49 6131 211639-1 . fax +49 6131 211639-2
> http://brainbot.com/ mailto:ralf at brainbot.com
>
>
> --__--__--
>
> Message: 9
> Date: Mon, 16 Feb 2004 22:47:47 -0300
> From: Daniel Henrique Debonzi <debonzi at gwyddion.com>
> To: twisted-python at twistedmatrix.com
> Subject: [Twisted-Python] Client Prompt
> Reply-To: twisted-python at twistedmatrix.com
>
> Hi everybody,
>
> Maybe this is a very stupid question, but I realy couldn't find a way
> to
> solve it by myself.
>
> Would be enough for my aplication if this example works.
>
> So, I want to have a client that read a string from the stdin and then
> send it to the server. Wich changes should I do in this code to this
> client work like a discrebed
>
> #################################################################
> from twisted.internet import reactor, protocol
>
> class EchoClient(protocol.Protocol):
>
> def connectionMade(self):
> """I don't want to write it here
> I want to take data from stdin and send o server,
> but were and how to do it? Too much dificult?
> """
> self.transport.write("hello, world!")
>
> def dataReceived(self, data):
> print data
>
> def connectionLost(self, reason):
> print "connection lost"
> from twisted.internet import reactor
> reactor.stop()
>
>
> class EchoFactory(protocol.ClientFactory):
> protocol = EchoClient
>
> def clientConnectionFailed(self, connector, reason):
> print "Connection failed - goodbye!"
> reactor.stop()
>
> def clientConnectionLost(self, connector, reason):
> print "Connection lost - goodbye!"
> reactor.stop()
>
>
> def main():
> f = EchoFactory()
> reactor.connectTCP("localhost", 5555, f)
> reactor.run()
>
> if __name__ == '__main__':
> main()
>
> #################################################################
>
> Thanks for all
>
>
>
> --
> Daniel Henrique Debonzi Linux user number 166903
> Gwyddion Embedded Free-Software Development
> www.gwyddion.com
> São Carlos, São Paulo
> Brazil
>
>
>
> --__--__--
>
> Message: 10
> To: twisted-python at twistedmatrix.com
> Subject: Re: [Twisted-Python] [PATCH] nasty bug in
> twisted.names.client.createResolver
> From: ralf at brainbot.com
> Date: Tue, 17 Feb 2004 03:20:28 +0100
> Reply-To: twisted-python at twistedmatrix.com
>
> Itamar Shtull-Trauring <itamar at itamarst.org> writes:
>
>> On Mon, 2004-02-16 at 20:05, ralf at brainbot.com wrote:
>>> The parameter variable 'hosts' gets overwritten by the import
>>> statement
>>> in createResolver. hosts.Resolver constructor is called with a module
>>> instead of a filename.
>>
>> Please add patches and bugs to twistedmatrix.com/bugs/
>>
>
> Lots of bug reports in the bug tracker have status 'unread'. That's
> not very motivating. Just committing this patch -even if it was wrong-
> would not make things worse, as the original code is definitely
> broken. But, I'll add it to the bugtracker tomorrow.
>
> - Ralf
>
> --
> brainbot technologies ag
> boppstrasse 64 . 55118 mainz . germany
> fon +49 6131 211639-1 . fax +49 6131 211639-2
> http://brainbot.com/ mailto:ralf at brainbot.com
>
>
>
> --__--__--
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> End of Twisted-Python Digest
More information about the Twisted-Python
mailing list