[Twisted-Python] Seeking OSS 'point to point' data transfer solution

Worsham, Michael michael.worsham at morris.com
Wed Jan 21 08:50:56 EST 2009


I was recommended to look at the Twisted application framework for
replacing a legacy application that relies on RPC portmapper, so here's
my issue and question...

 

Currently, we use an in-house designed application called 'Fileman',
which acts as a combined bi-directional shared library (client) and
server system to transfer data files between other 'Fileman' server
points. The problem is that the Fileman application requires portmapper
under Linux which exposes all TCP ports (i.e. a network security
nightmare). Fileman also has the ability to send data in either ASCII or
binary format and at receiver end -- change directories, modify user id
(uid) and group id (gid) of file once data is received.

---

Here is what Fileman can perform via following functions from client
side code:

struct StatResult   *pFileMan_FileStat(char *host, char *file, int
userid, int groupid);
struct StatResult   *pFileMan_GetDirectoryListing(char *host, char *dir,
int userid, int groupid, int *entries);
int                 iFileMan_PutAsciiFile(char *host, char *file, char
*cpBuf, int bMakeDir, int userid, int groupid);
char                *cpFileMan_GetAsciiFile(char *host, char *file, int
userid, int groupid, int *result);
int                 iFileMan_PutBinaryFile(char *host, char *file, char
*cpBuf, int bMakeDir, int userid, int groupid, int length);
char                *cpFileMan_GetBinaryFile(char *host, char *file, int
userid, int groupid, int *result, int *length);
int                 iFileMan_DeleteFile(char *host, char *file, int
userid, int groupid);
int                 bFileMan_Ping(char *cpHost);
void                vFileMan_ResetStats(void);
int                 iFileMan_GetNumOps();
double              dFileMan_GetSecsSpent();
void                vFileMan_SetTimeout(int iSecs);

Above functions translates to 'over the wire' execution of:
stat
opendir, readdir
fopen, fwrite with any necessary mkdir, chown/chgrp to given group
fopen, read, subject to uid/gid restrictions
fopen, fwrite with any necessary mkdir, chown/chgrp to given group
fopen, read, subject to uid/gid restrictions
unlink subject to uid/gid restrictions
noop response from server
Internal stat change
Internal stat info
Internal stat info
Set network timeout parameters for operation

 

Usage/Syntax from 'Fileman' client:

 

void printUsage()

{

    printf("Usage: \n");

    printf("  \n");

    printf(" fileman Get [remote_host] [remote_file] [remote_uid]
[remote_gid] [local_file] \n");

    printf(" fileman GetAscii [remote_host] [remote_file] [remote_uid]
[remote_gid] [local_file] \n");

    printf(" fileman GetBinary [remote_host] [remote_file] [remote_uid]
[remote_gid] [local_file] \n");

    printf(" fileman Put [remote_host] [remote_file] [remote_uid]
[remote_gid] [local_file] \n");

    printf(" fileman PutAscii [remote_host] [remote_file] [remote_uid]
[remote_gid] [local_file] \n");

    printf(" fileman PutBinary [remote_host] [remote_file] [remote_uid]
[remote_gid] [local_file] \n");

    printf(" fileman Stat [remote_host] [remote_file] [remote_uid]
[remote_gid] \n");

    printf(" fileman Ping [remote_host] \n");

    printf("  \n");

}


---

My question is can the Twisted framework be utilized in a way that can
do actions of sending and receiving data in either ASCII or binary
format, change uid or gid permissions of files once delivered, run as a
client/server client & listener and utilize one (or two) TCP ports for
both sending and listening upon?

Known application replacement requirements:
1) Application must be able to run upon RHEL/CentOS 5.x Linux platform.
2) Application must be able to be locked down to one or two TCP ports
for listening and sending data upon.
3) Application must be able to take command line (CLI) instructions
4) Application must be able to transfer data in ASCII or binary format
(i.e. graphic images)
5) Application must be able to [at receiver end]: change directories,
modify user id (uid) and group id (gid) of file once data is received.

Current solutions we are actively looking at now:
 -- Apache's WebDAV: Uses port 80; Needs apache instance on each server;
cannot change uid or gid permissions of files
 -- Apache's ActiveMQ: Uses channel/queue/broker system; requires
trigger queue and external program for dumping queue data; Needs Java
platform
 -- JBoss Messaging: Suffers same issues as Apache's ActiveMQ; Requires
costly licensing
 -- IBM Websphere MQ: Suffers same issues as Apache's ActiveMQ; Requires
costly licensing

 -- XML-RPC Library for C & C++ -- http://xmlrpc-c.sourceforge.net/


Anyone have any ideas or other plausible solutions for solving this
problem?

 

-- Michael

 

---
Michael Worsham
Sr. System Engineer, IT Architect
Morris Technology Services, IHS Division

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090121/b1becc80/attachment.htm 


More information about the Twisted-Python mailing list