Linux - Installing and configuring Proftpd with SFTP (mod_sftp)

I recently upgraded Proftpd on a CentOS box and added mod_sftp as a static module... This post will explain what the hell I have just said, and also, how to set it setup and running. I will include gotcha's (plenty!) and some general helpful advice along the way.

We will be building from source in this guide, so I recommend grabbing that one from the download site. Download Proftpd from their website Here. Take ProFTPD 1.3.3rc1 or later as this includes mod_sftp.


Prerequisites

1. You will need to install a copy of the OpenSSL source, if you haven't already. If you have it installed, check it's in your PATH. When compiling Proftpd, it requires some header files from the OpenSSL-dev release, so this is a must!

You can download OpenSSL from Here. Grab the latest non-beta if its going to go into production. That's the rule!

Here is a guide on installing OpenSSL


2. You also need the dev libraries for zlib. These can be found Here. Install them using the supplied ReadMe file. It will be something like this...

Code Snippet

1. ./configure
2. make
3. make install

End of Code Snippet

Things to note


Modules
There are some things to note before we get onto the configuration phase of Proftpd... Proftpd has numerous modules, each which aide different aspects of the FTP server. FOr example, for FTPS we use the mod_tls module.... for SFTP, we iuse mod_sftp module... To integrate FTP user accounts with a backend mysql database, we use mod_mysql and mod_sql_mysql. etc....

Note: Some modules are not compiled by default! mod_sftp being one of them...

While there are a lot of modules included as standard, you can download other modules Here


Static Modules vs. DSO's (Dynamic Shared Objects)

Modules can either be statuc or in DSO form. A static module basically means you are compiling it into the Proftpd compilation. On the other hand, a DSO Will be a seperate physical file of the library that is loaded in dynamically. DSO's allow you to drop modules in quickly- without re-compiling Prtoftpd each time.

Read more on DSO's in Proftpd Here


Installation

1. First we need to configure the installation of Proftpd. This is where you will need to decide

- How you want your modules to be represented (Static or DSO)
- What modules you actually want to include with the installation

I will be including mod_sftp in this installation, and im going to make it static.


2. Browse to the directory where the Proftpd source is, and execute the following command... Check the file paths beforehand though!

Code Snippet

1. ./configure --prefix=/usr --with-includes=/usr/local/ssl/include --with-libraries=/usr/local/ssl/lib --enable-openssl --with-modules=mod_sftp --enable-dso

End of Code Snippet

--prefix
Where Proftpd will be installed to. I'm using CentOS as a root user, so it will be placed in /usr/sbin

--with-includes
You need to specify the include directory for the dev installation of OpenSSL. This include directory will include the development library headers which Proftpd requires to build in mod_sftp support. You can add more paths to this parameter as you wish, seperated by a colon.

--with-libraries
Similar to above but these will point to the OpenSSL libraries.

--enable-openssl
Required to enable OpenSSL for mod_sftp

--with-modules=mod_sftp
Include the sftp module. You can specify more by adding a colon after each module name.

--enable-dso
This is for DSO modules only, but I have included it as I already had some modules built with a previous version, and wanted to include them.

Note: If you wish to use DSO's (Shared Modules)... then replace --with-modules with --with-shared and ensure --enable-dso is included. Simple!


More info on compiling can be found Here


3. Produce the Makefile

Code Snippet

1. make

End of Code Snippet

4. Install

Code Snippet

1. make install

End of Code Snippet

Installation - Troubleshooting

In file included from mod_sftp.c:30:
mod_sftp.h:83:18: error: zlib.h: No such file or directory
make: *** [mod_sftp.lo] Error 1

You didn't install the zlib-dev development libraries. See prerequisites above!


/usr/bin/ld: cannot find -lcrypto
collect2: ld returned 1 exit status
make: *** [shared] Error 1

or

/usr/bin/ld: openssl/crypto.h: No such file or directory
make[1]: *** [support.o] Error 1
make[1]: Leaving directory `/src/proftpd-1.3.2b/src'
make: *** [src] Error 2


You didn't install the OpenSSL development libraries. See prerequisites above! Ensure the libraries AND headers are present.


Configuration

1. Check proftpd has registered itself correctly and that your static modules have been installed. Execute the following command...

Code Snippet

1. proftpd -l

End of Code Snippet

mod_sftp should be included in the list here.


2. Check the correct version has been installed and verify the location of the proftpd configuration file. Execute the following command...

Code Snippet

1. proftpd -V

End of Code Snippet

The version should match (Just incase an upgrade went wrong!) and you can see which configuration file Proftpd is using here... It should reside within the /etc directory of the --prefix specified when configuration the installation. I.e. /usr/etc/proftpd.conf


3. Open proftp.conf with your favourite text editor. The config file has the same syntax as Apache server configs, so it's easier to get to grips with.

View sample configurations Here

If you take a look at the Complex Virtual example, you can see how different VirtualHosts are configured.

Note: The Global tag contains settings that wiull be included across all VirtualHost's within the configuration file. Therefore, there is only ever one instance of the Global tag.

4. Take a copy of this file, and setup your Virtual hosts based on this format. You can even use the simple config file to start with. However, it is useful to understand what is possible here.

5. Save your proftpd.conf file and restart the proftpd service.
See this link for restarting services on various distros of Linux: http://theos.in/desktop-linux/tip-that-matters/how-do-i-restart-linux-network-service/

This ensures the latest config is read and is compatible before we continue.

[root@servername proftpd]# service proftpd restart
Shutting down proftpd: [ OK ]
Starting proftpd: [ OK ]


6. Keep the proftpd.conf file open and add the following for SFTP support.

Code Snippet

1. <IfModule mod_sftp.c>
2.         <VirtualHost xx.xx.xx.xx>
3.       SFTPEngine on
4.       SFTPLog /etc/proftpd/sftp/sftp.log
5.  
6.       # Configure the server to listen on port
7.       Port 2222
8.  
9.       # Configure both the RSA and DSA host keys, using the same host key
10.       # files that OpenSSH uses.
11.       SFTPHostKey /etc/ssh/ssh_host_rsa_key
12.       SFTPHostKey /etc/ssh/ssh_host_dsa_key
13.  
14.  
15.       # Configure the file used for comparing authorized public keys
16.       SFTPAuthorizedUserKeys file:/root/.ssh/authorized_keys
17.  
18.       # Enable compression
19.       SFTPCompression delayed
20.  
21.       # Allow the same number of authentication attempts as OpenSSH.
22.       #
23.       # It is recommended that you explicitly configure MaxLoginAttempts
24.       # for your SSH2/SFTP instance to be higher than the normal
25.       # MaxLoginAttempts value for FTP, as there are more ways to authenticate
26.       # using SSH2.
27.       MaxLoginAttempts 6
28.  
29.         </VirtualHost>
30. </IfModule>

End of Code Snippet

Note: Replace xx.xx.xx.xx with IP or hostname on which to accept incoming connections.

Note: Set the port for SFTP. If you have OpenSSH installed, you will either need to change the port OpenSSH uses, or change the port within this VirtualHost. I have opted for the latter, and use port 2222 for SFTP connections. You can't have two things on one port! Check what is running on certain ports using netstat.

Code Snippet

1. netstat -tulpn

End of Code Snippet

Note: The SFTPAuthorizedUserKeys attribute points to an authorized_keys file or public key authentication. Huh?? See my guide here on what it is and how to get setup

Note: You will notice the VirtualHost is wrapped within an IfModule tag. This is a condition that will granted true if the mod_sftp loaded correctly.


7. Restart the Proftpd service and test the connectivity using an FTP client.

Code Snippet

1. lftp sftp://user@server:port    (I.e. lftp sftp://sean@server:2222)
2.  
3. or
4.  
5. sftp user@server:port    (I.e. sftp sean@server:2222)

End of Code Snippet

You can also use FileZilla by setting the connectivity type to SFTP and specifying your configured port (I.e. 2222).


OPTIONAL: Module Configuration (DSO Configuration Only)

If you would like to include DSO's into the proftpd configuration, then open the Proftpd.conf file and add the following line...

Code Snippet

1. Include                         /path/to/modules.conf

End of Code Snippet

This path should point to an empty file called modules.conf. You can create this using your favourite text editor.

Now open up your newly created modules.conf file and add the following...

Code Snippet

1. # This is the directory where DSO modules reside
2. ModulePath /usr/libexec/proftpd
3.  
4. # Allow only user root to load and unload modules, but allow everyone
5. # to see which modules have been loaded
6. ModuleControlsACLs insmod,rmmod allow user root
7. ModuleControlsACLs lsmod allow user *
8.  
9. # Load the modules
10. LoadModule MODULENAMEHERE1.c
11. LoadModule MODULENAMEHERE2.c
12. LoadModule MODULENAMEHERE3.c

End of Code Snippet

The directory you specified within the ModulePath attribute should point to the directory where your DSO's reside. Now replace MODULENAMEHERE with the name of your modules.
Once you have done this, you may need to restart the Proftpd service for the changes to take effect.



Useful features and Debugging help

Proftpd Debug mode

You can place Proftpd in debugging mode using the following command...

Code Snippet

1. proftpd -nd10

End of Code Snippet

Please note, that you will have to stop the Proftpd service first, or you will get an 'address in use' error. Makes sense!

How to stop a service: http://theos.in/desktop-linux/tip-that-matters/how-do-i-restart-linux-network-service/

This is level 10 debugging and will show you connecting clients and all activity within the console output. Very useful to troubleshoot connectivity issues and auth.

Full documentation on debugging Here


Trace Logging
The mod_sftp module supports different forms of logging. The main module logging is done via the SFTPLog directive. For debugging purposes, the module also uses trace logging, via the module-specific "scp", "sftp", and "ssh2" log channels. Thus for trace logging, to aid in debugging, you would use the following in your proftpd.conf:

Code Snippet

1. TraceLog /path/to/sftp-trace.log
2. Trace scp:20 sftp:20 ssh2:20

End of Code Snippet

This trace logging can generate large files; it is intended for debugging use only, and should be removed from any production configuration.

mod_sftp Documentation Here


Logging

See the official guide on how logging can help you debug Proftpd. Logging documentation can be viewed Here

Where is proftpd?

Code Snippet

1. which -a proftpd

End of Code Snippet

Any problems or issues, please leave a comment and I will assist you where I can. I wrote this up from my notes, and im confident I have included most of the vital parts!

Shoutouts for the 200th BLOG POST!!!

Unix/Linux - Setting up public key authentication [SFTP/SSH/SCP etc]

Today I had the task of setting up public key authentication onto an existing system. The primary task was to ensure that you could send a file via SFTP without a password (password authentication). Joy!


Public key authentication - What the...?
This is basically an authentication method, just like specifying a username and a password (password auth).... The only difference is that a 'key' (some long encrypted string) is generated on a client machine, and the server is made aware of this in a config file. So when the client attempts to connect, the server will check the config, realise its the client, match it against the username, and we're in! Sounds easy I guess.... The basic aim is to get in without a password, while still maintaining security....

Let's start with the client (The machine you want to connect from)

Client Configuration

Note: Firstly, use the user which you wish to allow public key authentication for. We will be storing the keys in the $HOME directory, so make sure you've switched to the correct user from the start.

1. Generate a public and private key. Run this command...

Code Snippet

1. ssh-keygen -t dsa

End of Code Snippet

Example
[root@someserver ~]# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): /root/.ssh/id_dsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
blah:blah:blah:blah:blah:blah:blah:blah:blah:blah root@someserver

Note:
- I am using the root user here... root will be replaced by the name of your user.
- I used the same path with was hinted by the ssh-keygen utility. The keys should be generated within your $HOME dir under the .ssh directory. I would maybe rename id_dsa to something else, as this is the default name. Just a security consideration as somebody could search for your keys using the default name.
- Leave passphrase blank for getting this up and running. It just means you would have to provide the phrase in future.


In this example....
Private Key: /root/.ssh/id_dsa
Public Key: /root/.ssh/id_dsa.pub


You public key is the one you advertise... the private key, never let out of your sight! It's as good as a password to gain entry! But you know that already right?

2. Set permissions if you haven't already... security is number 1!

Code Snippet

1. chmod 700 ~/.ssh
2. chmod 600 ~/.ssh/id_rsa
3. chmod 644 ~/.ssh/id_rsa.pub

End of Code Snippet

3. Open the public key file and copy the contents. I recommand not using 'more' to copy because it puts new lines in where the console window wraps. Use something like vi text editor.

4. Keep this public key content handy because we'll need to tell the server about it!


Server Configuration

Note: I will assume you are logged in as a root user here.

1. First of all, lets enable publickey authentication! Open the sshd_config file for editing. I personally use vi. sshd_config is either in /etc/ , /etc/ssh/ or /etc/conf.d/

Mine is here: /etc/ssh/sshd_config


2. You need to enable RSAAuthentication and PubkeyAuthentication... You may need to uncomment those lines. Also, set the AuthorizedKeysFile to point to the authorized_keys file on the server. This file sits in the .ssh directory with the $HOME dir (Just like in the client config). However, we are using the root user, so it should be here: /root/.ssh or simply: ~/.ssh. Check the path is correct before you save.

Code Snippet

1. RSAAuthentication yes
2. PubkeyAuthentication yes
3. AuthorizedKeysFile     ~/.ssh/authorized_keys

End of Code Snippet

3. Save changes and close sshd_config. I usually restart the sshd service just incase.

Code Snippet

1. service sshd restart

End of Code Snippet

See this link for restarting services on various distros of Linux: http://theos.in/desktop-linux/tip-that-matters/how-do-i-restart-linux-network-service/


4. Now lets tell the server about our client's public key! Exciting right? Navigate to the .ssh directory (as discussed in step 2). Either open or create the file authorized_keys. This will contain various public keys for connecting clients. Edit this file and add the public key from step 3 in the client configuration. One entry per line if you already have content within there. Save the file.

5. Lets test!


Testing

1. Go back to your client and attempt to SSH onto your server.

Code Snippet

1. ssh username@servername

End of Code Snippet

You can also force ssh to use publickey auth (you may have various types of auth enabled).

Code Snippet

1. ssh -vvv -o PreferredAuthentications=publickey username@servername

End of Code Snippet

2. Simple SFTP test.

Code Snippet

1. sftp username@servername

End of Code Snippet

Any problems, please comment and I can assist where necessary! Have fun I guess....

Unix/Linux - Packet Analyzers - snoop and tcpdump

snoop
snoop is a very flexible command line packet analyzer included as part of Sun Microsystems' Solaris operating system.


tcpdump
tcpdump is a common packet analyzer that runs under the command line. It allows the user to intercept and display TCP/IP and other packets being transmitted or received over a network to which the computer is attached.


Examples

Troubleshoot a host

Code Snippet

1. snoop -v -d qfe0 -x0 host 192.168.1.87
2. tcpdump -i en0 host 192.168.1.87

End of Code Snippet

Exclude the host you're connected from

Code Snippet

1. snoop -x0 -d hme0 not host 192.168.1.20
2. tcpdump -i eth0 not host 192.168.1.20

End of Code Snippet

View only SSL packets

Code Snippet

1. snoop -x0 port 443
2. tcpdump port 443

End of Code Snippet

For more examples, view the man pages.

Linux - Manually and automatically adding system users

Please view this link for an indepth guide on how to add system users to /etc/passwd
http://floppix.ccai.com/adduser.html

Please Note

[root@server seantest]# passwd seantest
Changing password for user seantest.
passwd: Authentication token manipulation error

If u are running shadowed passwords there might be no entry for this user.


1. Make a backup of /etc/shadow

Code Snippet

1. cp /etc/shadow /etc/shadow.backup

End of Code Snippet

2. Delete /etc/shadow

Code Snippet

1. rm /etc/shadow

End of Code Snippet

3. Convert /etc/passwd using password convert

Code Snippet

1. pwconv

End of Code Snippet

And the same for groups...

1. Make a backup of /etc/gshadow

Code Snippet

1. cp /etc/gshadow /etc/gshadow.backup

End of Code Snippet

2. Delete /etc/gshadow

Code Snippet

1. rm /etc/gshadow

End of Code Snippet

3. Convert /etc/group using group convert

Code Snippet

1. grpconv

End of Code Snippet

Full path of service and service temporary directory when running

Execute the following command to view all processes and do a search of a service name (The service your searching for)

Code Snippet

1. ps auxwwwe | grep SERVICENAMEHERE

End of Code Snippet

This will give you back some info about the service. We are really only interested in the ProcessID (PID). This will be an integer.

Example return
root 24466 0.0 0.0 1476 280 ? S 2009 0:00 supervise sshd

PID will be: 24466

So now navigate to /proc and view the contents. You will see a directory for your PID. View the contents of this directory to see the info for the service.

The file path is located @ exe in this example: exe -> /usr/sbin/servicename

proftpd - Adding FTPS Support (mod_tls module)

This guide assumes you have proftpd installed with OpenSSL libs. If not, follow this guide

Firstly, I will quickly say....

FTPS or SFTP

People intend to mix FTPS and SFTP together, but both are actually completely differend.

FTPS is a normal FTP server but using SSL encrytion.
SFTP is a ftp kind of session over SSH (so everything is encrypted just like in SSH).


Notes
Users Guide: http://proftpd.org/localsite/Userguide/linked/userguide.html

Steps

*** Ensure mod_tls module is available within your proftpd installation.
*** Ensure you are a root user


1. Open proftpd.conf and add an include to a config file we are going to create (tls.conf). Add the following line below...

Code Snippet

1. Include         /etc/proftpd/tls.conf

End of Code Snippet

2. Now use vi to create the config file in the specified location...

Code Snippet

1. vi /etc/proftpd/tls.conf

End of Code Snippet

3. Enter the following information into the file (How to use vi)

Code Snippet

1. <IfModule mod_tls.c>
2.  
3.         TLSEngine                               on
4.         TLSLog                                  /var/log/proftpd/tls.log
5.         TLSProtocol                             SSLv23
6.         TLSRSACertificateFile                   /etc/proftpd/ssl/proftpd.cert.pem
7.         TLSRSACertificateKeyFile                /etc/proftpd/ssl/proftpd.key.pem
8.        
9.         #
10.         # Avoid CA cert and allow client renegotiation (to overcome 1.3.2c bug 3324)
11.         #TLSOptions                             NoCertRequest AllowClientRenegotiation
12.         #
13.         # Authenticate clients that want to use FTP over TLS?
14.         #
15.        
16.         TLSVerifyClient                         off
17.        
18.         #
19.         # Are clients required to use FTP over TLS when talking to this server?
20.         #
21.        
22.         TLSRequired                             on
23.        
24.         #
25.         # Allow SSL/TLS renegotiations when the client requests them, but
26.         # do not force the renegotations.  Some clients do not support
27.         # SSL/TLS renegotiations; when mod_tls forces a renegotiation, these
28.         # clients will close the data connection, or there will be a timeout
29.         # on an idle data connection.
30.         #
31.        
32.         TLSRenegotiate                          required off
33.        
34. </IfModule>

End of Code Snippet

4. Generate certificate using OpenSSL

Code Snippet

1. openssl req -new -x509 -days 365 -nodes -out /etc/proftpd/ssl/proftpd.cert.pem -keyout /etc/proftpd/ssl/proftpd.key.pem

End of Code Snippet

5. Save and close the file.

6. Now restart proftpd for the changes to take effect.

Code Snippet

1. /etc/init.d/proftpd restart

End of Code Snippet

7. Test FTPS connectivity with the server. See below...
Note: if there are any issues with the connection process, check the log file within the tls.conf file we created: /var/log/proftpd/tls.log


Testing FTPS with lftp

1. Execute following command

Code Snippet

1. lftp -u USERNAMEHERE -e 'set ftp:ssl-force true,ftp:ssl-protect-data true' SERVERNAMEHERE

End of Code Snippet

2. Enter password for user.

3. Perform a simple command...

Code Snippet

1. ls -l

End of Code Snippet

Solaris 9 [SunOS 5.9] - Installing Python from Source with SSL [This example uses 2.7.2]

This is a simple guide on how to install python on a Solaris 9 system. There are a few gotcha's which I am sharing and writing for future use.

1. Download Python. I took the compressed source tarball (.tgz). You are essentially compiling the source on your system.

2. Optional: Transferring it to the server. I had to transfer it to the server to install, so if you need to do that, see my previous post

3. Unzip the package using the following command. It will unzip, then untar.

Code Snippet

1. gunzip -c PYTHONFILENAME.tgz |tar xvf -

End of Code Snippet

4. You now need to configure the source. This will produce a Makefile based on your system. Navigate to the Python source directory, and execute the following command...

Code Snippet

1. ./configure --prefix=/usr/local

End of Code Snippet

5. Now we need to compile our Makefile that has been created.

Code Snippet

1. make

End of Code Snippet

6. Ensure you are a root user before this step ("su root" - to change). Execute the following command to install Python.

Code Snippet

1. make -i install

End of Code Snippet

7. If everything went well (it probably didn't - see below!). Add Python to your system PATH variable. This way, you don't need to refer to /usr/local everytime you execute a script. See my previous blog post on how to do this.

8. Simply execute the following command to check Python has set itself up correctly... Do this outside of the source directory, so you can test the PATH variable aswell.

Code Snippet

1. python --version

End of Code Snippet

9. Get an ice cold beerski in!


Troubleshooting

During the make procedure, you receive the following...

./Parser/asdl_c.py -c ./Python ./Parser/Python.asdl
/usr/bin/env: No such file or directory
make: *** [Python/Python-ast.c] Error 127

Simply touch the libraries it requires (see below), and re-try... (Run make clean before re-try)

Code Snippet

1. touch Include/Python-ast.h Python/Python-ast.c

End of Code Snippet

During the install procedure, you receive the following...

make: ar: Command not found

You need to add ar to your PATH variable. This is located in /usr/ccs/bin. See my previous blog post on how to do this.

Note: If you are receiving an error while re-trying or you wish to remove temporary install files, simply execute the following command... "make clean"


Optional: Adding SSL Support to Python
Python needs to be compiled with SSL support... You can enable this by firstly installing the OpenSSL development libraries (libssl-dev download here) before the initial Python installation. Ensure you can open the OpenSSL console by typing openssl... If not, add it to your PATH. You then uncomment a few lines from the Modules/Setup.dist file.

Open the 'Modules/Setup.dist' file for editing, and uncomment the following lines (Assuming you installed OpenSSL to the default location)...

Code Snippet

1. 204: # Socket module helper for SSL support; you must comment out the other
2. 205: # socket line above, and possibly edit the SSL variable:
3. 206: SSL=/usr/local/ssl
4. 207: _ssl _ssl.c \
5. 208:     -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
6. 209:     -L$(SSL)/lib -lssl -lcrypto</pre>

End of Code Snippet

Save and close the file... then copy the file to /Setup. Otherwise Python will warn you this this file is newer than the Setup copy. You can now proceed to install Python with the instructions as above.

You can verify the installation by running the following test script...

Code Snippet

1. python /usr/local/lib/python2.5/test/test_socket_ssl.py

End of Code Snippet

Module Errors
If you experience any errors with modules... then execute python's setup.py file. This will install all of the modules by default.

Code Snippet

1. python setup.py install

End of Code Snippet

Bash/sh/csh/tcsh - Updating PATH environment variable in session and on logon

When updating your PATH varible, it's usually because an installation requires programs and utilities within a directory, and the knowledge of the full path is not known. Either that, or you would like to refer to a command program within specifying the full path. You will usually receive the following error message if a program cannot be found...

xxx: Command not found

So lets check our current PATH using the following command...

Code Snippet

1. echo $PATH

End of Code Snippet

We can now view the current directories included in our PATH variable.
Example: /usr/sysmgr/bin:/bin:/usr/sbin:/usr/bin:/usr/ucb:/usr/sysmg/bin:/etc:/usr/local/bin:.

For example, we may wish to install a program, and it requires the ar tool (A tool to aid archiving). It is unaware of the full system path, so we need to add the directory it resides in to our environment variable.

We can either do this temporarily or permanently... It also depends on which shell you are using.


Note: To find out which shell you are using, execute the following command.

Code Snippet

1. echo $SHELL

End of Code Snippet

Temporarily

tcsh/csh Shell (Seperate by spaces and use set command)
set path=(/usr/sysmgr/bin /bin /usr/sbin /usr/bin /usr/ucb /usr/sysmg/bin /etc /usr/local/bin .)

bath/sh Shell (Seperate by colon and use export command)
export PATH=$PATH:/path/to/dir1:/path/to/dir2


Permanently

Bash Shell (Edit /.bash_profile or /.bash_profile files)
http://www.cyberciti.biz/faq/change-bash-profile/

tcsh/csh Shell (Edit /.login or /.cshrc files)
http://osr507doc.sco.com/en/OSUserG/_The_C-shell_login_and_cshrc.html

In both cases, you are simply adding the command into a login/shell startup script to that the variable is always set with the extra paths. To edit these files, I recommend using vi (text editor).

vi Help
vi in Solaris help
vi in Unix help

Transfer Files From One UNIX Server To Another using scp

In Unix, you can use the scp command to copy files and directories securely between remote hosts without starting an FTP session or logging into the remote systems explicitly. The scp command uses SSH to transfer data, so it requires a password or passphrase for authentication. Unlike rcp or FTP, scp encrypts both the file and any passwords exchanged so that anyone snooping on the network can't view them.

Warning: Be careful when copying between hosts files that have the same names; you may accidently overwrite them.

From Server to Local

Code Snippet

1. scp -r user@server1:/directory/files /localDirectory

End of Code Snippet

From Local to Server

Code Snippet

1. scp -r /localDirectory user@server1:/directory/files

End of Code Snippet

Python SUDS (SOAP API) full example with WSSE and complex types

I began this task in Perl originally, then decided to switch to Python and have an easier ride. I will upload the Perl example when I sort out a few minor issues with the SOAP::Lite library.

Anyway, in this example, you can specify a WSDL and WSSE (Web Service Security Extensions) username and password (sent in clear text btw), and it will send a SOAP request out and get a sample response back.

I have purposely consumed a service that has some complex types available (basically not just strings and ints). You can see how it is to work with the library and consume your own methods with this example.


Here are some useful things to note
- There have been problems with SUDS generating empty tags for optional properties for complex types. If this is the case, you will receive this error in your SOAP Body's response... "Server was unable to read request. ---> There is an error in the XML document. ---> Instance validation error: '' is not a valid value for PROPERTY_HERE."... To get around this, simply specify those properties (example below)

- client.factory.create() is used to let Python know about the complex types.

- "print client" (Using the example below) will tell you everything you need to know about your service (namespaces, types, methods, properties etc).

- The logger is your friend! Don't be a hero! Start out small and go big! The technique is to analyze the SOAP response and query the errors. If you can get a hold of what the correct SOAP envelope should look like, then compare this against the SOAP request you are sending out. This is the easiest way to solve any problems.

- Coming from a .NET background, I added a service reference and made a simple call with C#. You can then write more code to analyze the SOAP Request, or simply install Fiddler2 (If you haven't got it already, only 600kb and very useful!) to get the correct SOAP envelope to compare against.

Code Snippet

1. #!/usr/bin/python
2. #
3. # Sean Greasley. TutorialGenius.com 2012.
4. #
5. # Creates a portfolio object using the exacttarget SOAP API. An image must exist at the specified URN
6. # before alerting the system that the image is ready to be processed,
7. #
8. # USAGE:
9. #    -portfolio <Display Name> <URN> <File Name> <Optional: CustomerKey>
10. #    -portfoliowsdl <WSDL Address> <WSSE Username> <WSSE Password> <Display Name> <URN> <File Name> <Optional: CustomerKey>
11. #
12. #
13. # Imports
14. from suds.client import Client
15. from suds.wsse import *
16.  
17. # Logging Options
18. import logging
19. logging.basicConfig(level=logging.INFO)
20. logging.getLogger('suds.client').setLevel(logging.DEBUG)
21. logging.getLogger('suds.wsdl').setLevel(logging.DEBUG)
22. logging.getLogger('suds.wsse').setLevel(logging.DEBUG)
23.  
24.  
25. # Define usage options
26. def printUsage():
27.     print ""
28.     print "[USAGE]"
29.     print "------------------------------------------------------------------------"
30.     print "    " + sys.argv[0] + " -portfolio <Display Name> <URN> <File Name> <Optional: CustomerKey>"
31.     print "    " + sys.argv[0] + " -portfoliowsdl <WSDL Address> <WSSE Username> <WSSE Password> <Display Name> <URN> <File Name> <Optional: CustomerKey>"
32.     print ""
33.     return
34.  
35.  
36. # Validate argument input
37. if (len(sys.argv) <= 1):
38.     print "Invalid usage options..."
39.     printUsage()
40.     sys.exit(1)
41. elif (sys.argv[1] == "-portfolio" and (len(sys.argv) == 5 or len(sys.argv) == 6)):
42.     print "Setting up a portfolio"
43. elif (sys.argv[1] == "-portfoliowsdl" and (len(sys.argv) == 8 or len(sys.argv) == 9)):
44.     print "Setting up a portfolio with WSDL options"
45. else:
46.     print "Invalid usage options..."
47.     printUsage()
48.     sys.exit(1)
49.  
50.  
51.  
52. # Setup variables
53. WSDL_URL = "https://webservice.s4.exacttarget.com/etframework.wsdl"
54. WSSE_USERNAME = "Username here!"
55. WSSE_PASSWORD = "Password here!"
56. PORTFOLIO_DISPLAYNAME = "Test Sean Display Name1"
57. PORTFOLIO_URN = "http://www.ct4me.net/images/dmbtest.gif"
58. PORTFOLIO_FILENAME = "dmbtest.gif"
59. PORTFOLIO_CUSTOMERKEY = ""
60.  
61. if (sys.argv[1] == "-portfoliowsdl"):
62.     WSDL_URL = sys.argv[2]
63.     WSSE_USERNAME = sys.argv[3]
64.     WSSE_PASSWORD = sys.argv[4]
65.     PORTFOLIO_DISPLAYNAME = sys.argv[5]
66.     PORTFOLIO_URN = sys.argv[6]
67.     PORTFOLIO_FILENAME = sys.argv[7]
68.    
69.     try:
70.         PORTFOLIO_CUSTOMERKEY = sys.argv[8]
71.     except:
72.         print "No Customer key specified. Using default..."
73. elif (sys.argv[1] == "-portfolio"):
74.     PORTFOLIO_DISPLAYNAME = sys.argv[2]
75.     PORTFOLIO_URN = sys.argv[3]
76.     PORTFOLIO_FILENAME = sys.argv[4]
77.    
78.     try:
79.         PORTFOLIO_CUSTOMERKEY = sys.argv[5]
80.     except:
81.         print "No Customer key specified. Using default..."
82.  
83.  
84. # URL Detail
85. client = Client(WSDL_URL)
86.  
87.  
88. # WSSE Security
89. security = Security()
90. token = UsernameToken(WSSE_USERNAME, WSSE_PASSWORD)
91. security.tokens.append(token)
92. client.set_options(wsse=security)
93.  
94.  
95. # Build up portfolio
96. # 'Portfolio' is a complex type... so we use the create method to expose the properties to us. We can then populate the properties as normal.
97. portfolio = client.factory.create('Portfolio')
98. portfolio.DisplayName = PORTFOLIO_DISPLAYNAME
99. portfolio.CustomerKey = PORTFOLIO_CUSTOMERKEY
100. portfolio.Source = client.factory.create('ResourceSpecification')
101. portfolio.Source.URN = PORTFOLIO_URN
102. portfolio.FileName = PORTFOLIO_FILENAME
103.  
104.  
105. # For some reason the SUDS library tends to generate empty SOAP tags for optional properties. Here I have manually specified the defaults here. Just be aware of that!
106. createOptions = client.factory.create('CreateOptions')
107. createOptions.RequestType = "Synchronous"
108. createOptions.QueuePriority= "High"
109.  
110.  
111. # Attach Portfolio to array - Need to set at pos 0, as it returns 1 by default.
112. apiObject = [client.factory.create('APIObject')]  # Remember [ ], its an array!
113. apiObject[0] = portfolio
114.  
115.  
116. # Create portfolio
117. # This method also had 'out' parameters exposed
118. print client.service.Create(createOptions, apiObject)
119.  
120.  
121. # Uncomment this next line to find out useful information about your service.
122. # print client

End of Code Snippet