Unix Shells

csh (C-Shell)
The C shell is a command processor that's typically run in a text window, allowing the user to type commands which cause actions. The C shell can also read commands from a file, called a script. Like all Unix shells, it supports filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration. What differentiated the C shell, especially in the 1980s, were its interactive features and overall style. Its new features made it easier and faster to use. The overall style of the language looked more like C and was seen as more readable.

tcsh (TENEX C-Shell)
On many systems such as Mac OS X and Red Hat Linux csh is actually tcsh, an improved version of csh. One file containing the tcsh executable has links to it as both "csh" and "tcsh" so that either name refers to the same improved version of the C shell.

On Ubuntu and Solaris machines, there are two different packages: csh and tcsh. The former is a based on the original BSD version of csh and the later is the improved tcsh. To enable the improved version of csh (tcsh), just type 'tcsh'... You will now notice you have access to command history, working arrows etc...

Other shells exist, such as: bash, sh, ksh, zsh, rc etc... Each having their own unique advantages, however, the common shell principles still apply.



Shell startup Scripts (.files)
When you enter a shell (by logging in for example) or switch to another shell (su or executing a different shell), certain shell startup scripts will be executed. It can become confusing as to which shells use which files (I.e. .cshrc, .profile etc etc).

The following as a list of which shells use which files...

csh
Some versions have system-wide .cshrc and .login files. Every
version puts them in different places.

Start-up (in this order):
.cshrc - always; unless the -f option is used.
.login - login shells.

Upon termination:
.logout - login shells.

Others:
.history - saves the history (based on $savehist).

tcsh
Start-up (in this order):
/etc/csh.cshrc - always.
/etc/csh.login - login shells.
.tcshrc - always.
.cshrc - if no .tcshrc was present.
.login - login shells

Upon termination:
.logout - login shells.

Others:
.history - saves the history (based on $savehist).
.cshdirs - saves the directory stack.

sh
Start-up (in this order):
/etc/profile - login shells.
.profile - login shells.

Upon termination:
any command (or script) specified using the command:
trap "command" 0

ksh
Start-up (in this order):
/etc/profile - login shells.
.profile - login shells; unless the -p option is used.
$ENV - always, if it is set; unless the -p option is used.
/etc/suid_profile - when the -p option is used.

Upon termination:
any command (or script) specified using the command:
trap "command" 0

bash
Start-up (in this order):
/etc/profile - login shells.
.bash_profile - login shells.
.profile - login if no .bash_profile is present.
.bashrc - interactive non-login shells.
$ENV - always, if it is set.

Upon termination:
.bash_logout - login shells.

Others:
.inputrc - Readline initialization.

zsh
Start-up (in this order):
.zshenv - always, unless -f is specified.
.zprofile - login shells.
.zshrc - interactive shells, unless -f is specified.
.zlogin - login shells.

Upon termination:
.zlogout - login shells.

rc
Start-up:
.rcrc - login shells

pkg-get and pkgutil package management for Solaris/SunOS

pkg-get
Pkg-get is an open-source, Sun-approved software installation and management tool for Sun Solaris. You can use pkg-get to install a variety of pre-compiled software without having to worry about version compatibility, patch requirements, libraries, or any other prerequisites.

pkgutil
Pkgutil, written in Perl and licensed under GPL, is a tool to make installation of packages in Solaris easier. It handles package dependencies so all required packages are installed before the desired package automatically. This is based on functionality present in apt-get (Debian Linux).

Which tool?
Both tools offer a similar deal... They aide package management on SunOS, they work out dependencies for packages and versioning is controlled by the package manager. I personally use pkgutil, simply because The Open Community Software Project (OpenCSW) has now deprecated support for pkg-get users (This is a large database of free Solaris packages - Note: There are many others you can use with pkg-get)

OpenCSW/Blastwave
The Open Community Software Project (OpenCSW) is an open-source project providing Solaris binary packages of freely available or open-source software. This was previously hosted at Blastwave. The project was originally Blastwave, but this was forked to created The OpenCSW.
OpenCSW provides packages for Solaris 9 and 10, for 32 and 64-bit, x86 and SPARC architectures. Solaris 8 is no longer a 1st-class supported OS[2]: however, there still exists a legacy Solaris 8 archive, which occasionally gets updates.

This isn't the only place to retrieve Solaris packages, but with a nice large database (3500+ packages at time of writing), its quite beneficial to consume.

Getting Started with pkgutil
http://www.blastwave.org/jir/blastwave.fam


Mirrors
OpenCSW/Blastwave do not directly host packages... A number of mirror sites are available to get these, but you must set this in your pkgutil config file. This is typically located here: /etc/opt/csw/pkgutil.conf

To view a list of mirrors, visit The OpenCSW Mirror Page

When selecting a mirror, you must add '/stable' or '/unstable' to the end of the URL. Unstable packages will be more up to date, but may have bugs present. Stable, on the other hand, will have older packages but more robust.

Open your configuration file for editing and alter the "mirror" section. The below example is a stable UK mirror site.

Code Snippet

1. mirror=http://www.grangefields.co.uk/mirrors/csw/stable/

End of Code Snippet

Save the file and attempt to update the pkgutil catelog (Basically an index of all availalbe packages for a mirror).

Update the pkgutil catelog

Code Snippet

1. pkgutil -U

End of Code Snippet

Now the catelog is up to date, we can begin to use pkgutil to download and install packages.


Things to note
Ensure wget is in your system PATH (which wget)... pkgutil ships with it's own version, but most distributions of Solaris have a version located here: /usr/sfw/bin

Update pkgutil using the following command

Code Snippet

1. pkgutil -u pkgutil

End of Code Snippet

Install packages using the following (I.e. bash)

Code Snippet

1. pkgutil -i bash

End of Code Snippet

Download older versions of Oracle products

Welcome to the Oracle Software Delivery Cloud

Rar Command Line Usage Examples

Rar a single file/directory

Code Snippet

1. rar a NameOfYourArchive.rar /path/to/file

End of Code Snippet

Rar multiple files

Code Snippet

1. rar a NameOfYourArchive.rar /path/to/files/*

End of Code Snippet

Unrar a *.rar compressed file

Code Snippet

1. unrar x NameOfYourArchive.rar

End of Code Snippet

List the contents of a rar file without uncompressing it

Code Snippet

1. unrar l NameOfYourArchive.rar

End of Code Snippet

Restarting and managing services on Mac OS X

Method 1

Searching running processes to get he PID, then killing it (The service will automatically restart).

Example

Code Snippet

1. sudo ps -A | grep -i ssh

End of Code Snippet

23450 ?? S 0:00.12 /usr/sbin/sshd -i
23455 ?? S 0:00.03 /usr/sbin/sshd -i
23649 p8 S+ 0:00.00 grep -i ssh


Now use 'kill' on the PID...

Code Snippet

1. sudo kill 23450

End of Code Snippet

Method 2

launchd is a unified, open-source service management framework for starting, stopping and managing daemons, applications, processes, and scripts. Written and designed by Dave Zarzycki at Apple, it was introduced with Mac OS X Tiger and is licensed under the Apache License.

We will be using launchctl, a subset of this framework to manage services...

List all services

Code Snippet

1. launchctl list

End of Code Snippet

We can then use this list to identify the names of our services.

Here's an example with SSH...

Stopping a service

Code Snippet

1. launchctl stop com.openssh.sshd

End of Code Snippet

Starting a service

Code Snippet

1. launchctl start com.openssh.sshd

End of Code Snippet

How to enable X11 Forwarding with SSH on Mac OS X

The aim of this blog post is to discuss how to display an X window from a target machine on our local system.

A good example here would be the Oracle installer for linux... when the installer is executed, it utilises the windowing system configured to display the installer via. SSH. Lets see how we can set this up...


What is the X windowing system (or X11)
The X window system (commonly X Window System or X11, based on its current major version being 11) is a computer software system and network protocol that provides a basis for graphical user interfaces (GUIs) and rich input device capability for networked computers.

Advantages of using SSH for X11 Traffic
- No problems with firewalls, as long as they let SSH pass through (No need to allow access to tcp-port 6000+)
- Communication is encrypted


Note to other OS users [Windows etc]
As long as your software/OS implements the X windowing system, you will be able to display X windows. For windows users, I recommend PuTTY


Mac OS X Configuration

Enable X11 Forwarding in sshd_config
In order to allow X11 forwarding to pass through SSH, we need to enable X11 Forwarding in our config for SSH. This is located in a file called sshd_config and found typically here: /etc/sshd_config or /etc/ssh/sshd_config

Uncomment out the line "# X11Forwarding no" and set it to "yes" (If this hasn't been done already. You will be left with this configuration...

Code Snippet

1. X11Forwarding yes

End of Code Snippet

Save the file, and restart the sshd service...


Add your target system to the access control list
So our windowing system on our local machine has an access control list (We cant let anybody in as and when we please!)... So we need to allow our target system access to our windowing system.

Code Snippet

1. xhost fullyqualified.server.name

End of Code Snippet

You will then see this message to confirm...
fullyqualified.server.name being added to access control list


SSH onto our target machine (with the apps using the windowing system. I.e. Oracle Installer)
We can now SSH onto our target machine...

Code Snippet

1. ssh -X user@fullyqualified.server.name

End of Code Snippet

Ensure the DISPLAY environment variable is configured correctly
On the machine we just SSH'd onto, we need to make sure the DISPLAY environment variable is configured correctly. Our apps will use this to find out where our target machine that supports the X windowing system is located.

Code Snippet

1. DISPLAY=fullyqualifiedaddressforourhostmachine:0.0

End of Code Snippet

Ensure this has been set correctly...

Code Snippet

1. echo $DISPLAY

End of Code Snippet

Now lets test to see if our X apps can use our host windowing system via. SSH...

Code Snippet

1. xclock &

End of Code Snippet

If you do not have xclock, or maybe your trying to execute an installer for Oracle or something, then try and execute the installer.


Gotchas
- Do not set the DISPLAY variable in any login scripts (I.e. bashrc, bash_profile etc...)

Python - POST HTTP multipart form (Using standard urllib libraries)

The Python script in this example utilises standard Python libraries to POST a form with content type multipart/form-data over HTTP (or quite simply, a multipart form).

The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.

I won't delve deep into the details on multipart forms, but if you would like a good explanation, read the official W3C document. One thing if doesn't mention on there, is that the boundary can be between 1 and 70 characters long, consisting of alphanumeric, and the punctuation you see in the list. Spaces are allowed except at the end.

Code Snippet

1.     import itertools
2.     import mimetools
3.     import mimetypes
4.     from cStringIO import StringIO
5.     import urllib
6.     import urllib2
7.      
8.     class MultiPartForm(object):
9.         """Accumulate the data to be used when posting a form."""
10.      
11.         def __init__(self):
12.             self.form_fields = []
13.             self.files = []
14.            
15.             # Generate unique boundary string with the format: hostipaddr.uid.pid.timestamp.random
16.             self.boundary = mimetools.choose_boundary()
17.             return
18.        
19.         def get_content_type(self):
20.             return 'multipart/form-data; boundary=%s' % self.boundary
21.      
22.         def add_field(self, name, value):
23.             """Add a simple field to the form data."""
24.             self.form_fields.append((name, value))
25.             return
26.      
27.         def add_file(self, fieldname, filename, fileHandle, mimetype=None):
28.             """Add a file to be uploaded."""
29.             body = fileHandle.read()
30.             if mimetype is None:
31.                 mimetype = mimetypes.guess_type(filename)[0] or 'application/octet-stream'
32.             self.files.append((fieldname, filename, mimetype, body))
33.             return
34.        
35.         def __str__(self):
36.             """Return a string representing the form data, including attached files."""
37.             # Build a list of lists, each containing "lines" of the
38.             # request.  Each part is separated by a boundary string.
39.             # Once the list is built, return a string where each
40.             # line is separated by '\r\n'.  
41.             parts = []
42.             part_boundary = '--' + self.boundary
43.            
44.             # Add the form fields
45.             parts.extend(
46.                 [ part_boundary,
47.                   'Content-Disposition: form-data; name="%s"' % name,
48.                   '',
49.                   value,
50.                 ]
51.                 for name, value in self.form_fields
52.                 )
53.            
54.             # Add the files to upload
55.             parts.extend(
56.                 [ part_boundary,
57.                   'Content-Disposition: file; name="%s"; filename="%s"' % \
58.                      (field_name, filename),
59.                   'Content-Type: %s' % content_type,
60.                   '',
61.                   body,
62.                 ]
63.                 for field_name, filename, content_type, body in self.files
64.                 )
65.            
66.             # Flatten the list and add closing boundary marker,
67.             # then return CR+LF separated data
68.             flattened = list(itertools.chain(*parts))
69.             flattened.append('--' + self.boundary + '--')
70.             flattened.append('')
71.             return '\r\n'.join(flattened)
72.      
73.      
74.      
75.      
76.      
77.     if __name__ == '__main__':
78.      
79.         # Create the form with simple fields
80.         form = MultiPartForm()
81.        
82.        
83.         """
84.            Either specify a source, file or a url...
85.            The system will take the first one respectively if all three have content.
86.            This example uses a dummy file called test.sql, with a string of content.
87.       """
88.        
89.         # No source specified
90.         form.add_field('source', '')
91.        
92.         # REAL FILE EXAMPLE
93.         # fileHandle = open ('test.sql', 'r')
94.         # form.add_file('file', 'test.sql', fileHandle)
95.         # fileHandle.close()
96.        
97.         # Use dummy file
98.         form.add_file('file', 'test.sql', fileHandle=StringIO('FILE CONTENTS'))
99.        
100.         # No URL specified
101.         form.add_field('url', '')
102.        
103.        
104.        
105.         # Add remaining required parameters
106.         form.add_field('language', 'php')
107.         form.add_field('line_numbers', '2')
108.         form.add_field('word_wrap', 'on')
109.         form.add_field('tab_width', '8')
110.         form.add_field('highlight_keywords', 'on')
111.         form.add_field('default_color', '0000bb')
112.         form.add_field('keyword_colors[1]', 'DEFAULT')
113.         form.add_field('keyword_colors[2]', 'DEFAULT')
114.         form.add_field('keyword_colors[3]', 'DEFAULT')
115.         form.add_field('keyword_colors[4]', 'DEFAULT')
116.         form.add_field('comments_color', 'DEFAULT')
117.         form.add_field('escaped_chars_color', 'DEFAULT')
118.         form.add_field('brackets_color', 'DEFAULT')
119.         form.add_field('strings_color', 'DEFAULT')
120.         form.add_field('numbers_color', 'DEFAULT')
121.         form.add_field('methods_color', 'DEFAULT')
122.      
123.         # Build the request
124.         request = urllib2.Request('http://qbnz.com/highlighter/php_highlighter.php')
125.        
126.         # Hi i'm FireFox 5
127.         request.add_header('User-agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11')
128.        
129.         # We are required to specify the content type and length...
130.         body = str(form)
131.         request.add_header('Content-type', form.get_content_type())
132.         request.add_header('Content-length', len(body))
133.         request.add_data(body)
134.      
135.             # Take a look at what we are sending... useful to match up against the required request
136.         print
137.         print 'OUTGOING DATA:'
138.         print request.get_data()
139.      
140.             # View servers response
141.         print
142.         print 'SERVER RESPONSE:'
143.         print urllib2.urlopen(request).read()

End of Code Snippet

GeSHi - Generic Programming Code Syntax Highlighter

Code syntax highlighter written in PHP
http://qbnz.com/highlighter/

You can easily interface with blogger/blogspot by using the demo website and copying out the CSS for your chosen language. Each language will have a different CSS stylesheet, so if you have to tweak one language to fit your blogspot template, then you will need to apply this to the others.

Examples

Default code block stylings
This is a standard code block [<code>This is a standard code block</code>]


Using GeSHi stylings

Code Snippet

1. This is an example code snippet
2. # Here is a comment, btw we are using Bash code highlighting
3.  
4.   if [ $THIS -eq 999 ]; then
5.     eval "lets evaluate something here"
6.   fi
7.  
8. # Have fun!

End of snippet

Line numbers are excluded when copying a code snippet to the clipboard also, which is another good thing!

IMPORTANT UPDATE

With BlogSpot/Blogger, if you add a new post and give it a Label with the same name as a chosen language (I.e. Add Python code to a post, and assign a Python Label/Categroy to the post (which is quite common))... Your blog post will appear garbled. This is because Blogger assigns the label to a class and this will take on your CSS styles from GeSHi

Here is a link to a CSS file I have used with most of the languages I talk about in. It support all the default settings, but with word wrap and line numbers both ON. I have prefixed all CSS classes with "sh_" to avoid this problem. Remember to change the class of the main DIV when taking the styles from GeSHi.

Download link for modified CSS

Unix Shell Script - Get Tomorrows Date

This code snippet will calculate tomorrows date in the format DDMMYYYY. It takes into account correct monthly overlaps and leap years etc...

Code Snippet

1.   # default output format, change as necessary
2.   defaultof="%Y%m%d"
3.  
4.   # check for input format, else use default format,
5.   # refer to 'man date' for help on format
6.   # script only supports %Y %m %d at the moment
7.   of=$defaultof
8.   [ $# -eq 1 ] && of="$1"
9.  
10.   # get today's date
11.   eval "`date +'y=%Y m=%m d=%d'`"
12.  
13.   #check for max number of days in current month
14.   days=31
15.   if [ $m -eq 4 ] || [ $m -eq 6 ] || [ $m -eq 9 ] || [ $m -eq 11 ] ; then
16.     days=30
17.   fi
18.   # check for leap year if feb
19.   if [ $m -eq 2 ]; then
20.     days=28
21.     leap1=`expr $y % 4`
22.     leap2=`expr $y % 100`
23.     leap3=`expr $y % 400`
24.     if [ $leap1 -eq 0 ] ; then
25.       if [ $leap2 -gt 0 ] || [ $leap3 -eq 0 ] ; then
26.         days=29
27.       fi
28.     fi
29.   fi
30.  
31.   # increment date
32.   if [ $d -eq $days ]; then
33.     d=1
34.     m=`expr $m + 1`
35.     if [ $m -eq 13 ]; then
36.       m=1
37.       y=`expr $y + 1`
38.     fi
39.   else
40.     d=`expr $d + 1`
41.   fi
42.  
43.   #Solaris date does not accept -d
44.   #date -d "$y-$m-$d" +"$of"
45.  
46.   eval "y=`expr $y + 0` m=`expr $m + 0` d=`expr $d + 0`"
47.   eval "y=`printf "%04d" $y` m=`printf "%02d" $m` d=`printf "%02d" $d`"
48.   tomorrowsDate="${d}${m}${y}"

End of snippet

iptables/Netfilter - Command line overview

iptables/netfilter
Most distributions of linux have an inbuilt firewall, commonly refered to as iptables. In actuality and more accuratly, it is iptables/netfilter. iptables sits in user space where the user can interact and manage rulesets. Netfilter is a kernal module, built into the kernal, that actually does the filtering

iptables is used to set up, maintain, and inspect the tables of IPv4 packet filter rules in the Linux kernel. IPv6 addresses can be managed with ip6tables.

A number of GUIs can be used to manage the process, but often lack the flexibility of the command line. Lets take a look at the process in more detail...

Chains
With a new installation, three pre-defined chains exist (INPUT, OUTPUT and FORWARD)... You can add new chains, remove them and rename them. They simply serve as a container for holding rules. You will refer to the name of your chain when making new rules and managing existing ones.

Let's assume we stuck with the pre-defined chains...

INPUT - All packets destined for the host computer (I.e. keeping people/processes out!)
OUTPUT - All packets originating from the host computer.
FORWARD - All packets neither destined for nor originating from the host computer, but passing through (routed by) the host computer. This chain is used if you are using your computer as a router.


The ordering of rules in a chain does matter. A packet is checked against each rule in turn. Once a packet matches a specific rule (Starting at rule 1), then the action is taken on that packet (ACCEPT, DROP etc..), all other rules below this are ignored. If a packet makes it all the way down the rule chain with no matches, then the default action for that chain is taken. This is referred to as the default policy and may be set to either ACCEPT or DROP the packet.


Let's see a live example...


To display a list of current rules, execute the following command:

Code Snippet

1. iptables --list

End of Code Snippet

This will display the following...

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT icmp -- anywhere anywhere icmp any
ACCEPT esp -- anywhere anywhere
ACCEPT ah -- anywhere anywhere
ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns
ACCEPT udp -- anywhere anywhere udp dpt:ipp
ACCEPT tcp -- anywhere anywhere tcp dpt:ipp
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited


Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination


You can see that the default policy is to REJECT the packet on all protocols, from any source and to any destination.

You can view all of the commands for iptables by executing the following command:

Code Snippet

1. iptables --help

End of Code Snippet

In the examples section below, I have listed some common commands for managing firewall rules. Below each, discusses the parameters used. I have used a range of commands which allow me to explain the majority of the accepted values in detail...


EXAMPLES


List all IP chains with line numbers

Code Snippet

1. iptables --list --line-numbers

End of Code Snippet

- Lists all rules for all chains, also echo's line numbers top the left (This allows us to insert rules at certain positions)

Insert a rule at position 13 (Chain: INPUT, Protocol: TCP, Port 115)

Code Snippet

1. iptables --insert INPUT 13 -m state --state NEW -p tcp --dport 115 -j ACCEPT

End of Code Snippet

States
- Here we're using the -m switch to load a module (state). The state module is able to examine the state of a packet and determine if it is NEW, ESTABLISHED or RELATED
- NEW refers to incoming packets that are new incoming connections that weren't initiated by the host system.
- ESTABLISHED and RELATED refers to incoming packets that are part of an already established connection or related to and already established connection.

-j ACCEPT
-j (jump) to the target action for packets matching the rule - in this case ACCEPT.


Append a rule to the bottom of a chosen chain (In this case the INPUT chain) for an interface [localhost])

Code Snippet

1. iptables --append INPUT -i lo -j ACCEPT

End of Code Snippet

- 'lo' represents localhost (127.0.0.1)... This is generally required as many software applications expect to be able to communicate with the localhost adaptor.

Remove all rules in all chains

Code Snippet

1. iptables --flush

End of Code Snippet

Remove all rules in a chain

Code Snippet

1. iptables --flush [Chain Name]

End of Code Snippet

Remove a rule (Remove a rule at position 13)

Code Snippet

1. iptables --delete INPUT 13

End of Code Snippet

Saving Rules (Persisting changing on system reboot)

Firewall changes need to be saved in order to be persisted on system reboot. We can achieve this by using the following command...

Code Snippet

1. service iptables save

End of Code Snippet

Saving firewall rules to /etc/sysconfig/iptables: [ OK ]


This executes the iptables init script, which runs /sbin/iptables-save and writes the current iptables configuration to /etc/sysconfig/iptables. Upon reboot, the iptables init script reapplies the rules saved in /etc/sysconfig/iptables by using the /sbin/iptables-restore command.



References and more info
http://wiki.centos.org/HowTos/Network/IPTables