Solaris 9 [SunOS 5.9] - Installing Python from Source with SSL [This example uses 2.7.2]

This is a simple guide on how to install python on a Solaris 9 system. There are a few gotcha's which I am sharing and writing for future use.

1. Download Python. I took the compressed source tarball (.tgz). You are essentially compiling the source on your system.

2. Optional: Transferring it to the server. I had to transfer it to the server to install, so if you need to do that, see my previous post

3. Unzip the package using the following command. It will unzip, then untar.

Code Snippet

1. gunzip -c PYTHONFILENAME.tgz |tar xvf -

End of Code Snippet

4. You now need to configure the source. This will produce a Makefile based on your system. Navigate to the Python source directory, and execute the following command...

Code Snippet

1. ./configure --prefix=/usr/local

End of Code Snippet

5. Now we need to compile our Makefile that has been created.

Code Snippet

1. make

End of Code Snippet

6. Ensure you are a root user before this step ("su root" - to change). Execute the following command to install Python.

Code Snippet

1. make -i install

End of Code Snippet

7. If everything went well (it probably didn't - see below!). Add Python to your system PATH variable. This way, you don't need to refer to /usr/local everytime you execute a script. See my previous blog post on how to do this.

8. Simply execute the following command to check Python has set itself up correctly... Do this outside of the source directory, so you can test the PATH variable aswell.

Code Snippet

1. python --version

End of Code Snippet

9. Get an ice cold beerski in!


Troubleshooting

During the make procedure, you receive the following...

./Parser/asdl_c.py -c ./Python ./Parser/Python.asdl
/usr/bin/env: No such file or directory
make: *** [Python/Python-ast.c] Error 127

Simply touch the libraries it requires (see below), and re-try... (Run make clean before re-try)

Code Snippet

1. touch Include/Python-ast.h Python/Python-ast.c

End of Code Snippet

During the install procedure, you receive the following...

make: ar: Command not found

You need to add ar to your PATH variable. This is located in /usr/ccs/bin. See my previous blog post on how to do this.

Note: If you are receiving an error while re-trying or you wish to remove temporary install files, simply execute the following command... "make clean"


Optional: Adding SSL Support to Python
Python needs to be compiled with SSL support... You can enable this by firstly installing the OpenSSL development libraries (libssl-dev download here) before the initial Python installation. Ensure you can open the OpenSSL console by typing openssl... If not, add it to your PATH. You then uncomment a few lines from the Modules/Setup.dist file.

Open the 'Modules/Setup.dist' file for editing, and uncomment the following lines (Assuming you installed OpenSSL to the default location)...

Code Snippet

1. 204: # Socket module helper for SSL support; you must comment out the other
2. 205: # socket line above, and possibly edit the SSL variable:
3. 206: SSL=/usr/local/ssl
4. 207: _ssl _ssl.c \
5. 208:     -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
6. 209:     -L$(SSL)/lib -lssl -lcrypto</pre>

End of Code Snippet

Save and close the file... then copy the file to /Setup. Otherwise Python will warn you this this file is newer than the Setup copy. You can now proceed to install Python with the instructions as above.

You can verify the installation by running the following test script...

Code Snippet

1. python /usr/local/lib/python2.5/test/test_socket_ssl.py

End of Code Snippet

Module Errors
If you experience any errors with modules... then execute python's setup.py file. This will install all of the modules by default.

Code Snippet

1. python setup.py install

End of Code Snippet

Bash/sh/csh/tcsh - Updating PATH environment variable in session and on logon

When updating your PATH varible, it's usually because an installation requires programs and utilities within a directory, and the knowledge of the full path is not known. Either that, or you would like to refer to a command program within specifying the full path. You will usually receive the following error message if a program cannot be found...

xxx: Command not found

So lets check our current PATH using the following command...

Code Snippet

1. echo $PATH

End of Code Snippet

We can now view the current directories included in our PATH variable.
Example: /usr/sysmgr/bin:/bin:/usr/sbin:/usr/bin:/usr/ucb:/usr/sysmg/bin:/etc:/usr/local/bin:.

For example, we may wish to install a program, and it requires the ar tool (A tool to aid archiving). It is unaware of the full system path, so we need to add the directory it resides in to our environment variable.

We can either do this temporarily or permanently... It also depends on which shell you are using.


Note: To find out which shell you are using, execute the following command.

Code Snippet

1. echo $SHELL

End of Code Snippet

Temporarily

tcsh/csh Shell (Seperate by spaces and use set command)
set path=(/usr/sysmgr/bin /bin /usr/sbin /usr/bin /usr/ucb /usr/sysmg/bin /etc /usr/local/bin .)

bath/sh Shell (Seperate by colon and use export command)
export PATH=$PATH:/path/to/dir1:/path/to/dir2


Permanently

Bash Shell (Edit /.bash_profile or /.bash_profile files)
http://www.cyberciti.biz/faq/change-bash-profile/

tcsh/csh Shell (Edit /.login or /.cshrc files)
http://osr507doc.sco.com/en/OSUserG/_The_C-shell_login_and_cshrc.html

In both cases, you are simply adding the command into a login/shell startup script to that the variable is always set with the extra paths. To edit these files, I recommend using vi (text editor).

vi Help
vi in Solaris help
vi in Unix help

Transfer Files From One UNIX Server To Another using scp

In Unix, you can use the scp command to copy files and directories securely between remote hosts without starting an FTP session or logging into the remote systems explicitly. The scp command uses SSH to transfer data, so it requires a password or passphrase for authentication. Unlike rcp or FTP, scp encrypts both the file and any passwords exchanged so that anyone snooping on the network can't view them.

Warning: Be careful when copying between hosts files that have the same names; you may accidently overwrite them.

From Server to Local

Code Snippet

1. scp -r user@server1:/directory/files /localDirectory

End of Code Snippet

From Local to Server

Code Snippet

1. scp -r /localDirectory user@server1:/directory/files

End of Code Snippet

Python SUDS (SOAP API) full example with WSSE and complex types

I began this task in Perl originally, then decided to switch to Python and have an easier ride. I will upload the Perl example when I sort out a few minor issues with the SOAP::Lite library.

Anyway, in this example, you can specify a WSDL and WSSE (Web Service Security Extensions) username and password (sent in clear text btw), and it will send a SOAP request out and get a sample response back.

I have purposely consumed a service that has some complex types available (basically not just strings and ints). You can see how it is to work with the library and consume your own methods with this example.


Here are some useful things to note
- There have been problems with SUDS generating empty tags for optional properties for complex types. If this is the case, you will receive this error in your SOAP Body's response... "Server was unable to read request. ---> There is an error in the XML document. ---> Instance validation error: '' is not a valid value for PROPERTY_HERE."... To get around this, simply specify those properties (example below)

- client.factory.create() is used to let Python know about the complex types.

- "print client" (Using the example below) will tell you everything you need to know about your service (namespaces, types, methods, properties etc).

- The logger is your friend! Don't be a hero! Start out small and go big! The technique is to analyze the SOAP response and query the errors. If you can get a hold of what the correct SOAP envelope should look like, then compare this against the SOAP request you are sending out. This is the easiest way to solve any problems.

- Coming from a .NET background, I added a service reference and made a simple call with C#. You can then write more code to analyze the SOAP Request, or simply install Fiddler2 (If you haven't got it already, only 600kb and very useful!) to get the correct SOAP envelope to compare against.

Code Snippet

1. #!/usr/bin/python
2. #
3. # Sean Greasley. TutorialGenius.com 2012.
4. #
5. # Creates a portfolio object using the exacttarget SOAP API. An image must exist at the specified URN
6. # before alerting the system that the image is ready to be processed,
7. #
8. # USAGE:
9. #    -portfolio <Display Name> <URN> <File Name> <Optional: CustomerKey>
10. #    -portfoliowsdl <WSDL Address> <WSSE Username> <WSSE Password> <Display Name> <URN> <File Name> <Optional: CustomerKey>
11. #
12. #
13. # Imports
14. from suds.client import Client
15. from suds.wsse import *
16.  
17. # Logging Options
18. import logging
19. logging.basicConfig(level=logging.INFO)
20. logging.getLogger('suds.client').setLevel(logging.DEBUG)
21. logging.getLogger('suds.wsdl').setLevel(logging.DEBUG)
22. logging.getLogger('suds.wsse').setLevel(logging.DEBUG)
23.  
24.  
25. # Define usage options
26. def printUsage():
27.     print ""
28.     print "[USAGE]"
29.     print "------------------------------------------------------------------------"
30.     print "    " + sys.argv[0] + " -portfolio <Display Name> <URN> <File Name> <Optional: CustomerKey>"
31.     print "    " + sys.argv[0] + " -portfoliowsdl <WSDL Address> <WSSE Username> <WSSE Password> <Display Name> <URN> <File Name> <Optional: CustomerKey>"
32.     print ""
33.     return
34.  
35.  
36. # Validate argument input
37. if (len(sys.argv) <= 1):
38.     print "Invalid usage options..."
39.     printUsage()
40.     sys.exit(1)
41. elif (sys.argv[1] == "-portfolio" and (len(sys.argv) == 5 or len(sys.argv) == 6)):
42.     print "Setting up a portfolio"
43. elif (sys.argv[1] == "-portfoliowsdl" and (len(sys.argv) == 8 or len(sys.argv) == 9)):
44.     print "Setting up a portfolio with WSDL options"
45. else:
46.     print "Invalid usage options..."
47.     printUsage()
48.     sys.exit(1)
49.  
50.  
51.  
52. # Setup variables
53. WSDL_URL = "https://webservice.s4.exacttarget.com/etframework.wsdl"
54. WSSE_USERNAME = "Username here!"
55. WSSE_PASSWORD = "Password here!"
56. PORTFOLIO_DISPLAYNAME = "Test Sean Display Name1"
57. PORTFOLIO_URN = "http://www.ct4me.net/images/dmbtest.gif"
58. PORTFOLIO_FILENAME = "dmbtest.gif"
59. PORTFOLIO_CUSTOMERKEY = ""
60.  
61. if (sys.argv[1] == "-portfoliowsdl"):
62.     WSDL_URL = sys.argv[2]
63.     WSSE_USERNAME = sys.argv[3]
64.     WSSE_PASSWORD = sys.argv[4]
65.     PORTFOLIO_DISPLAYNAME = sys.argv[5]
66.     PORTFOLIO_URN = sys.argv[6]
67.     PORTFOLIO_FILENAME = sys.argv[7]
68.    
69.     try:
70.         PORTFOLIO_CUSTOMERKEY = sys.argv[8]
71.     except:
72.         print "No Customer key specified. Using default..."
73. elif (sys.argv[1] == "-portfolio"):
74.     PORTFOLIO_DISPLAYNAME = sys.argv[2]
75.     PORTFOLIO_URN = sys.argv[3]
76.     PORTFOLIO_FILENAME = sys.argv[4]
77.    
78.     try:
79.         PORTFOLIO_CUSTOMERKEY = sys.argv[5]
80.     except:
81.         print "No Customer key specified. Using default..."
82.  
83.  
84. # URL Detail
85. client = Client(WSDL_URL)
86.  
87.  
88. # WSSE Security
89. security = Security()
90. token = UsernameToken(WSSE_USERNAME, WSSE_PASSWORD)
91. security.tokens.append(token)
92. client.set_options(wsse=security)
93.  
94.  
95. # Build up portfolio
96. # 'Portfolio' is a complex type... so we use the create method to expose the properties to us. We can then populate the properties as normal.
97. portfolio = client.factory.create('Portfolio')
98. portfolio.DisplayName = PORTFOLIO_DISPLAYNAME
99. portfolio.CustomerKey = PORTFOLIO_CUSTOMERKEY
100. portfolio.Source = client.factory.create('ResourceSpecification')
101. portfolio.Source.URN = PORTFOLIO_URN
102. portfolio.FileName = PORTFOLIO_FILENAME
103.  
104.  
105. # For some reason the SUDS library tends to generate empty SOAP tags for optional properties. Here I have manually specified the defaults here. Just be aware of that!
106. createOptions = client.factory.create('CreateOptions')
107. createOptions.RequestType = "Synchronous"
108. createOptions.QueuePriority= "High"
109.  
110.  
111. # Attach Portfolio to array - Need to set at pos 0, as it returns 1 by default.
112. apiObject = [client.factory.create('APIObject')]  # Remember [ ], its an array!
113. apiObject[0] = portfolio
114.  
115.  
116. # Create portfolio
117. # This method also had 'out' parameters exposed
118. print client.service.Create(createOptions, apiObject)
119.  
120.  
121. # Uncomment this next line to find out useful information about your service.
122. # print client

End of Code Snippet

Perl: SOAP::Lite - 500 Can't connect to webservice.service.com:443 (Invalid argument) https

I had a problem recently with trying to call a HTTPS Web Service using Perl's SOAP::Lite library, and I was receiving this error:
"500 Can't connect to webservice.service.com:443 (Invalid argument)"...

Note: Port 443 ssl. This is the default port for HTTPS/SSL. This is the encrypted form of HTTP that normally runs over port 80.

Okay, so problem identified... Cannot communicate with service over 443... What next?

We use cpan to install another library called "Crypt::SSLeay"... sounds spooky! But don't worry, it simply provides support for the HTTPS protocol in LWP (Library for WWW in Perl).

cpan - what the heck? View my previous post on cpan here

Simply issue this command, and cpan will do the rest...

Code Snippet

1. sudo perl -MCPAN -e "install Crypt::SSLeay"

End of Code Snippet

Perl: SOAP::Lite Simple .NET Service Call

Ever wondered how to call a .NET web service using the SOAP::Lite library?
I didn't think so, but just incase you did!....

Code Snippet

1. #!/usr/bin/perl
2. use SOAP::Lite 'trace', 'debug'; # adds debug and tracing to view SOAP msgs
3. use SOAP::WSDL;
4. use strict;
5. use warnings;
6.  
7. my $soap = SOAP::Lite
8. -> uri('http://services.aonaware.com/webservices')
9. -> on_action( sub {sprintf '%s/%s', @_} )
10. -> proxy('http://services.aonaware.com/DictService/DictService.asmx');
11.  
12.  
13. print $soap->ServerInfo()->result;

End of Code Snippet

Key Points to note
1. Use the tracing and debugging options, this will tell you what is wrong (trust me, it will happen!)
2. URI - This is a reference to the NAMESPACE.... Simple view your WSDL or ASMX service, and view the 'targetNamespace' attribute for the base node.... This references the namespace... This is required when making SOAP calls.
3. PROXY - This is the actual URL to the service or WSDL.
4. on_action - What to do when the action is raised.... similar to an event.... In the example, its simply printing the response. I am also manually printing the response in the code... so you will see output twice.


For reference, there is a really good post here that troubleshoots these issues further.
Microsoft also did a good post here

Perl: Can't locate [Library Here] in @INC - (CPAN Usage)

So i've been looking into Perl this week and I came across an error... quite simple when you know how... this post explains in !English! the options you can take to tackle this... CPAN is utilised and explained in this post.


The Error
SeanMAC:tmp localhome$ perl test.pl
Can't locate SOAP/Lite.pm in @INC (@INC contains: /Library/Perl/Updates/5.8.8/darwin-thread-multi-2level /Library/Perl/Updates/5.8.8 /System/Library/Perl/5.8.8/darwin-thread-multi-2level /System/Library/Perl/5.8.8 /Library/Perl/5.8.8/darwin-thread-multi-2level /Library/Perl/5.8.8 /Library/Perl /Network/Library/Perl/5.8.8/darwin-thread-multi-2level /Network/Library/Perl/5.8.8 /Network/Library/Perl /System/Library/Perl/Extras/5.8.8/darwin-thread-multi-2level /System/Library/Perl/Extras/5.8.8 /Library/Perl/5.8.6 /Library/Perl/5.8.1 .) at test.pl line 17.
BEGIN failed--compilation aborted at test.pl line 17.


So I created a test script and attempted to execute it. It threw up an error at Line 17 complaining that I didn't have the SOAP::Lite library. I.e. "Can't locate SOAP/Lite.pm"

In .NET we would probably either Google the library or use NuGet (NuGet.org - NuGet is a Visual Studio extension that makes it easy to install and update open source libraries).

If we would ever try to compare Perl to the .NET world, we could relate CPAN to NuGet in terms of what it does for us. That is, gets third party libraries, unpacks them and installs them for us.


CPAN
CPAN, the Comprehensive Perl Archive Network, is an archive of over 100,000 modules of software written in Perl, as well as documentation for it.

To use cpan, open up a terminal or command window, and use the following command. This will run cpan as ROOT user. This is sometimes necessary as it requires access to shared library directories.

Code Snippet

1. perl -MCPAN -e 'shell'

End of Code Snippet

You will then be in the cpan shell. Type 'h' for help....

command	argument	description
a,b,d,m	WORD or /REGEXP/	about authors, bundles, distributions, modules
i	WORD or /REGEXP/	about anything of above
r	NONE	reinstall recommendations
ls	AUTHOR	about files in the author's directory
get		download
make		make (implies get)
test	MODULES	make test (implies make)
install	DISTS, BUNDLES	make install (implies test)
clean		make clean
look		open subshell in these dists' directories
readme		display these dists' README files

If we wish to install our library (This includes get/download, make, test and install) then we simply issue the command....

Code Snippet

1. install [Library Name]

End of Code Snippet

with example

Code Snippet

1. install SOAP::Lite

End of Code Snippet

You will be asked a few options, mostly regarding ideal location to download from, connection timeouts etc... You can simply keep hitting enter to use the defaults.

If all is successful, your library will be installed and included within the @INC array (Containing paths to libraries used by Perl).

Happy Scripting!

iOS - Viewing hidden files and directories

Write the following into a terminal window....

Code Snippet

1. defaults write com.apple.finder AppleShowAllFiles TRUE
2. killall Finder

End of Code Snippet

This will show you all of the hidden files and folders on your operating system. If you want to reverse the command replace TRUE with FALSE.

REST Web Services

Good article on RESTful Web Service Architecture

LINQ to SQL vs. ADO.NET Entity Framework

LINQ to SQL and the Entity Framework have a lot in common, but each have features targeting different scenarios in the Orcas timeframe.

LINQ to SQL has features targeting “Rapid Development” against a Microsoft SQL Server database. Think of LINQ to SQL as allowing you to have a strongly-typed view of your existing database schema. LINQ to SQL supports a direct, 1:1 mapping of your existing database schema to classes; a single table can be mapped to a single inheritance hierarchy (i.e. , a table can contain persons, customers, and employees) and foreign keys can be exposed as strongly-typed relationships. You can build LINQ queries over tables/views/table valued functions and return results as strongly typed objects, and call stored procedures that return strongly typed results through strongly typed methods.

A key design principle of LINQ to SQL is that it “just work” for the common cases; so, for example, if you access a collection of orders through the Orders property of a customer, and that customer’s orders have not previously been retrieved, LINQ to SQL will automatically get them for you.

LINQ to SQL relies on convention, for example default insert, update, and delete logic through generated DML can be overwritten by exposing appropriately named methods (for example, “InsertCustomer“, “UpdateCustomer“, “DeleteCustomer“). These methods may invoke stored procedures or perform other logic in order to process changes.

The Entity Framework has features targeting “Enterprise Scenarios“. In an enterprise, the database is typically controlled by a DBA, the schema is generally optimized for storage considerations (performance, consistency, partitioning) rather than exposing a good application model, and may change over time as usage data and usage patterns evolve. With this in mind, the Entity Framework is designed around exposing an application-oriented data model that is loosely coupled, and may differ significantly, from your existing database schema. For example, you can map a single class (or “entity”) to multiple tables/views, or map multiple classes to the same table/view. You can map an inheritance hierarchy to a single table/view (as
in LINQ to SQL) or to multiple tables/views (for example, persons, customers, and employees could each be separate tables, where customers and employees contain only the additional columns not present in persons, or repeat the columns from the persons table). You can group properties into complex (or “composite”) types (for example, a Customer type may have an “Address” property that is an Address type with Street, City, Region, Country and Postal code properties).

The Entity Framework lets you optionally represent many:many relationships directly, without representing the join table as an entity in your data model, and has a new feature called “Defining Query” that lets you expose any native query against the store as a “table” that can be mapped just as any other table (except that updates must be performed through stored procedures). This flexible mapping, including the option to use stored procedures to process changes, is specified declaratively in order to account for the schema of the database evolving over time without having to recompile the application.

The Entity Framework includes LINQ to Entities which exposes many of the same features as LINQ to SQL over your conceptual application data model; you can build queries in LINQ (or in “Entity SQL”, a canonical version of SQL extended to support concepts like strong typing, polymorphism, relationship navigation and complex types), return results as strongly typed CLR objects, execute stored procedures or table valued functions through strongly-typed methods, and process changes by calling a single save method.

However, the Entity Framework is more than LINQ to Entities; it includes a “Storage Layer” that lets you use the same conceptual application model through low-level ADO.NET Data Provider interfaces using Entity SQL, and efficiently stream results as possibly hierarchical/polymorphic DataReaders, saving the overhead of materializing objects for read-only scenarios where there is no additional business logic.

The Entity Framework works with Microsoft SQL Server and 3rd party databases through extended ADO.NET Data Providers, providing a common query language against different relational databases through either LINQ to Entities or Entity SQL.

So while there is a lot of overlap, LINQ to SQL is targeted more toward rapidly developing applications against your existing Microsoft SQL Server schema, while the Entity Framework provides object- and storage-layer access to Microsoft SQL Server and 3rd party databases through a loosely coupled, flexible mapping to existing relational schema.

I know this is a confusing area, and we’re trying to figure out how best to describe these differences to help customers make the appropriate choices. Please let me know if this helps, or if there are still areas of confusion…

Thanks,
Michael Pizzo
Principal Architect
Microsoft Data Programmability