Tuesday, June 16, 2009

Asp.Net and Python-cgi communication

Technorati Tags: ,,

I recently had my first shot at writing cgi scripts in python. It was an altogether different experience having come from a .Net and Turbogears background. I also got my hands on a Linux server. Infact, it was the first time I got to use vi, ls, cat, cp etc, since college. It is really amazing how much you can do with just the shell commands on Linux.


The Linux machine had an apache server which ran some cgi scripts to do some basic tasks like creating a new user record in the database, deleting, editing etc. I think that was the reason they did not go beyond cgi. These tasks were manually done by an admin after logging into the application. Now, these scripts were required to be made callable from an asp.net web application.


ASP.NET application <===========> Python cgi scripts
CreateDemo.aspx _____________ create_demo.cgi
(collects user info
for demo record) _____________ (requires authentication)

So, we decided to create another script, let us say 'authenticate.cgi', in which we used urllib python library to by pass the authentication on create_demo.cgi and then CreateDemo.aspx would call authenticate.cgi. Here is the new flow-

CreateDemo.aspx --> authenticate.cgi --> create_demo.cgi

This write-up addresses two key areas:
1. How to communicate with a python cgi script from an asp.net web page?

//Request url and request method

HttpWebRequest httpRequest = HttpWebRequest)WebRequest.Create("http://192.1.1.1/no_auth/authenticate.cgi?name=harpreet”);

httpRequest.KeepAlive = false;

httpRequest.Method = "GET";

//Read the web response from the URI

string strResponse = string.Empty;

using (WebResponse webResponse = (WebResponse)httpRequest.GetResponse())

{

System.Text.Encoding enc = System.Text.Encoding.GetEncoding(1252);

using (StreamReader srResponseStream = new StreamReader(webResponse.GetResponseStream(), enc))

{

//Read the response into a string or you could use XML also

strResponse = srResponseStream.ReadToEnd();

}

}

2. How to by pass basic authentication on cgi script using urllib?

#!/usr/bin/env /usr/local/bin/python

import urllib2,urllib

import sys,os,cgi,cgitb

import re,datetime

import base64

from urlparse import urlparse

def handler():

form = cgi.FieldStorage()

if form.list:

data = {}

for field in flds:

data[field]=form.getvalue(field,'')

theurl = 'http://localhost/admin/create_demo.cgi’

# you'll need to supply a protected page with your username and password

username = 'scott'

password = 'tiger' # a very bad password

req = urllib2.Request(theurl)

try:

handle = urllib2.urlopen(req, urllib.urlencode(data))

except IOError, e:

# do whatever you want to handle the exception here, 'pass' in this example

pass

else:

# If we don't fail then the page isn't protected

print "This page is not protected by any authentication."

sys.exit(1)

if not hasattr(e, 'code') or e.code != 401:

# we got an error - but not a 401 error

print "This page is not protected by any authentication."

print 'Some other reason for failure'

sys.exit(1)

authheader = e.headers['www-authenticate']

# this gets the www-authenticate line from the headers

# which would contain the authentication scheme and realm

authobj = re.compile(r'''(?:\s*www-authenticate\s*:)?\s*(\w*)\s+realm=['"]([^'"]+)['"]''',re.IGNORECASE)

# this RE is used to extract scheme and realm

matchobj = authobj.match(authheader)

# if the authheader isn't matched by the regular expression

# then there is probably some error in the header

if not matchobj:

print 'The authentication header is badly formed.'

print authheader

sys.exit(1)

scheme = matchobj.group(1)

realm = matchobj.group(2)

# check schem (this example works for basic authorization only

if scheme.lower() != 'basic':

print 'This example only works for BASIC Authorization.'

sys.exit(1)

base64string = base64.encodestring('%s:%s' % (username, password))[:-1]

authheader = "Basic %s" % base64string

req.add_header("Authorization", authheader)

try:

handle = urllib2.urlopen(req,urllib.urlencode(data))

except IOError, e:

print "The username or password is not valid."

sys.exit(1)

thepage = handle.read()

print "Content-type: text/html; charset=utf-8"

print

print thepage

handler()

Security was not really a major concern for this cgi, but an IP based security was added to limit unwanted access.

* The code snippets are not really tested but, will hopefully work :-)

Friday, May 1, 2009

Will the history repeat itself, with Grabbler?


Word games have been around the corner for quite some time now. Some give clues to guess the word, some give letters to make maximum possible words. Each one trying out a different strategy but to obtain a simple objective - form words. I remember to have played a few word games during my childhood days Scrabble, Word Crosswords being the notable ones. Well, times change, paper-pen games are remnants of the bygone days and now its an era of games on Facebook.

On and off, I had been playing Scrabble on Facebook, until i found Grabbler. It is a multi-player game which tests your word making skills, with its unique cubes. Err...on second thoughts, not so unique cubes. Actually, the possiblity that you would get the same cube again in a game is 144 in 26!/20! i.e. 1 in 1,151,150. Hence, the not so unique cubes. Over the last 45 days I have been hooked on to the game. I think I have even played a few moves in my sleep. And, tonight, I have decided I will stop playing it, at least for a few weeks. I would like to catch up more sleep in the coming few days. This decision has come after a lot of deliberations. The addict wanting to sniff more and the reasonable self in me wanting to shun it.

But, this does not stop me from appreciating the game. After a long time, a game caught my imagination. The game has a pleasant interface and has the right combination of skill and luck. It is well matched with a test of patience too. It has been fun, having beaten some very good players and remaining invincible. The eye catching feeds have been such a delight to post on the profile. Although, the game has been introduced very recently, I think this game will definitely climb the ladder pretty fast. Moreover, it is a good change for some one bored by the age old Scrabble.

That reminds me of the fact that Scrabble was created by Alfred Mosher Butts during the Great Depression in the 1930s and the game went on to occupy space in every other household. Almost eight decades later, the world is going through another major economic slowdown and Grabbler has been created. Will the history repeat itself?

Well that remains to be seen, but for now, it is adios Grabbler,for me.

Saturday, March 7, 2009

How I set up my Turbogears based development environment...

The start of 2009 brought along a new project for me. It was based on Turbogears 1.0.4.4 with Postgresql 8.3 as the database. Here is a list of steps I followed to setup my development environment.

1. Installed Python 2.4.4.

2. I was working on a windows platform and required support for it so I installed extensions for windows. PyWin32 is a collection of modules for advanced windows-specific support. These include utilities for COM, Win32 API calls, Registry, Event Log and MFC user interfaces.

3. Installed setuptools, so as to be able to build and distribute packages. It includes a utility easy_install, which lets you automatically download, build, install, and manage Python packages and their dependencies.

4. Installed the Turbogears web application framework by running the command:

>easy_install-2.4 -f http://files.turbogears.org/ "TurboGears==1.0.4.4"

5. Installed win-psycopg, a windows port of the psycopg python-postgresql database interface.

6. Installed SQLObject, an Object Relational Manager which provides an object interface to your database, with tables as classes, rows as instances, and columns as attributes. Used the following command to install it:

>easy_install-2.4 "SQLObject==0.10.2"

7. Installed other packages like PyPDF 1.9, ElementTree 1.2.6, ElementTidy and PyLucene for my text based application.

8. Installed TinyMCE 1.0.6, a JavaScript HTML WYSIWYG editor control, using the following command:

>easy_install-2.4 "TurboTinyMCE==1.0.6"

9. Downloaded the ExtJS 2.2 Library. It has some excellent built-in components ready to use.

10. Installed Eclipse 3.2 IDE and added the PyDev plugin using the manual and configured it accordingly.

There I was, ready to play!

Friday, March 6, 2009

Remote access to PostgreSQL Server

Error: "could not connect to Server: Connection refused"

The resolution is a five step process : -

1. Firstly, check if your firewall allows the port (default is 5432) to establish a connection.

2. By default, PostgreSQL does not allow remote incoming connections. This has been implemented in this way, because of security concerns. You need to enable incoming TCP/IP connection requests from the client. This can be done by adding

#
listen_addresses = '*'
#

in the configuration file postgresql.conf. This configuration is valid for postgresql 8.x.

3. You will have to inform Postgresql who can connect to the server. This has to be done by modifying the configuration file pg_hba.conf

host all all 127.0.0.1/32 trust
host all all 192.168.0.0/16 trust

This would allow any user on the local network to access all the databases on the server in "trust" mode.

4. After you have saved the configuration files you will have to restart the server. Remember, these configuration files are read on server startup.

5. Test your config. Go to the command prompt and run the following command.


>psql -h serverIP -U postgresUser -d postgresdb


And, you should be set for remote access to postgresql server.

Thursday, March 5, 2009

JSON

JSON is an acronym for JavaScript Object Notation. It is a means of transferring serialized JavaScript objects so that a JavaScript application can evaluate them and transform them into JavaScript objects which, the application can utilize to perform its operations. In short, JSON is a data interchange format. It is easy to parse and generate in almost any language.

A sample JSON:-
{"continents": {
"asia":[{"country": "india",
"code": "101"
},
{"country": "japan",
"code": "102"
}],
"europe":[{"country": "france",
"code": "401"
}]
}
}


The most notable features of JSON are:-

1.
It is basically a collection of name-value pairs ("key": "value") separated by a colon.
2.
It is not only easy to read for humans but also easy to parse by the machines.
3. Values can be a string, an integer, an object or an array.
Standard literals like True, False, null are also acceptable.
4. Each object is placed between {...}.
5. Arrays are place between [...] and are comma separated.
6. It can be passed using httpWebRequest.
7. Reserved keywords from JavaScript are not allowed in JSON.
8. JSON can be parsed using the eval() method in JavaScript.
9. It uses Unicode.


Common errors like the "missing } after property list" can be corrected
using a JSON validator like www.jsonlint.com




Tuesday, March 3, 2009

Existing database tables and SQLObject

Define the existing table "Bookmark" as a class in python:-

class Bookmark(SQLObject):

_fromDatabase = True # the table structure will be loaded from the database
BookmarkName = StringCol (dbName="bookmark_name")
# dbName is the name of the column in the database, BookmarkName is the name reference in code.
ProjectId = IntCol(dbName="project_id")

(Make sure the indentation is correct)

Connecting to an existing database using TurboGears framework

Your application would need a DB connection to access the database. A connection is how you tell SQLObject to locate your database. SQLObject is an object relational mapper, a software layer that maps data stored in a relational database to an object model in an object-oriented programming language. The connection string resembles a URI in the following format:


scheme://[user[:password]@]host[:port]/database[?parameters]
For instance:
postgres://user:pwd@localhost:5432/db


SQLObject provides a debug mechanism, which can be turned on easily by appending ?debug=True to the end of your database URI in the dev.cfg file.

postgres://user:pwd@localhost:5432/db?debug=True
This would publish the query generated onto the console.
..........

Define the existing table Bookmark as a class in model.py:-

class Bookmark(SQLObject):

_fromDatabase = True # the table schema be loaded from the database

BookmarkName = StringCol (dbName="bookmark_name")

ProjectId = IntCol(dbName="project_id")

# dbName is the name of the column in the database, BookmarkName is the name reference in code.

..........
Using the following code in controller.py you should be able to retrieve the second record from the Bookmark table

 
b=Bookmark.select()
print ": %s " % b[2].BookmarkName