Monthly Archives

December 2012

Cool Stuff

Creating Progress Bars with Python

December 31, 2012

Ever since I can remember, progress bars had my interest. I think it is a neat way to convey to the user that his or her request is being processed. Furthermore, progress bars can give an indication on the percentage of the task that has been completed and/or timing when applicable. In this post I want to explore different types of progress bars and how you can incorporate them in your program. If you are really interested in the history of progress bars, they pre-date the modern computer back to 1896 and are connected with Gantt Chart. For more information of the topic I advise a Google search for progress bars or the Wikipedia entry for progress bar.

The first thing you should know about progress bars is when to use them. Just like any code, they have a certain overhead (negligible) and just because you can does not mean you should. So when should a programmer use a progress bar? When the task requires more than a reasonable time. You will know what a reasonable time is based on the program and your test runs. Along their other usage, progress bars can indicate to the user that work is being done and that the computer did not "freeze" on them. As a rule of thumb, try to used the progress bar when a large task is on hand, not something small. Loading a file is reasonably fast (most of the time). Loading 20GB of files into the memory of the program might take some time. In the latter scenario you probably should use a progress bar. It is very frustrating to the user when they hit ‘enter’ or ‘return’ and nothing happens instantly. Give the user some comfort knowing their request is being processed.

wpid-progress_bar-2012-12-29-19-192.gif

To simplify things I will focus on command line interfaces using Python2.6 (Documentations for python2.6 for offline reference). The same ideas presented here can be applied to GUI interfaces, but that is not trivial. Nothing with GUI development is trivial at all. The first thing to do when considering using a progress bar is to examine the problem. Different programs need different types of progress bars. There are three large sub-categories to all problems that require progress bars:

  • Problems that are solved by doing smaller identical tasks of known quantity
  • Problems that are solved by doing multiple tasks but known quantity.
  • Problems that are solved by doing unknown number of tasks and/or unknown quantity.

Lets start with the first category. Lets assume that we have a function called do_task() that does something. For the purpose of this post all the function will use python sleep function for a small amount of time. That will emulate as if the program was accomplishing a real task. We are going to invoke the function multiple times and each time it completes we will print a dot ('.'). The code will look something like this:


import time
import sys

def do_task():
	time.sleep(1)

def example_1(n):
	for i in range(n):
		do_task()
		print '\b.',
		sys.stdout.flush()
	print ' Done!'
	
print 'Starting ',
example_1(10)

The output will look like this:

wpid-pythonprogressbarexample1-2012-12-29-19-192.png

For a small amount of tasks, something like this will suffice. However, what if the program had to run 100, 1,000 or 1 million times. Clearly a million dots ('.') printed out to the screen will not be a good idea. We can modify the program to print a single dot ('.') for every 10% compilation of the task. Note that at any point in any of the examples discussed in this post the dot (‘.’) can be replaced by whatever you want. It can be an ‘x’ or a ‘#’ or any other printable ascii character will do.


import time
import sys

def do_task():
	time.sleep(1)

def example_1(n):
	steps = n/10
	for i in range(n):
		do_task()
		if i%steps == 0:
			print '\b.',
			sys.stdout.flush()
	print ' Done!'
	
print 'Starting ',
sys.stdout.flush()
example_1(100)

python loading bar mid way

Now let us add some more information about the process. First lets add some brackets so the user has some indication of how many more dots they are about to see. Notice that in previous examples and in the next examples we utilize sys.stdout.flush(), suppressing the carriage return using a comma (‘,’) at the end of the print statement and print a special character, ‘\b’. The special character ‘\b’ returns the printing cursor one step backwards. This is what allows us to print ahead and move backwards. Also note that every time we use the python print statement with a comma, the next print statement has a space in front of it. This is way using the special character ‘\b’ becomes necessary. Take a look at the next example:


import time
import sys

def do_task():
	time.sleep(0.1)

def example_1(n):
	steps = n/10
	for i in range(n):
		do_task()
		if i%steps == 0:
			print '\b.',
			sys.stdout.flush()
	print '\b]  Done!',
	
print 'Starting [          ]',
print '\b'*12,
sys.stdout.flush()
example_1(100)


And the output mid way and at the end:

wpid-pythonprogressbarmidway-2012-12-29-19-19.png

wpid-pythonprogressbarwithbracketsend-2012-12-29-19-19.png

After we have gone through the basics of the first type of loading bars lets move on to the second. Let us assume that we need to complete x tasks and each one is similar in time. For our code we can emulate this by using the sleep function again. Since the tasks are not the same tasks, we will not have the loop. Instead, we will call the functions one at a time. After each task is complete we will update our progress bar.


import time
import sys

def update_progress_bar():
	print '\b.',
	sys.stdout.flush()
	
print 'Starting ',
sys.stdout.flush()

#task 1
time.sleep(1)
update_progress_bar()

#task 2 
time.sleep(1)
update_progress_bar()

#task 3 
time.sleep(1)
update_progress_bar()

#Add as many tasks as you need. 

print ' Done!'

wpid-pythonmultitaskprogressbar-2012-12-29-19-19.png

We have moved forward in this example to create a function that updates the progress bar and to the second category of progress bars. Although by now we can write our own progress bar, let us take a look at some libraries that are already written nice progress bars you can use.  The first one is by Anler Hernández Peral and available through ActiveState Recipes and progress bar v.1 code. Running the code gives a quick example like this:

wpid-progress_bar_1exampeoutput-2012-12-29-19-19.png

In order to utilize the code for your own program you will have to import it. Following the example for ActiveState, Here is 2 ways you can use the code and their output: (Note: I have named the module progress_bar_1.py)


from progress_bar_1 import ProgressBar

p = ProgressBar()
print p

for i in range(10):
	print p+1

wpid-pythonprogress_bar_1output_1-2012-12-29-19-19.png


from progress_bar_1 import ProgressBar

custom_options = {
	'end': 10,
	'width': 10,
	'fill': 'x',
	'format': '%(progress)s%% [%(fill)s%(blank)s]'
}

p = ProgressBar(**custom_options)
print p

for i in range(10):
	print p+1


python progress_bar_1 output_2.1

Note that now we have moved into creating an object for our progress bar. That allows us to update and print it as we see fit. It also allows for some more custom setting as we can see in the second example. I will let you explore those on your own, but if you need help please contact the Captain. The second library will introduce some more fancy output. It is called progressbar 2.2 and the code for progressbar2.2. The code for it is a little lengthy (about 360 lines), so I will not include it in this post. Running the module as is, yields 4 example outputs that should look like this:

progressbar examples output

A closer look at the examples will provide you all the information you need to use this module. The ideas are very similar to the last example where you have a progress bar object you instantiate and then progress as needed. In the examples it was included in a for loop, but may be applied for multiple tasks as well.

After we have examine all these example, lets move to to the final category, the unknown one. Let us assume our program will need to complete an operation that either we do not know how many steps are involved or how long it will take. The solution is rather simple, a spinning loading. We still acknowledge to the user that the process is being executed, but cannot provide any insight as to the progression. Please note that you should note use this progress bar unless it is an absolutely must.


import sys
import time

print 'Loading....  ',
sys.stdout.flush()

i = 0

while i <= 10:
	if (i%4) == 0: 
		sys.stdout.write('\b/')
	elif (i%4) == 1: 
		sys.stdout.write('\b-')
	elif (i%4) == 2:
		sys.stdout.write('\b\\')
	elif (i%4) == 3: 
		sys.stdout.write('\b|')

	sys.stdout.flush()
	time.sleep(0.2)
	i+=1
        
print '\b\b done!'


loading bar mid way

Similar to the examples above, if we want to use this in an application we can build a class around. Unlike the other examples in this case you will have to use threads in order to run the spinning loading bar while the program is running. A small note, you will not be able to print from the code that is running as it will mess up the loading bar. Also notice that now we are using while loops as oppose to the for loop we used before. This is because now we do not have a measure to estimate compilation of the program. The output will be the same as the example above.


import sys
import time
import threading

class progress_bar_loading(threading.Thread):
    
    def run(self):
            global stop
            global kill
            print 'Loading....  ',
            sys.stdout.flush()
            i = 0
            while stop != True:
                    if (i%4) == 0: 
                    	sys.stdout.write('\b/')
                    elif (i%4) == 1: 
                    	sys.stdout.write('\b-')
                    elif (i%4) == 2: 
                    	sys.stdout.write('\b\\')
                    elif (i%4) == 3: 
                    	sys.stdout.write('\b|')

                    sys.stdout.flush()
                    time.sleep(0.2)
                    i+=1
                    
            if kill == True: 
            	print '\b\b\b\b ABORT!',
            else: 
            	print '\b\b done!',

      
kill = False      
stop = False
p = progress_bar_loading()
p.start()

try:
	#anything you want to run. 
	time.sleep(1)
	stop = True
except KeyboardInterrupt or EOFError:
         kill = True
         stop = True


So now what? Now you know how to create your own custom progress bars in python. You may choose to implement the principles and I might do so myself. If I do I will make sure to post it up. As a last point I would like to mention the usage of progress bars once more. As a programmer you are responsible for the human computer interaction. Progress bars are a tool to convey to the user to stand by. Please use progress bars to actually mean something. Do not just display dots like the first examples. Try to estimate how long it will take, how much of the task has been accomplished, anything at all you can tell the user might help. As a user, there is no progress bar more annoying then ones that tell you nothing. In 2006, Peter Beddow wrote an article On the Abolition of the Computer Progress Bar. I suggest you read that article before using progress bars in large scale applications. Let us as programers bring back the loading bars the actually help the user, not that ones that annoy everyone.

Holiday

Some Christmas Light Madness

December 29, 2012

Cool Stuff

Playing around with Strings and Python

December 28, 2012

Python is my favorite language. It allows anything from high level to low level programming. It is very easy to understand. Python can be written as an algorithm that you can execute. Today, I want to show you some cool tricks on how to manipulate strings in python.

NOTE: I am running Python2.6. I highly recommend you read the Python 2.6 String documentation prior to proceeding. I suggest you also download python2.6 documentation for offline reference.

Python stores strings in it’s virtual memory as arrays. This is very useable since you can address each character individually. Strings are 0 based arrays and support reverse indexing. However, unlike arrays, strings are not mutable. Strings also support concatenation and other cool stuff. Let’s take a look at the following output:
wpid-wpid-pythonstringmanipulation-2012-12-28-01-53-2012-12-28-01-53.png
Now, lets have fun with the output and give it some more meaning. First lets ask the user for a string and print out the characters of the string each on a line. To do so we will run this code:


a_string = raw_input("Enter some text: ")
for char in a_string:
	print char

Running the program should give you the following output for the input “Hello World!”:

wpid-wpid-pythonprintcharactersofstring-2012-12-28-01-53-2012-12-28-01-53.png

Now lets use the python built in sleep function from the time library. This will delay the printout of each char by a given time. This will cause a neat effect for the user.


import time

a_string = raw_input("Enter some text: ")

for char in a_string:
	print char
	time.sleep(1)

So far it is still dull. Let’s take it a step further and have colors. There is a great library to add colors to python standard output called termcolor. I suggest you take a look at the code for termcolor. To use it we can either create a text object and color it, or use the provided cprint function. Here is the example of printing in res through standard output:


import time
import termcolor

a_string = raw_input("Enter some text: ")

for char in a_string:
	termcolor.cprint(char, "red")
	time.sleep(1)

wpid-wpid-pythonstringredcolor-2012-12-28-01-53-2012-12-28-01-53.png

Termcolor also support printing on color, blinking, bold, italic and others. There is a brief example on the termcolor web page on how to use. That should cover anything you need to know. If I get the time I will come back and write a tutorial for it. If you have questions before such time, feel free to contact the Captain. For the next one, we will print each letter in a different color, like this:


import time
import termcolor

a_string = raw_input("Enter some text: ")
i = 0

for char in a_string:
	if (i%7 == 0):
		termcolor.cprint(char, "grey")
	elif (i%7 == 1):
		termcolor.cprint(char, "red")
	elif (i%7 == 2):
		termcolor.cprint(char, "green")
	elif (i%7 == 3):
		termcolor.cprint(char, "yellow")
	elif (i%7 == 4):
		termcolor.cprint(char, "blue")
	elif (i%7 == 5):
		termcolor.cprint(char, "magenta")
	elif (i%7 == 6):
		termcolor.cprint(char, "cyan")

	time.sleep(1)
	i += 1

wpid-wpid-pythonprintingindiffrentcolors-2012-12-28-01-53-2012-12-28-01-53.png

Now what if you want to print in the same line? Well, we will need to use another standard library called sys.stdout. What we will do is suppress the carige return from the print statement by placing a comma (‘,’) at the end of the print statement. Without flushing the print statement, the output will just appear after the entire line was printed out. Since we want the suspense, we add the flush. Thus printing the input string back to standard output, one character at a time. Check it out:


import time
import termcolor
import sys

a_string = raw_input("Enter some text: ")

for char in a_string:
	print char,
	sys.stdout.flush()
	time.sleep(1)

wpid-wpid-pythonprintacharacteratatime-2012-12-28-01-53-2012-12-28-01-53.png

If we want a different color on each character and have them all on the same line we will need to use the object colored and the regular print statements like before. The code will look like this:


import time
import termcolor
import sys

a_string = raw_input("Enter some text: ")
i = 0

for char in a_string:
	if (i%7 == 0):
		print termcolor.colored(char, "grey"),
	elif (i%7 == 1):
		print termcolor.colored(char, "red"),
	elif (i%7 == 2):
		print termcolor.colored(char, "green"),
	elif (i%7 == 3):
		print termcolor.colored(char, "yellow"),
	elif (i%7 == 4):
		print termcolor.colored(char, "blue"),
	elif (i%7 == 5):
		print termcolor.colored(char, "magenta"),
	elif (i%7 == 6):
		print termcolor.colored(char, "cyan"),

	sys.stdout.flush()
	time.sleep(1)
	i += 1

The screen capture:

wpid-wpid-pythonsamelinediffretcolorcharacters-2012-12-28-01-53-2012-12-28-01-53.png

That is all for. Now you have some extra tools in you arsenal to making some neat command line programs using python. If you are really into it I suggest you take a look at the following program, it is really sci-fi.

download python string sci-fi


# written and executed on mac OSX 10.6.8 running python 2.6.8

import time
import termcolor
import sys
import random

colors = ("grey", "red", "green", "yellow", "blue", "magenta", "cyan")
attributes = ("bold", "underline", "blink", "reverse")

def get_color():
	global colors
	return colors[random.randint(0,len(colors)-1)]
	
def get_on_color():
	global colors
	return "on_" + colors[random.randint(0,len(colors)-1)]
	
def get_attributes():
	global attributes
	return attributes[random.randint(0,len(attributes)-1)]
	
for i in range(100**2):
	x = random.randint(0,6)
	the_char = chr(random.randint(33,126))
	if (x%6 == 0):
		print termcolor.colored(the_char, get_color(), get_on_color(), attrs=[get_attributes()]),
	elif (x%6 == 1):
		print termcolor.colored(the_char, get_color(), get_on_color()),
	elif (x%6 == 2):
		print termcolor.colored(the_char, get_color()),
	elif (x%6 == 4):
		print termcolor.colored(the_char, get_color(), attrs=[get_attributes()]),
	elif (x%6 == 5):
		print termcolor.colored(the_char, get_on_color(), attrs=[get_attributes()]),
	else:
		print the_char,
	
	sys.stdout.flush()
	time.sleep(0.3)


Screen capture of the string secret program - 1

Screen capture of the string secret program - 2

Uncategorized

Steganography in Images

December 26, 2012

(Note: I am by no means an expert on this matter, I am simply expressing my personal experience with the subject. For more information Steganography or Google would be a good start)

Steganography is the art of hiding message in such way that no one beside the intended receiver will find (if done correctly). This is by no means a new idea, the history of Steganography can be traced back to the 15 century. For our purposes, I am going to discuss Steganography in the general sense in the computer world, mostly done in Images, but not limited to them. For this example I am only going to consider 256 gray and colored images for simplicity of example, this can be done however with other images and data as well.

How information is stored?

Any computer data today consists of bits of data, 0 or a 1. A bunch of them constitute a byte, which is normally 4 bits. The length of a bit depends on a number of things such as implementation, system, usage and other factors. An image is usually stored as one of two options:

1. Black & White (Grayscale) 
2. RGB colors (Normal)

In both cases the image is built from pixels, or a dot in a 2 dimensional array that has a certain value. In case of option 1 it would be on a scale from 0-255, 0 being black. For option 2 there aren three values that constitute one color as output, Red - Green - Blue, each with the same scale of values from 0 - 255, 0 being black.

How to hide?

So lets say we have the following image (Grayscale) stored in memory as the following pixles:

wpid-GrayscaleWhiteimage-2012-12-24-15-35.png

This would give us an empty 4 X 4 pixel white image. Keep in mind that any of these cell blocks is actually stored in memory as 255 = 11111111 in binary. Now you would agree that the character ‘A’ is equivalent to 65 in ascii which in binary is equal to 01000001. You would also agree that if we change one of these cell values to 254 it would be unnoticeable to the naked eye.

The ‘Magic’ bit...

We are going to store the individual values of A’s binary representation in the least significant bits of the image values. This means we would go over the cell or pixels of the image and change only the last bit of the value, changing the color by only 1. The cells of an image are numbered as follow:

wpid-ImageCellNumbers-2012-12-24-15-35.png


In order for to be stored in memory we will used the first 8 bits, changing the values in the LSB to match to ‘A’. The result will still be a white image, stored in memory like this:

wpid-HidingTheLetterAinanImage-2012-12-24-15-35.png

Color and Extraction

The same idea we applied to this example could be applied to an RGB file or any other data type. One could also encrypt the message hidden so even if it is found by anti-Steganography software it would be useless. Based on how you hide the memory you would also need an extraction code. For our example we would simply need to extract and concatenate the first 8 LSB and convert them back to a character, which would give us ‘A’.

Holiday

Happy Holidays

December 24, 2012

Wishing everyone Happy Holidays & a great New Year!

wpid-holidayspirit-2012-12-24-17-33.jpg

Programming Languages

Programming Languages to a Non-Programmer

December 24, 2012

As a non-programmer, you have often heard programming languages names being used. Marketers have always tried to used technical words to market new products and to claim superiority using “fancy” words. Many have come across HTML, java, flash, but do not know what that means. It is to my intend to educate those who may not know.

As for the programmers among you, many of us do not go a day without using several programming languages. If you really think about it, after the first few languages, learning new ones become easier. This may not encompass human languages, but learning new computer languages became easier for me with time. One thing that helped me the most was learning about the different types of programming languages. Knowing where to categories a new language helped me relate things I already know with new information. Hopefully you too will find this true. Learning new programming languages is a vital tool any programmer needs to acquire.

A Programming language is a communication method for humans to interact with machines. That is a very high level view. Machines, computers to be exact, work on binary sequences. That is strings of 0’s and 1’s. At the end of the day that is all computers understand. Further more, the entire operations a computer knows how to do is limited to operations on those strings of 0’s and 1’s. Any more complicated operations is done via multiple operations on said strings. Thankfully, we can store and reuse a sequence of operations. This is the closet level of operations, Machine Language. In the early days of computing, computers were programmed in such method. However, it is very impractical to program anything complex in this matter. Every set of commands in any languages must end up as a Machine Language instruction set suitable for a target computer architecture.

To take a step farther form Machine Language, we created Assembly. Assembly is the attempt to combine machine language operations into symbolic ones. It often group several Machine Language instructions into a single Assembly command. This enables programmer to create more complex programs, however still resides close to Machine Language. It provides better readability, with the same efficacy of Machine language. Assembly instructions are compiled into their Machine Language representations in order to be executed on a target architecture.

The next step away from Machine Languages are the low and high level programming languages. Some are easy to specify as while others are in-between. Languages that allow direct access to memory are often considered low. Languages such as c, c++, java to some extend, are low level. High level languages include matlab, matamitica, php, and other. Python for example, could be argued to be somewhere in-between. Depending on the programmer and the libraries used, python could be utilized as a low level or a high level language.

There are many types of programming languages. The Wikipedia entry for List of programming languages by type counts 43 languages as of the date of this post. Some of the ones I highly recommend on learning are (in no particular order of preference):

  • c, java, python
  • Lisp
  • Prolog
  • web langs: php, javascript, mysql, html, css
  • matlab