Web Scraping

Web Scraping using Python

Hi all, i'll be telling how to start web scrapping in python. In this script we are using accuweather to scrap data, to check whats the current temperature of your city.


   We are going to use two module on this tutorial.

1) Beautiful soup is the module, that we will use for scrapping.

2) Requests is to get the data from webpage.

Make sure to check if Beautiful soup is installed in your system, as below. In case if it's not installed, google for how to install beautifulsoup and requests.

shanky@Unity:~$ pip list |grep beautifulsoup4

beautifulsoup4 (4.5.3)

Code:

#imports  

from bs4 import BeautifulSoup as bs

import requests

#requests.get() function connects to given url, and collects the web page data.

#change url with, go to accuweather, and select your city.
#example, if you were in banglore, URL would be "http://www.accuweather.com/en/in/bengaluru/204108/weather-forecast/204108"

page = requests.get("url")

#now we need to extract content from the data we just got. and parse it in html format. This will enable to use, search what we are looking for using html tags.

soup = bs(page.content, 'html.parser')

#Here we are searching for a class="large-temp" first instance in the HTML we just extracted. you can check HTML from website, just right click and choose Inspect element from your browser.

#get.text() collects the string, inside the searched tag.

x=soup.find_all('span', class_="large-temp")[0].get_text()

#here we need to use slicing to get the required data, also the string i got was encoded to UTF-8 format, so we need to encode it back to ascii.

print ("Current temperature is " + str(x[0:2].encode()) + "C")

This is it, for more information, contact me or research on BeautifulSoup for web scrapping, it packs way too many powerful functions.

FYI, i am using accuweather for educational purpose only. you need to modify the code as per the website, if you are using other website.

Text Based Adventure Game

Text Based Adventure Game

Hi All, this one is a text based adventure game, and the best thing is you can write the story as you like and the same code will work for it.

This one consist of 3 files.
1) Code
2) Story line input file in text format
3) Text file than contains all the information about story.

I'll come to code later, but first information text file.
This file contains one line for every scenario for you story.

Eg.

story.txt

Hi, I am Naruto. I want to be Hokage(Village leader).

Day 1, It's finally the day i'll become Genin. I need to take the test first.

(At Examination Hall) Oh no ! , I can't do this jutsu, do you know which jutsu is this.

1) Shadow Clone jutsu.

2) Fire ball jutsu.

lines.txt

3 2 1

lines.txt first line is 3 2 1, it provide information how to read story.txt. 3 means we have 3 line for story setup, like in above example. it can be any number as long as you take care of the screen size. 2 means we are going to provide 2 options for the user. And last 1 means that choice one is correct.

so you can create you own story, you just have to follow the rule as above.

Code:

import cursesfrom time import sleep

#function to read story and check if user is given the correct answers

def story(scr): scr.refresh()

#reads both the file in read only mode.

strText=open('/path/to/file/story.txt','r') strLine=open('/path/to/file/lines.txt','r')

#reads line by line from lines.txt

for x in strLine:

x=x.rstrip('\n')      #removes \n from end of line

x=x.split(' ')        #split all elements line by line, on the basis of space

#read line by line from story.txt on the basic of number of lines from lines.txt 

for y in range(len(x)):

#checks if the element is last one, also checks, if your input is correct

if y == len(x)-1: if chr(usr_input) == x[y]: continue else:

#in case user input is wrong, game end.

story_draw(15,scr, "Nope, wrong choice, you have to watch Naruto Again.") scr.getch() curses.endwin() exit() else: for y1 in range(int(x[y])): a=strText.readline().rstrip('\n') story_draw(y1+3,scr,a) usr_input=scr.getch() scr.clear() scr.border() scr.refresh() scr.refresh() scr.getch()

# function to print string for output on curses.

def story_draw(y,win,a): for x in range(len(a)): win.addstr(y,x+5,a[x])

#just to add little animation 😆

sleep(.01) win.refresh() return
def main(): screen = curses.initscr() screen.border() screen.keypad(1) curses.noecho() curses.cbreak() curses.curs_set(0) story(screen) curses.endwin()


if __name__ == "__main__":
    main()

Screenshot 1: This is how story looks.

Screenshot 2: Example of how the options can be.

Screenshot 3:Scene 2 starts, if you choose correct answer.

Screenshot 4: Just in case you loose.

Side note : Yes i like Naruto way too much.
Author Shanky.Rawat

Menu based Basic Calender Management System

Menu based Basic Calendar Management System

We will be working on super simple, hourly based, basic calendar management system. We will be using dictionary, data structure provided by python.

#function to display all events in calendar

def event_disp():

   system('clear')

   print ("\n\n#####################################\n\n")

   print ("Time : \t\tEvent")

   for key, value in hour.items():

      print key, "\t\t" + value

   print ("\n\n#####################################\n\n")

   return

#function to display main menu and taking user inputs

def disp():

   # using system in built command for clearing screen

   system('clear')

   print ("\n\n#####################################\n\n")

   print ("\t1. Add event to calander")

   print ("\t2. Remove event from calander")

   print ("\t3. View Events from yout calander")

   print ("\n\n#####################################\n\n")

   userInput = int(input("Choose 1,2 or 3 for options:\t"))

   if userInput == 1:

      inputTime = int(raw_input("Enter time in format (hh):\t"))

      if inputTime >= 24 or inputTime < 0:

         print "Time is only supported from 00 to 23 hours"

      else:

         #checking if the event already exist for the same time.

         if not hour.get(inputTime):

            inputEvent = raw_input("Enter event to store:\t")

hour[inputTime] = inputEvent

         else:

            print ("Event is already schedule for this time.")

   elif userInput == 2:

      event_disp()

      inputTime = int(raw_input("Enter the time you want to clear:\t"))

      #deleting the event using key from dictionary 

      del hour[inputTime]

      event_disp()

   elif userInput == 3:

      event_disp()

   cont=raw_input("Back to main menu(y/n):\t")

   if cont == 'y' or cont == 'yes':

      disp()

   else:

     exit()

#importing system from os, for clear command we used earlier

from os import system

#Adding some random input for our dictionary.

hour = {11:"This is sparta", 12 : "Sleep"}

disp()


Screenshots:

1 Main screen and adding event

2 View Events

3 Removing event

4 List after removing event

Password Generator and Validation

Password Generator and Validation

Today, we will be writing code for password generator or creator, and also a validation of the given password.

We will be creating three different programs today.

1st : Password Generator

Code:

#using random function to take characters from given set, though it can generate week passwords also.

import random

s = "abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%^&*()?"

#make sure output password is 8 character long

passlen = 8

p =  "".join(random.sample(s,passlen ))

print p

2nd : Password Generator

Using inbuilt libraries

Code:

import string
import random

#creating a function to create a password using string and random function.

def pw_gen(size, chars=string.ascii_letters + string.digits + string.punctuation):
    return ''.join(random.choice(chars) for _ in range(size))

print(pw_gen(int(input('How many characters in your password?:\t'))))

3rd : Password validation

This forces, to put below specification in given password

• At least 8 character long.
• Have at least 1 digit, 1 lower, 1 capital letter.
• Have at least 1 special character in it

Code:

import re

def validate():

    while True:

        password = raw_input("Enter a password: ")

        if len(password) < 8:

            print("Make sure your password is at lest 8 letters")

        elif re.search('[0-9]',password) is None:

            print("Make sure your password has a number in it")

        elif re.search('[A-Z]',password) is None: 

            print("Make sure your password has a capital letter in it")

        elif re.search('[@#$%^&+=]',password) is None: 

            print("Make sure your password has a Special character in it")

        else:

            print("Strong password")

            break

validate()

Caesar Cipher

--- Caesar Cipher---

Hi,

    Today we will be implementing Caesar cipher algorithm for encryption. Caesar cipher is one of the basic encryption technique. that uses a make shift number to interchange the position of text. 

for E.g. 

if the shift value is 2,

A --> C, B--> D, c-->e, d-->f and so on...

and if shift value is -2

A --> Y, B--> Z, c-->a, d-->b and so on...

Code:

#getting input from use for text to encrypt and shift value

text = raw_input("Please enter the text to be converted..!! : \n");

change=input("Enter the shift number for encryption: \n")

#using list to save the conversion, for end result

new_text = []

#using Caesar cipher algo, we are converting upper and lower case alphabets separately, and ignoring others

for x in text:

        if x.islower():

                new_text.append(chr((ord(x) + change - 97) %26  + 97))

        elif x.isupper():

                new_text.append(chr((ord(x) + change - 65) % 26 + 65))

        else:

                new_text.append(x)

print "Text : \t\t"+text

#using join inbuilt function for text join from list

print "New Text :\t"+''.join(new_text)

E.g:

Please enter the text to be converted..!! :

THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG

Enter the shift number for encryption:

2

Text :          THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG

New Text :      VJG SWKEM DTQYP HQZ LWORU QXGT VJG NCBA FQI

We can use to decipher the encrypted code  by same algorithm. we need the shift value or key by which it was encrypted earlier. for above example we use shift of 2. so, for decryption we need to use same algorithm with reverse shift value, for this case -2.

E.g:

Please enter the text to be converted..!! :

VJG SWKEM DTQYP HQZ LWORU QXGT VJG NCBA FQI

Enter the shift number for encryption:

-2

Text :          VJG SWKEM DTQYP HQZ LWORU QXGT VJG NCBA FQI

New Text :      THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG

MasterMind

--- MasterMind --- 

Number guessing game

The player enters 4 digit number, and the computer tells the player how many(but not which) of the 4 digits are correct.

Number changes randomly every time, when new game starts.

E.g.

Guess the number in few tries as possible

Enter the number :      1234
Enter the number :      4567
*
Enter the number :      7533
***
Enter the number :      7593
you used 25 chances, to guess.

CODE

#imports required for validation, i was using older python version for print, i used below import

from __future__ import print_function

import re

from random import randint

print ("Guess the number in few tries as possible\n");

number=[]

inp=str(randint(0000,9999))

for x in inp:

        number.append(x)

check = True

count=0

#loop until guess is correct

while check == True:

        guess=[]

        input=raw_input("\nEnter the number :\t");

        count+=1

#validation if the input is numeric 

        if not re.match("^[0-9]*$", input):

                print ("\nError! Only numeric value allowed")

                continue

# validation if the number is 4 digit longs or not

        if len(input) != 4:

                print ("\nError! Must be 4 digit long number")

                continue

        for x in input:

                guess.append(x)

        if guess == number:

                check = False

                break

        for i in range(4):

                if number[i] == guess[i]:

                        print ("*", end='')

print ("\nYou used {0} chances, to guess".format(count))

Basic Mp3 Player in Python

You'll need to test for below modules if present in your linux machines

1. Python - Just type python and you'll get something like below if you already have python installed.

     2. gst-launch-0.10 - Try below command and you will hear a Bell ringing, if not try to install it.
               gst-launch-0.10 audiotestsrc ! audioconvert ! audioresample ! pulsesink

     3. Python curses module - Try import curses in python.

Snapshot of the player:

Code:

#list all songs from the given directory    
def song_input():
    songs = (subprocess.check_output(["locate","-e","-L","*.mp3"]).split('\n'))
    return songs

#Play arugmented passed song using Popen and gst-launch-0.10
def player(Song):

    #Check if any process is running song, if yes then kills it before starting new one
    if processes:
        processes.pop().terminate()
    song="location="+Song
    song_process = subprocess.Popen(["gst-launch-0.10","-q", "filesrc", song," ! mad ! audioconvert ! alsasink"])
    processes.append(song_process)
    return

#curses used to select song from terminal based UI, then sends request to play to played function
def song_selection():
    screen.border()
    screen.nodelay(0)
    curses.noecho()
    selection = -1
    song_name = []
    count = 0
    song_name = song_input()
    while selection < 0:
        screen.clear()
        screen.addstr(dims[0]/2-2,dims[1]/2-len(song_name[count - 2].split('/')[-1])/2,(song_name[count - 2]).split('/')[-1])
        screen.addstr(dims[0]/2-1,dims[1]/2-len(song_name[count - 1].split('/')[-1])/2,(song_name[count - 1]).split('/')[-1])
        screen.addstr(dims[0]/2  ,dims[1]/2-len(song_name[count    ].split('/')[-1])/2,(song_name[count    ].split('/')[-1]),curses.A_BOLD|curses.A_REVERSE)
        screen.addstr(dims[0]/2+1,dims[1]/2-len(song_name[(count + 1) % len(song_name)].split('/')[-1])/2,(song_name[(count + 1) % len(song_name)]).split('/')[-1])
        screen.addstr(dims[0]/2+2,dims[1]/2-len(song_name[(count + 2) % len(song_name)].split('/')[-1])/2,(song_name[(count + 2) % len(song_name)]).split('/')[-1])
        screen.refresh()
        action = screen.getch()
        if action == curses.KEY_UP:
            count = (count - 1) % (len(song_name)-1)
        elif action == curses.KEY_DOWN:
            count = (count + 1) % (len(song_name)-1)
        elif action == ord('\n'):
                player(song_name[count])
        else:
            return


#Script starts Here
#required imports
import curses
import subprocess

processes=[]

screen = curses.initscr()
dims = screen.getmaxyx()
screen.keypad(1)
curses.curs_set(0)
song_selection()
while processes:
    processes.pop().terminate()
curses.endwin()