Coding Journal

I'm toying with the idea of attending a coding camp, so I was looking at a review website .  I noticed I couldn't really read all the reviews I was looking at on one page or (as far as I could tell) sort them by ranking.  I thought how useful it would be if I could just figure out a way to download all the review data and put it into a database. Scanned the HTML for a bit, but didn't make any progress figuring out where the data I wanted was.  Did a few searches that turned up some complicated looking stuff (a tutorial from a company that devotes itself exclusively to HTML scraping), gave up, came back, and finally found this .  Unfortunately, the tutorial was written for use with Python 2, so I had to figure out how to modify the libraries -- now at least I understand a bit more about why Chuck's class imports urlopen .  After that, it was as easy as mousing-over the element I wanted on the review page and requesting the text. Needs further development in a...

The apps I'm going to be analyzing are part of Dr. Charles Severance's MOOC on Python and Databases and work together according to the following structure (which applies both in this specific case and more generally to any application that creates and interprets a database using online data). The data source, in this case, is Google's Google Maps Geocoding API.  The "package" has two components: geoload.py  and geodump.py .  geoload.py  reads a list of locations from a file -- addresses for which we would like geographical information -- requests information about them from Google, and stores the information on a database ( geodata.db ).  geodump.py  reads and parses data from the database in JSON, then loads that into a javascript file.  The javascript is then used to create a web page on which the data is visualized as a series of points on the world-map.  Dr. Severance's course focuses on Python, so I'm only going to work my way through ...

As a final project for Week 5 of "Python For Everybody: 4," we had to work with an application that retrieves geographical data from a Google API for a list of locations, stores the data in a database, produces representations of the data in JSON, and then displays them visually in HTML -- a kind of home-made Google Maps. I didn't have to write any of this code.  All I had to do, really, was get an API key from Google and paste it into the code / HTML (because Google now requires API keys for JavaScript as well). That was the hard-way -- I could have kept the key  variable False  and just used a local version of the data.  But as it is, I now have a Google API project called "Geodata-xxx" and two keys.  Well, it isn't exactly Kingdom Hearts. What I would like to do in the next few posts is make sure I understand the code I used.  This is a point I've been thinking about since yesterday: the scariest thing whenever you have to write anything, code bei...

I guess I should really be putting these things up in GitHub.  The way I see it, the coding journal is just a place to share the code I write or study along with any notes I have about it.  It's sort of a documentation LiveJournal, if you will. Anyway, this is a "study" for my project idea: create an app that will prompt the user for two dates, then calculate the difference between them. The burden of this study is twofold: (1) convert dates in standard American form (e.g. December 15, 1993) into dates in standard American numeric form (e.g. 12/15/1993); (2) create a numerical representation of the date. To process the date, I started with a list of the months.  I then used a loop to create a dictionary that would attach a value to each month. Next I had to parse the user entry (I haven't added any debugging for incorrect entries yet). I did so by splitting the entry into "raw" data.  I used my dictionary to process the month name into a number, str...

#import libraries to open url's, interact with the Twitter API, use json, #use SQL, and ignore security certificates. import urllib.request, urllib.parse, urllib.error import twurl import json import sqlite3 import ssl #relevant Twitter API. TWITTER_URL = 'https://api.twitter.com/1.1/friends/list.json' #create the SQL database and cursor. conn = sqlite3.connect('friends.sqlite') cur = conn.cursor() #Create the People and Follows tables. People will list all the users #we've encountered and let us know whose friend lists we've retrieved; it #also assigns a unique number to each one. (Should be 1-1 between id's and #names.) cur.execute('''CREATE TABLE IF NOT EXISTS People     (id INTEGER PRIMARY KEY, name TEXT UNIQUE, retrieved INTEGER)''') cur.execute('''CREATE TABLE IF NOT EXISTS Follows     (from_id INTEGER, to_id INTEGER, UNIQUE(from_id, to_id))''') #creat...

Today I've been playing around (or rather copying from a book that's playing around) with the Twitter API.  Below is a program that creates a SQL database for a given Twitter user listing that user's friends, whether that friend has been searches, and the number of times that friend has appeared in our searches.  I can't say I 100% understand how the machine works yet, but I  have a running commentary that sheds a bit of light on something I certainly did not understand the first couple of times I looked at it.  What the app is supposed to do is crawl through the friends of an account we select, then the friends of those friends, and so on.  The end result would be a Twitter network that tells us the number of people from the network who have "friended" each of its members.  It would also probably take years to create, since it can only be run 14 times a day... #Import all the necessary libraries to open internet connections (urlllib), #access the Twitter...

Just like that, I've managed to program something (to a very loose approximation) like intelligence into my parser.  The problem whose solution eluded me yesterday was how to get the right position of the stem using find() (or some variant thereof).  I had mentioned the possibility of using len()  to figure out which part of the string to look in.  I thought about the problem more today, and this is what I came up with -- basically, a test to see whether the stem I'm looking for is in the right part of the string: def parse(verb):     for conj in conjugations:         if verb.rfind(conj) == len(verb)-len(conj):             return(stems[conj])         else:             continue What I realized is that I need to start at the end of the string and go back to the beginning of the stem.  So it's actually pretty simple: if parse ()   finds the beginn...