w8d(-1) - making image thumbnails

I spent a lot of the weekend sucking the marrow out of my Costa Rica experience, so to speak.  I did manage to figure out how to dynamically create a thumbnail of a user-uploaded image.  It was not easy.

Challenges:

1. the PIL (python image library) that most stack overflow questions referenced is no longer supported.  Pillow, a fork of that library, is still supported, however.
2. The Pillow tutorial for making a thumbnail referred to opening files from the users file structure, but I wanted to use the Image file object without writing it to disk.

I wrote a step by step guide since others in the class had expressed interest in this.  I'm fairly proud that I was able to get all this to work.  I'd like to, next, get it to run in the background since the user experience is currently held up while the files upload to S3 and get resized.

Guide:

Creating thumbnails from uploaded files (in django/python 2.7)

ultimate goal: accept a file from a user, create a thumbnail from the file, upload both original file and thumbnail to S3, and record link to respective files.  Note: this does not take into account a user uploading an enormous file which would have to be processed in pieces (Django default limit is 2.5 MB)

preparatory stuff:

a. pip install Pillow (http://pillow.readthedocs.org/en/3.1.x/installation.html), a python module for image manipulation.  Any stack overflow answers that refer to PIL (python image library) should work as written, so long as you have installed Pillow. (PIL is no longer maintained, but Pillow is a fork that is still being maintained).  Exceptions may exist

b. in views:

            from django_boto.s3 import upload

            import StringIO

            from PIL import Image

            from django.core.files.uploadedfile import InMemoryUploadedFile

1. I needed to create a form that included a file upload as one of several fields (vs. a form that accepted ONLY a file upload)

corresponding html for the upload file input (name=”pic”).  The accept=”image/*” means that the form will post only when an image file has been selected.  The * means any image format is ok.  You could constrain the format to jpg, tiff, png etc.

           

2.  Once the form is sent via POST request, the file is accessed via request.FILES[‘pic’]

            new_upload=request.FILES[‘pic’]

3. Open the uploaded file as an Image object (from the Pillow library):

            newThumb=Image.open(StringIO.StringIO(new_upload.read())

4. set a variable that contains a size for the thumbnail.  The format is a tuple with height and width (h,w).  To keep the same proportions, but resize for the long side to be 128 px, for example:

5. make the damn thumbnail! Note that calling the .thumbnail method changes the Image object in place.  You cannot set newThumb.thumbnail equal to a variable.  The Image.ANTIALIAS ensures your downsized image still looks reasonably good)

            newThumb.thumbnail(thumb_size, Image.ANTIALIAS)

6. You need a name for the new thumbnail file.  Remember, “new_upload) is an object.  It has already been stripped of it’s file path, but it still has the original upload name (eg. “fancycat.jpg”)

           

            upload_name = new_upload.name.split(.)

            thumb_name=upload_name[0] + “_thumb” + upload_name[1]

result is “fancycat_thumb.jpg”

7. The fun part!  Create a “django file-like object” to hold a temporary version of your thumbnail.  Using this method, you don’t have actually create a file and write it to your file system. (source: http://stackoverflow.com/questions/3723220/how-do-you-convert-a-pil-image-to-a-django-file)

Think of how you do

       with (“file.txt”, wb+) as f:

                   f.readlines()

‘f’ is the ‘file-like object’

To create the file-like object, however, you use:

            thumb_io=StringIO.StringIO()

8. save the thumbnail image object (newThumb) to the file-like object using the Image method .save():

           

            newThumb.save(thumb_io, format=”JPEG”)

9. Associate file properties with the file-like object.  (I think that’s what this step is doing.  I’ll give you a dollar/550 colones if you want to look up the documentation because it’s 11PM and IDGAF)

            thumb_file=InMemoryUploadedFile(thumb_io, None, thumb_name, ‘image/jpeg’, thumb_io.len, None)

10.  Finally, you can upload your simulated file (and your original full-size file).  Remember to catch the url:

            upload_path=upload(new_upload)

            thumb_path=upload(thumb_file)

11.  Finally finally, create a new instance of the model that will store the data.  In my case, the model is called Cat_pictures

            new_pic=Cat_pictures()

     

            (in models.py):

            class Cat_pictures(models.Model):

                        url=models.TextField(blank=True)

                        thumb_path=models.TextField(blank=True)

      the blank=True argument allows users to submit a new sighting without submitting a photo if wanted.

           

            in views.py:

            new_pic.url=upload_path

            new_pic.thumb_path=thumb_path

12. Don’t forget to save the new instance.

            new_pic.save()

w7d5 - google maps, and javascript!

My location page actually works now! It can:

1. allow a user to choose automatic detection of location from their current device or to enter street address
2. find a user's location using the html5 location methods
3. display that location on an embedded google map
4. accept a standard google text search (i.e. anything you'd normally use on google maps, like a zip code, or a street address, or a city name)
5. show the street address location on a map
6. allow users to drag the icon to a desired location
7. capture the latitude and longitude of the marker and build a redirect url using javascript
8. capture data passed in to the location page via url api and include that in the redirect url after the location has been captured.

So, yes, most of the js was actually written by google in their documentation.  I had to adapt several examples, though, to do my more complex implementation (to give users a choice of methods based on mouse events).

That was Friday.  The weekend gets its own post this time.  Cheers!

on page load:

on selecting Address:

on entering an address:

Stuff appears and disappears like magic!

Now I just have to make it look less shitty...

w7d3&4 More django project and the eternal hell of forms

I missed a blog entry!  I did so since our illustrious TA is leaving a week early and our going-away-party lasted into the wee hours of Wednesday night/Thursday morning.  I'm proud to report that I did my part to uphold GoCode's honor at the fussball table.

In other news, catfinder progress continues at a snail's pace.  I am, however, learning quite a lot.

After my glorious discovery of custom template tags (all the way back on Tuesday) I was faced with a daunting prospect of a lot of cumbersome in-template dictionary lookups and for-loops with for-loops within for-loops.  So I went back to the drawing board and figured out a way of creating a custom object in the views.py that holds all the relevant data from the 4 lookup tables neccessary for the "local cats page" (each cat with multiple sightings, each sighting with multiple pictures and a single (human) user).  Bonus: since it's a single object it can be serialized and returned as a JSON object should (when!) FriendlyCatFinder.com becomes an important information source.

I can just imagine the code schools of future with projects based around Yelp clones that show bars in a certain area, modified by number of accessible cats within walking distance.

Wednesday morning was a tutorial in Amazon S3 for image hosting, and getting django to upload the image and save the resulting path. Woo.

Today was spent in integrating the file upload and modifying it work with my forms.  And getting my "new sighting" form to store everything properly.  I don't know why it took so long to do, even with ignoring any kind of styling at all.

My plan, such as it is, is to get all the pieces working, and to make it look good afterwards.  Otherwise, I know I'll get stuck in an endless loop of tweaking css to make the corners of my form boxes just the perfect degree of roundness.

Done: front page, login, logout, new user, admin functionality, local cats, new sighting, new cat, image uploads

To do:  get google maps api working (BIG), user profiles, user editing existing info

w7d2 - Progress, step by bloody step by custom django template tag

It's a weird feeling to realize that I'm still taking in vast amounts of information, while feeling like I'm spinning my wheels, so to speak.  I'm at the point where I need to know so much that each little thing I add to my stock of knowledge disappears almost without a trace.

The lecture today was on making our own APIs and returning information from an external query as json or xml.  The exciting part was learning how to implement this within my own project.

For example: I have a "location" page that asks users to determine the location they wish to search.  For some reason, cats are not yet stored as destinations in google maps, so I need to query the catfinder database to find all the cats in a given area.  To do this I needed:

• to work with decimal data types since geolocation data is best handled as decimal(10,6) data type (i.e. up to 10 total characters, with no more than 6 places to the right of the decimal.  Not ridiculously difficult, but I spent some time playing with mathematical operations to see that I wouldn't be fouling anything.
• work out how to query sql for distance from a point.  The general formula I found is 

SELECT id, ( 3959 * acos( cos( radians(37) ) * cos( radians( lat ) ) * cos( radians( lng ) - radians(-122) ) + sin( radians(37) ) * sin( radians( lat ) ) ) ) AS distance FROM markers HAVING distance < 25 ORDER BY distance LIMIT 0 , 20;

although that's for a distance of a mile from a point, taking into account the curve of the Earth's surface.  My distances need to be pretty close (since most cats don't wander far), so I'm going to ignore curvature.

My solution was to define a search "radius" as 0.010000 decimal degrees.  This translates to about 1 km (courtesy of http://www.csgnetwork.com/gpsdistcalc.html) and to return all cat sightings that are within 0.010000 degrees of the search point.

• (back to bullets)
• Next problem: My data for "local cats" page (showing proximal cats) looks like this 
  
• to populate that page, I need to organize by "cat".  One cat can have many sightings (cat_id is a foreign key in sightings)
• Each sighting has several photos (in a separate cat_photos table with a foreign key from each photo to a sighting)
• Each sighting has a user who submitted the info, and who may wish to edit old submissions

I worked out the query structure in for use with django (a bunch of filter requests) and returned the data as dictionary. 

Dictionaries are awesome!  So easy to use in python!

Not in django templates, however.  Or at least not as we were taught django templates.  You can pass a dictionary into a template as part of context, but you can't readily access the values using standard dictionary __get_item__. (see here for relevant Stack Overflow discussion).  

I'll spare you the misery I experienced as I tried everything I could think of, before getting the right search results to figure out what was going on.  

Solution (I think) is to write a custom tag.  I spent all evening looking reading and rereading sections, but I got one that works for dictionary lookup!  It's too late now to fully test it, and I don't have my locations page working like I want yet...but it works in basic form.  

tag:

@register.filter(name='lookup',is_safe=True)

def lookup(dict,key):

    return dict[key]

implementation:

{{ <dict>| lookup:<key> }}

• Also (this deserves it's own bullet): I was able to assign a variable using the templating language {% with info| lookup:"id" as x %}
                  <p>cat name: {{ x.name }}</p>
                  {% endwith %}
• woo hoo!

So, now that I look back on it, that's a huge amount of ground covered in one day.  I don't 100% understand everything (decorators are still beyond my own efforts), but I was able to read and implement the documentation.  I worked with a new data type (decimal).  I figured out my own (hacktastic!) geo-distance formula.  I avoided a tantrum at the 100th error message.

So.  Win.

w7d1 - So many pieces

So this page took all morning.  In my local test environment, the links all work.

I have a lot to learn about bootstrap and flex boxes.  At present, the best I can say is that I'm (loosely) using the bootstrap column model and some homebrew css...and I didn't get my buttons to the center of the page by using a ton of <br>.  So there's that.

In the afternoon, I worked on adjusting my models as I learned more about field types and the ability to let users leave a field in a form blank by setting "blank=True" as an argument when defining a model.

Then, I updated the admin file to allow me to create records the easy way via the admin panel, and I made some test subjects to populate my site with.

This evening was setting up a bit of formatting to keep individual pages looking uniform (via a base.html extension for each page).  Then, I started wading into building queries to populate different pages.

Just the "local cats" page, for example:

1. get location from the "location page" and send it via form to the "/local_cats" page.  The query I want to use translates to "get all cat sightings within 1000 meters of the user's location".  I could do that easily in sql, but I'm only just working out the django version since the basic filter function doesn't do numeric comparisons.

Perhaps we'll see tomorrow if my cunning plan will work.

w6d5 - Really beginning the catfinder project

On Friday and over the weekend, I started building the catfinder project in Django.  As of Sunday, I have the models (i.e. database structure) built.  The site has register, login and logout capability.  And that's it.

I'm having my familiar new-project sense of overwhelming options making it impossible to proceed.  I'm working through it by biting off small pieces, by making to-do lists in evernote, and by trying to be selective in what I search for online.  It's possible to find yourself reading some fairly advanced advice on setting up new projects, and before you know it you're reading about virtual environments and load-balancing...

All I want right now is to start building some functional forms and see if they can interact with my databases.  The register and login forms do, but they're built off templates from the in-class projects.

The roadblock?  CSS.  Should I build my pages from scratch to show I can do it, or use a drawing-assist program like web designer or even just a button-drawing assist like ui parade.  I doubt anyone really makes their own stuff if they can avoid it but...

I'm seeing where my instincts to learn the inner workings of things is running up against the complexity of modern systems.  I think I may focus on the back end stuff and getting a really good handle on that side, and use as many assists from available tools as I can (so long as I can convince myself that I have some sense of what's going on, anyway).

w6d4 - Javascript, jQuery and blackjack

I woke up at 4 or 5 this morning and lay in the dark thinking of stabby rainbow triangles, as one does.  And, as often happens, a solution to my drawing problems came to me.

The problem lay in drawing n triangles arranged such that each triangle had the same base (width) but an arbitrary height corresponding to temperature.  The triangles should touch only at the base corners.  In looking at the <canvas> documentation, my initial impression was that rotate or transform would rotate or move a shape (it's the shapes that move in MS Paint or other simple drawing programs I've used.)  Trying to rotate the images last night met with miserable failure.

My o'dark-hundred realization was that it's the canvas that moving, not the image.  The other realization was that if I defined a new "center" for the canvas (using context.translate), then started drawing my triangle a suitable distance "above" the center, then rotate the canvas by a suitable portion of a circle, the triangles would all be tangent at the corners...by magic!  Or math.

(image from draw.io)

The "base" is the base width of the triangles (which I can specify).  The circle represents the center of the canvas as defined by me.  The arrow represents the "offset" distance - the distance "above" the center where the triangle should be drawn.  It's found using the tangent of angle A, where the opposite side is length base/2.  

My next realization is that I could draw a stellate polygon of n sides by setting A = 180/n and the offset as base/(2*tan(A)).  The context rotation between each triangle would be 360/n...or rather 2*PI/n since javascript loves radians.

My code right now is a god-awful spaghetti mess, so I'm not going to put it up yet.  I really need to improve on writing long programs in neat packets.  I got pretty good with python, but javascript has knocked me back into my reflexive write-everything-in-one-big-blob mode.

For now, I'll just say that I thoroughly enjoyed this once I figured out what I was doing.   It feels so good to stretch my math muscles again, however small the stretch may be.

I still would like to make the output more interactive, so the stars rotate when clicked, and display an image of the day's weather.  I know how I'd like to do it (just redraw the stars starting at one place further along the array).

The other project for the day was expanding a javascript version of blackjack that one of the instructors had started.  The base version had a rudimentary game engine, and the cards and deck defined (with images).  There was a game clock that monitored the state and redrew cards and printed text as needed.  

Reading expert code may be my greatest joy here, other than my own occasional breakthroughs.  It's so interesting to see how people break a problem into pieces and make them work.

The really fun part was unravelling what was going on since there was no documentation provided.  I'm happy to say that by the end of the day (about 4 hours), I'd managed to implement a scoring system, and add hit and stand capability.  I can see the way to finish the project.  

I have so much to learn still.  I'm definitely at the "OMFG" stage of learning, or the "desert of despair" from the Viking Code School blog. I'm at the point where I'd felt like I kind of knew what was going on with python...and now I'm realizing just how deep the well is, with sql, django, javascript, ajax, server management...it's dispiriting. It's not that I don't want to learn all that (I love learning, after all).  It's more that there's just so much before I can do anything useful.

Unless you count n-sided rainbow stellate polygons as useful, of course. :)

Hah!

w6d3 - HTML5, ajax, APIs and jQuery, or: How I spent 3 hours drawing a triangle

Today was jQuery basics and a revisitation of the openweather.com api.  The challenge was, starting with a simple form (that we built), get a city and the number of days of forecast desired.  Request the relevant info from openweather via ajax query and parse the resulting jQuery object (as opposed to the python/json parsing from our previous encounter).  Then, display the data in table.

The extension project was to display the data in a bar graph using the HTML5 <canvas> tag.  I was able to get both working after some initial faffing about with forms and data retrieval.  The bulk of my time was spent making a) a rainbow fill for a drawing object (not that hard), b) trying to draw triangles instead of rectangles (also not too hard) and c) trying to rearrange the triangles into a stellate pentagon (i.e. a star) which is far more difficult that I could manage.  It's apparantly weird enough that not even StackOverflow has a solution.  I have an idea of how to approach it, but it involves using the matrix transformation in HTML5...and it's been 20 years since linear algebra.  I can even move the triangle using the transformation, but I don't sufficiently understand how the context and positioning work such that I can change locations systematically

Here's my rainbow triangle, anyway:

And here's the Dallas (temperature) forecast for the next 5 days:

You can try it for yourself if you scroll all the way to the bottom of the post.  WARNING: You must type the city with capital letters in the proper place: New York, Columbus, San Francisco, Berlin.

Also, you should refresh the page after each search.

Finally, it doesn't handle large negative temperatures very well.


City:

w6d2 - DOM, bootstrap & javascript

For as much time as I spent learning javascript on my own, today was pretty disappointing.  (months!  The codeacademy track and the first 5 chapters of Eloquent Javascript were my mainstay.)  To be fair, it's been since September since I last did anything substantial with JS...but still.

To be even more fair, I did understand my goals with all the JS exercises (write for loops, while loops, and even a recursive function for factorial) I at least understood the challenge.  It was the syntax of js that had become unfamiliar after all this python.  Fortunately, I was able to pick it up again reasonably quickly.

Tonight, I started the CodeAcademy jQuery track afresh.  Hopefully that will give me a head start on tomorrow.

I wish I had more to add.  Bootstrap looks interesting since css is my least part of web development.

I'd like to write a more "meta" post on how this is all feeling...but it's always so late by the time I get to write that my more eloquent side is already dozing.  One day...

w6d1 - Planning the CatFinder

The cat finder project is moving ahead a bit:  I have user stories, a database scheme, some wireframes (that tail off into rudimentary...) and a Google Maps api key.  Woo.

Here's the super short Draw.io flow.  Each site user can register a sighting of a cat.  There may be many sightings that pertain to a single cat, so each cat "object" in Django will potentially have many sightings.  Ideally, each sighting will have just one user and just one cat.  The cat it refers to can change based on further information.  There is no independent verification of any particular cat's existence, so the first new sighting of a cat will be a new cat instance.  A sighting of an already existing cat will count as an additional review/sighting for that cat.





The sighting will be the primary data entry: user (foreign key), text description, rating, friendliness, distinguishing characteristics, location, timestamp of sighting.

Photos will go in a separate table since each sighting could include multiple photos/videos.  Each photo will have a foreign key to link it to a unique sighting.

When a user logs in and has found a cat:

they will be prompted to enter a location (using google maps geolocate or a physical address).  They will then be asked if their sighting refers to a cat already in the database (based on existing sightings within 0.5 miles):

If a previous sighting of this cat exists, they can add a new sighting linked to the existing cat.  (yeah, this one is pretty rudimentary but it has the essential information for populating a table, along with auto-generated time stamp and the geolocation info from the initial entry.)

If not, the new sighting will count as a new cat.  Each cat will have a page with all the pictures from the sightings that refer to the same cat (much like a restaurant can have multiple reviews on a review site.).

There will also be "flag this entry/photo" options throughout to allow users to report spam and inappropriate content (like pictures of dogs).  I'm hoping for a reddit-style moderation system with volunteer moderators for local areas and a few admins (i.e. me) who can appoint moderators and resolve bigger conflicts.  Moderators would be able to join sightings that were reported to be of unique cats, but are really of the same animal.  Or, the reverse (split sightings inappropriately tagged as the same animal).  Finally, if I'm able to impelement a "lost pet bulletin" a local moderator would have to approve an email sent out to site members who have opted in to the "Orange-Collar Alert"

Finally, users can create a profile for themselves, with photos, social media integration (of course!), and stats on how many cats and how many sightings they are responsible for.

Finally, finally, a user can search for cats near a specific location.  The location of any particular cat will be approximate (to a half mile or so) to avoid making it too easy to find friendly cats.  The exact location will be available to the person who uploaded it, but to no one else.

__

As I'm writing this up, I realize that I could use the same basic format for birders, or a catalog of plants in a city park (or crowd sourcing street tree data for a city).  A cat finder is inherently silly, and perilously close to the dreaded "facebook for ferrets".  It's contained, though, and of a scope I can imagine completing (as in it will have rudimentary functionality for limited users) in the next few weeks.  In the process, I'll learn to set up my own Django project with interconnected sql tables, I'll design some basic UI and forms, I'll apply some simple Javascript, and I'll learn some Google api mechanics.  There is a lot of room for science education and crowd-sourced science initiatives and I think this project will be a good ramp up to a larger project.

w5d5 - Planning something big

Day 5 saw the implementation of some more elements of a simple web store- a checkout page and a user profile page.  These I did without input from the instructors.  They work ok, but it's nothing super exciting.  I suppose the good part is that I'm starting to 'get' basic django well enough that I don't need step-by-step instructions.

Friday afternoon, and much of today, was spent working on plans for a bigger, standalone project.  We're supposed to go through a complete cycle of project development, starting with "user stories" and wireframe mockups of pages.  I was divided between a teaching tool that an instructor could use to divide students by ability (set according to parameters the instructor chooses), and a light-hearted "Yelp for cats".  I went with FriendlyCatFinder as my project.  I really like the student working group project, but it's much more than two weeks' work at this point in my development. (The plan was to let a teacher upload a short quiz, then split students in to equal, random groups such that each smaller group has a mix of top, medium, and low-performing students.  Instructors with large classes face the problem that they don't know individual students well enough to make groups on their own.  The smaller groups would then be given a case study or other long-format problem to consider, and answer as a unit.

FriendlyCatFinder, by contrast, has a much more unified structure.  It would be a simple geolocation app with a form for rating cats encountered in public.  How pettable, how approachable, etc.  It's cute and fun, and I'll get a chance to learn the google maps api as well as build a site with minimal support.

Tomorrow: bootstrap, and a return (for me) to javascript.

w5d4 - More Django, and a capstone project

Django is becoming a little less like magic, and a bit more like a really advanced technology.  To put it another way, I've been working pretty hard on pulling the pieces apart to see why things work.  I know there's some tons of stuff that's entirely beyond me at this point that happens underneath, but I'm at least starting to get a handle on the surface things.

We're still working with a super-simple web store.  Today we split the items with pagination, and added a shopping cart (of sorts - more a bin for holding items we select).  I figured out how to delete things from the cart (adding things was mapped out for us), and how to show a total based on the prices for the individual items.  I also figured out how to add links to all the pages to jump around more easily (not on the program at all, but it seems weird to have a site without internal links.

It all sounds so trivial when written out.  It didn't feel trivial going through it.  It didn't feel trivial, either, when I was going over the code this evening with a friend in the class.  Understanding the properties of the request object, the user object, and the shopping cart object, and how they interact with the product objects...it takes a bit of work.  Frequent use of dir(), command line sqlite3 to keep up with changes, and multiple print statements to monitor the status of the program all helped too.

It's dispiriting at times to think that I'm just 3 weeks from being done, with so much left to do.  Seriously, this stuff feels so minor and inconsequential...but I suppose everyone starts off small and works up.  Hopefully this is the steepest part of the curve....

Regardless, I need to be thinking of a capstone project.  I have a few ideas, but they seam either derivative, or too complex.  We'll see what happens.

w5d3 - Django, and the perils of paint-by-numbers

Today was more Django. I wish I could say it was enlightening, but it's still mostly cut, paste, and pray.  We're working on a super-simplistic ecommerce site with several pages of pseudo products. My only real extension was working on adding images to pages via templating, with each product associated with a particular picture.  I got all the elements working ok (a links field added to the model, images in the static files folder, but I just could not figure out the templating syntax.  Eventually, with an instructors help, I was able to get the images working.

Perhaps more interestingly, I looked into the python scripts that comprise Django. Holy spiderweb, batman! Every module references several other modules.  I had some idea of trying to unravel the underpinnings of Django,but I'm not sure that's reasonable in one human life time.  Instead, I think I will try to work out how the render function works, and especially how it parses the templating script.


All of this section makes me very uncomfortable.  Python was just feeling reasonably comfortable.  Even if I didn't know how to make something work, I understood enough to ask google for help in an intelligent way.  So much of Django is just magic so far.  I often don't even know  how to formulate a searchable question.  Hopefully, familiarity through repetition will make the framework more approachable.

w5d2 - git

So, yes, we did cover the basic git intro today.  After several tutorials online and today's class, I feel I'm getting the hang of it.

Honestly, the concept is pretty simple.  I get the idea of version control.  I even get the idea of branching, pulling from a repository, pushing, etc.  I could, do local commits since the beginning of class, a month ago, thanks to code academy and others.

The information that was missing, and that I'm still not 100% sure on, is "git best practices".  It was never clear from the tutorials that renaming a file would screw up your git commit history.  Renaming files is standard practice in most of the world, as is moving a file while reorganizing content.  With git, though, those things are REALLY BIG DEALS.  Yes, there are work-arounds, but it's just another layer of complexity at this point.

Some day, when I really understand how things work, I'll come back to this idea and write my own tutorial just on 'best practices'.  As I understand them now:

1) One project per (hard drive) folder.  Each project may contain multiple files.
2) One folder to one github repository.
3) Pick descriptive files names at the start, and stick with them.
4) Use git version control rather than renaming files to roll back changes as needed.
5)  Commit only code that is functional, since each commit could potentially be a roll-back point.  Getting broken code from a rollback makes it harder to deal with bugs that arose subsequent to that commit.
6) There are plenty of instructions available for generating ssh keys, and for walking newbies through the establishment of a new repository.  There's not enough on the care of ssh keys and the consequences of screwing up your passphrase.

I think it's all working ok now, though.

One interesting thing I discovered:

In the process of playing with git, I was looking back over older projects.  Early in the first week, I wrote a module that would print out (to the terminal) the names of variables discovered via dir(), with the magic variables removed for clarity.  the script would also print out the type of each variable and its value (via eval(var) - unsafe in settings where strangers can access the code, but probably ok if you're using it for internal debugging).

The point of the table was to illustrate a point made in class about following the variables in your code.  I suppose it's not terribly useful at this point...

The reason I brought it up, though, is that I managed to refactor 4 functions into 4 lines of code with, probably, a gain in clarity.  The "variable_printer.py" script has both the old and new versions, along with instructions for use.

Hooray for refactoring!

w5d2 - git, finally. And some refactoring.

We had a long weekend in the center of Costa Rica, near Arenal. My plant-obsessed photos are here.  It was a magnificent experience, and a welcome respite from the unrelenting heat and brown foliage of Guanacaste. It was also a welcome chance to rest my brain after the feast of the first 4 weeks.

On returning, I managed to finish my refactoring of the Blackjack game from Week 3.  Because what else would you do after 5 hours of driving across Costa Rica?  A playable demo is here.  The github is here.

On Jeremy's (one instructor) advice, I reworked the play-handling such that each turn was handled by a method in the Game class, rather than passing the dealer's hand to a method in each player instance.  As Jeremy pointed out, you really don't want classes on the same organizational level talking to each other directly.  I also cleaned up a lot of detritus and cleaned up some variables (i.e. temporary variables used only within a particular method don't need to be 'self'd.), swapped out a few for iterators for list comprehensions, etc.

The other big changes were to add betting game play (with blackjacks handled appropriately), the ability to add or remove players after each round (and even to sit out a round, then come back in).  I also built a new function for my 'entry validation' module that refuses to accept an answer that is in a list.  (If you're adding a player, you don't want to have duplicate names.)  My existing validation functions only checked whether an entry was in a list of acceptable answers.  Also, it now uses a 6-deck shoe that gets restored when it reaches 75% used.  Take that, card counters!

The game still doesn't handle doubling bets, insurance, or splitting a hand.  Splitting a hand would be the hardest to add since the Player class is currently configured to hold only one hand per instance.  To split hands, I would have to make each player have an array of Hand instances (analogous to how the Game instance has an array of player instances).  All of the player Hand methods would have to be updated to "for hand in player.hands: <do thing>".  Maybe there's an easier way, but I can't see it right now. I need to focus on the Django projects we're working on now, so I'll leave Blackjack as-is.

To be fair, it's in pretty good shape, and I'm reasonably happy with it.

w4d4- half way

I've officially been at GoCode for 4 weeks now.  It seems impossible that it's been so long.  I'm both blown away by how much I've improved, and appalled at how far there is still to go.

We kept working with django today, reviewing the basic MVC architecture and how we can use python to manage the pieces.  The most interesting, and most difficult, thing we learned was the django template formatting, and the use of templates to integrate content drawn from a database.

My personal victory today was figuring out how to pull data from two tables in the app-associated database and to get chosen bits to show up in the same page.  The pseudo-python of the templating language led me to believe that I could manipulate it in ways that it just couldn't work.

My other discovery was finding the source code for python modules.  I really enjoyed reading such high quality work.  I want to get to the place where I can follow and appreciate what each part is doing.

I also got the blackjack game working, with money bets.  I have a few more tweaks, and I should be ready to show it off.

This midterm weekend is for travel.  I'll be away tomorrow and monday.  No blog, alas.  I look forward to seeing what the second half of the class brings.

w4d3 - Ironically, we built a blog using Django today. Also refactoring hell.

In our first day of working with Django, we built a blog.  It's not really useful, of course, and it just runs off a local host, but...it works!

Part of the project was learning the MVC (Model View Controller) scheme. We used some basic regex to detect the url request (/home, /about, /blog, /blog/page1, etc), served up an html template formatted with info pulled from a freshly-made sqlite database.

I won't lie - it's pretty exciting to see all the pieces I've been working on for months coming together.  I like seeing how a framework operates, too.  I want to dig into the django modules and see exactly what each piece is doing.

After working on the basic web stuff all day, though, (and building a blog once in code-along, and once from notes), I went back to refactor my blackjack game from last week.  I've added betting to the game, and greatly simplified the structure to make sure class instances are not interacting directly (i.e. the dealer's hand object had been passed from the game.play() method to each player in turn to evaluate win/loss/push).  Of course, changing so much broke a ton of things and I've been hours in fixing it.  I see why people hate refactoring.  It's just a few hundred lines long, but there's so much to keep track of.

In a side note, I've been working on learning shortcuts in the text editor to speed things up.  Today I figured out how to set bookmarks so I didn't have to scroll past walls of text every time I wanted to look another function up.  I find that using bookmarks keeps my attention focused much better.

Speaking of which, I failed, again, at an early bed time.  Soon, perhaps, I can port this blog to a site of my own devising.

w4d2 - API, SQL and python

I love how we're integrating different elements now.  The project today was to use python and the Open Weather api to get a 16-day forecast for a city, to take the resulting json-formatted data and push it to an sqlite database we initialized.We're still at the super basic level and dealing with canned scenarios, but I can see the pieces interacting.

It's also really fun (is that word I want here?  I'm sticking with it) to be able to have python interact with the world beyond my computer.  Almost everything I've done so far has run off text files or output to the terminal.  Now I can request information directly from the web, and push to a local database.

The 'extension' project was to build a python script that would ask a user for city and use their answer to query Open Weather for a forecast, then put that into the sql database.  Not too challenging by itself.  A step deeper, though, and I was looking through the documentation for how queries in the Open Weather api are structured.  It's a whole mini-language hiding in almost plain site in the url!

I'd always known the extra long URLs that look like random gibberish must hold some meaning, but I'm getting an idea of just how much information is packed into URLs.  Yesterday's toy server essentially replicated a directory tree in how web pages were stored and accessed.  Every word or character in the url had a counterpart in the directory tree the information was pulled from.

As part of my own investigations, I found a 20 MB file correlating the numeric code Open Weather uses to represent each weather station in their network.  It's 209,000 lines of text!  Small by many measures, but huge for me.  Opening it and using it required finding the json module for python, then porting the info to my sqlite database.  Not too difficult, once I found all the pieces I needed.

The harder realization was that there are multiple entries for cities.  Some entries refer to different weather stations in the same city, while others refer to cities of the same name in disparate places. (Paris, Texas!).  There are country codes in the data set to narrow the options some, but the codes are simple two-letter abbreviations.

Next step: a lookup table for code/country name.  Getting the code into python was pretty easy, but getting the data into SQL was impossible due to utf-8 conflicts with module (I think).  StackOverflow searches took me to some info on converting utf-8 to ASCII.  I can make it work, but information is lost as non-ASCII characters in country names are stripped. Eventually, I decided to just let it rest for now.  I can add unicode to the long list of things to investigate further.

I did manage to get the project working, and now it's once again 1130 PM and I'm almost cross-eyed with exhaustion.

W4D1 - Local Web Server, regex, and madness

Today we played with using a simple python script (handed to us) to run a local web server.  It gave us a chance to see a "GET" request and to do some simple string parsing using real info.  Also: super-basic templates (dynamically replace "###Title###" in an html file with "My Page Title" as the web server sends a GET request for the page) and using a function lookup dictionary to have a function return the relevant html string in response.

The last exercise is what kept me up until 11 (on a school night!).  Using regex to pull up the relevant function based on match to the GET request.  Even with the regex patterns pre-loaded, it was a challenge since I've only every done a copy-paste from a stackoverflow thread (with a little modification on my part).  Understanding the output from a re.match function and routing that output to a function was...interesting.  I'm not sure if it's my unfamiliarity that makes it so hard, or if I'm just missing something really obvious.

I've wanted to learn regex better for some time.  I played around with Learn Regex the Hard Way.  I suspect I'll have to come back after boot camp to really work with it.  Or maybe I can do the recommended one exercise a day while I'm here to at least get started.  It's such a powerful tool!  Plus, I'm fascinated by patterns so the thought of a mini-language devoted to describing patterns is a real lure.

In the mean time, I went to the tutorials point page and figured out enough to work through the homework.