w8d(-1) - making image thumbnails

I spent a lot of the weekend sucking the marrow out of my Costa Rica experience, so to speak.  I did manage to figure out how to dynamically create a thumbnail of a user-uploaded image.  It was not easy.

Challenges:

1. the PIL (python image library) that most stack overflow questions referenced is no longer supported.  Pillow, a fork of that library, is still supported, however.
2. The Pillow tutorial for making a thumbnail referred to opening files from the users file structure, but I wanted to use the Image file object without writing it to disk.

I wrote a step by step guide since others in the class had expressed interest in this.  I'm fairly proud that I was able to get all this to work.  I'd like to, next, get it to run in the background since the user experience is currently held up while the files upload to S3 and get resized.

Guide:

Creating thumbnails from uploaded files (in django/python 2.7)

ultimate goal: accept a file from a user, create a thumbnail from the file, upload both original file and thumbnail to S3, and record link to respective files.  Note: this does not take into account a user uploading an enormous file which would have to be processed in pieces (Django default limit is 2.5 MB)

preparatory stuff:

a. pip install Pillow (http://pillow.readthedocs.org/en/3.1.x/installation.html), a python module for image manipulation.  Any stack overflow answers that refer to PIL (python image library) should work as written, so long as you have installed Pillow. (PIL is no longer maintained, but Pillow is a fork that is still being maintained).  Exceptions may exist

b. in views:

            from django_boto.s3 import upload

            import StringIO

            from PIL import Image

            from django.core.files.uploadedfile import InMemoryUploadedFile

1. I needed to create a form that included a file upload as one of several fields (vs. a form that accepted ONLY a file upload)

corresponding html for the upload file input (name=”pic”).  The accept=”image/*” means that the form will post only when an image file has been selected.  The * means any image format is ok.  You could constrain the format to jpg, tiff, png etc.

           

2.  Once the form is sent via POST request, the file is accessed via request.FILES[‘pic’]

            new_upload=request.FILES[‘pic’]

3. Open the uploaded file as an Image object (from the Pillow library):

            newThumb=Image.open(StringIO.StringIO(new_upload.read())

4. set a variable that contains a size for the thumbnail.  The format is a tuple with height and width (h,w).  To keep the same proportions, but resize for the long side to be 128 px, for example:

5. make the damn thumbnail! Note that calling the .thumbnail method changes the Image object in place.  You cannot set newThumb.thumbnail equal to a variable.  The Image.ANTIALIAS ensures your downsized image still looks reasonably good)

            newThumb.thumbnail(thumb_size, Image.ANTIALIAS)

6. You need a name for the new thumbnail file.  Remember, “new_upload) is an object.  It has already been stripped of it’s file path, but it still has the original upload name (eg. “fancycat.jpg”)

           

            upload_name = new_upload.name.split(.)

            thumb_name=upload_name[0] + “_thumb” + upload_name[1]

result is “fancycat_thumb.jpg”

7. The fun part!  Create a “django file-like object” to hold a temporary version of your thumbnail.  Using this method, you don’t have actually create a file and write it to your file system. (source: http://stackoverflow.com/questions/3723220/how-do-you-convert-a-pil-image-to-a-django-file)

Think of how you do

       with (“file.txt”, wb+) as f:

                   f.readlines()

‘f’ is the ‘file-like object’

To create the file-like object, however, you use:

            thumb_io=StringIO.StringIO()

8. save the thumbnail image object (newThumb) to the file-like object using the Image method .save():

           

            newThumb.save(thumb_io, format=”JPEG”)

9. Associate file properties with the file-like object.  (I think that’s what this step is doing.  I’ll give you a dollar/550 colones if you want to look up the documentation because it’s 11PM and IDGAF)

            thumb_file=InMemoryUploadedFile(thumb_io, None, thumb_name, ‘image/jpeg’, thumb_io.len, None)

10.  Finally, you can upload your simulated file (and your original full-size file).  Remember to catch the url:

            upload_path=upload(new_upload)

            thumb_path=upload(thumb_file)

11.  Finally finally, create a new instance of the model that will store the data.  In my case, the model is called Cat_pictures

            new_pic=Cat_pictures()

     

            (in models.py):

            class Cat_pictures(models.Model):

                        url=models.TextField(blank=True)

                        thumb_path=models.TextField(blank=True)

      the blank=True argument allows users to submit a new sighting without submitting a photo if wanted.

           

            in views.py:

            new_pic.url=upload_path

            new_pic.thumb_path=thumb_path

12. Don’t forget to save the new instance.

            new_pic.save()

w7d5 - google maps, and javascript!

My location page actually works now! It can:

1. allow a user to choose automatic detection of location from their current device or to enter street address
2. find a user's location using the html5 location methods
3. display that location on an embedded google map
4. accept a standard google text search (i.e. anything you'd normally use on google maps, like a zip code, or a street address, or a city name)
5. show the street address location on a map
6. allow users to drag the icon to a desired location
7. capture the latitude and longitude of the marker and build a redirect url using javascript
8. capture data passed in to the location page via url api and include that in the redirect url after the location has been captured.

So, yes, most of the js was actually written by google in their documentation.  I had to adapt several examples, though, to do my more complex implementation (to give users a choice of methods based on mouse events).

That was Friday.  The weekend gets its own post this time.  Cheers!

on page load:

on selecting Address:

on entering an address:

Stuff appears and disappears like magic!

Now I just have to make it look less shitty...

w7d3&4 More django project and the eternal hell of forms

I missed a blog entry!  I did so since our illustrious TA is leaving a week early and our going-away-party lasted into the wee hours of Wednesday night/Thursday morning.  I'm proud to report that I did my part to uphold GoCode's honor at the fussball table.

In other news, catfinder progress continues at a snail's pace.  I am, however, learning quite a lot.

After my glorious discovery of custom template tags (all the way back on Tuesday) I was faced with a daunting prospect of a lot of cumbersome in-template dictionary lookups and for-loops with for-loops within for-loops.  So I went back to the drawing board and figured out a way of creating a custom object in the views.py that holds all the relevant data from the 4 lookup tables neccessary for the "local cats page" (each cat with multiple sightings, each sighting with multiple pictures and a single (human) user).  Bonus: since it's a single object it can be serialized and returned as a JSON object should (when!) FriendlyCatFinder.com becomes an important information source.

I can just imagine the code schools of future with projects based around Yelp clones that show bars in a certain area, modified by number of accessible cats within walking distance.

Wednesday morning was a tutorial in Amazon S3 for image hosting, and getting django to upload the image and save the resulting path. Woo.

Today was spent in integrating the file upload and modifying it work with my forms.  And getting my "new sighting" form to store everything properly.  I don't know why it took so long to do, even with ignoring any kind of styling at all.

My plan, such as it is, is to get all the pieces working, and to make it look good afterwards.  Otherwise, I know I'll get stuck in an endless loop of tweaking css to make the corners of my form boxes just the perfect degree of roundness.

Done: front page, login, logout, new user, admin functionality, local cats, new sighting, new cat, image uploads

To do:  get google maps api working (BIG), user profiles, user editing existing info

w7d2 - Progress, step by bloody step by custom django template tag

It's a weird feeling to realize that I'm still taking in vast amounts of information, while feeling like I'm spinning my wheels, so to speak.  I'm at the point where I need to know so much that each little thing I add to my stock of knowledge disappears almost without a trace.

The lecture today was on making our own APIs and returning information from an external query as json or xml.  The exciting part was learning how to implement this within my own project.

For example: I have a "location" page that asks users to determine the location they wish to search.  For some reason, cats are not yet stored as destinations in google maps, so I need to query the catfinder database to find all the cats in a given area.  To do this I needed:

• to work with decimal data types since geolocation data is best handled as decimal(10,6) data type (i.e. up to 10 total characters, with no more than 6 places to the right of the decimal.  Not ridiculously difficult, but I spent some time playing with mathematical operations to see that I wouldn't be fouling anything.
• work out how to query sql for distance from a point.  The general formula I found is 

SELECT id, ( 3959 * acos( cos( radians(37) ) * cos( radians( lat ) ) * cos( radians( lng ) - radians(-122) ) + sin( radians(37) ) * sin( radians( lat ) ) ) ) AS distance FROM markers HAVING distance < 25 ORDER BY distance LIMIT 0 , 20;

although that's for a distance of a mile from a point, taking into account the curve of the Earth's surface.  My distances need to be pretty close (since most cats don't wander far), so I'm going to ignore curvature.

My solution was to define a search "radius" as 0.010000 decimal degrees.  This translates to about 1 km (courtesy of http://www.csgnetwork.com/gpsdistcalc.html) and to return all cat sightings that are within 0.010000 degrees of the search point.

• (back to bullets)
• Next problem: My data for "local cats" page (showing proximal cats) looks like this 
  
• to populate that page, I need to organize by "cat".  One cat can have many sightings (cat_id is a foreign key in sightings)
• Each sighting has several photos (in a separate cat_photos table with a foreign key from each photo to a sighting)
• Each sighting has a user who submitted the info, and who may wish to edit old submissions

I worked out the query structure in for use with django (a bunch of filter requests) and returned the data as dictionary. 

Dictionaries are awesome!  So easy to use in python!

Not in django templates, however.  Or at least not as we were taught django templates.  You can pass a dictionary into a template as part of context, but you can't readily access the values using standard dictionary __get_item__. (see here for relevant Stack Overflow discussion).  

I'll spare you the misery I experienced as I tried everything I could think of, before getting the right search results to figure out what was going on.  

Solution (I think) is to write a custom tag.  I spent all evening looking reading and rereading sections, but I got one that works for dictionary lookup!  It's too late now to fully test it, and I don't have my locations page working like I want yet...but it works in basic form.  

tag:

@register.filter(name='lookup',is_safe=True)

def lookup(dict,key):

    return dict[key]

implementation:

{{ <dict>| lookup:<key> }}

• Also (this deserves it's own bullet): I was able to assign a variable using the templating language {% with info| lookup:"id" as x %}
                  <p>cat name: {{ x.name }}</p>
                  {% endwith %}
• woo hoo!

So, now that I look back on it, that's a huge amount of ground covered in one day.  I don't 100% understand everything (decorators are still beyond my own efforts), but I was able to read and implement the documentation.  I worked with a new data type (decimal).  I figured out my own (hacktastic!) geo-distance formula.  I avoided a tantrum at the 100th error message.

So.  Win.

w7d1 - So many pieces

So this page took all morning.  In my local test environment, the links all work.

I have a lot to learn about bootstrap and flex boxes.  At present, the best I can say is that I'm (loosely) using the bootstrap column model and some homebrew css...and I didn't get my buttons to the center of the page by using a ton of <br>.  So there's that.

In the afternoon, I worked on adjusting my models as I learned more about field types and the ability to let users leave a field in a form blank by setting "blank=True" as an argument when defining a model.

Then, I updated the admin file to allow me to create records the easy way via the admin panel, and I made some test subjects to populate my site with.

This evening was setting up a bit of formatting to keep individual pages looking uniform (via a base.html extension for each page).  Then, I started wading into building queries to populate different pages.

Just the "local cats" page, for example:

1. get location from the "location page" and send it via form to the "/local_cats" page.  The query I want to use translates to "get all cat sightings within 1000 meters of the user's location".  I could do that easily in sql, but I'm only just working out the django version since the basic filter function doesn't do numeric comparisons.

Perhaps we'll see tomorrow if my cunning plan will work.

w6d5 - Really beginning the catfinder project

On Friday and over the weekend, I started building the catfinder project in Django.  As of Sunday, I have the models (i.e. database structure) built.  The site has register, login and logout capability.  And that's it.

I'm having my familiar new-project sense of overwhelming options making it impossible to proceed.  I'm working through it by biting off small pieces, by making to-do lists in evernote, and by trying to be selective in what I search for online.  It's possible to find yourself reading some fairly advanced advice on setting up new projects, and before you know it you're reading about virtual environments and load-balancing...

All I want right now is to start building some functional forms and see if they can interact with my databases.  The register and login forms do, but they're built off templates from the in-class projects.

The roadblock?  CSS.  Should I build my pages from scratch to show I can do it, or use a drawing-assist program like web designer or even just a button-drawing assist like ui parade.  I doubt anyone really makes their own stuff if they can avoid it but...

I'm seeing where my instincts to learn the inner workings of things is running up against the complexity of modern systems.  I think I may focus on the back end stuff and getting a really good handle on that side, and use as many assists from available tools as I can (so long as I can convince myself that I have some sense of what's going on, anyway).

w6d4 - Javascript, jQuery and blackjack

I woke up at 4 or 5 this morning and lay in the dark thinking of stabby rainbow triangles, as one does.  And, as often happens, a solution to my drawing problems came to me.

The problem lay in drawing n triangles arranged such that each triangle had the same base (width) but an arbitrary height corresponding to temperature.  The triangles should touch only at the base corners.  In looking at the <canvas> documentation, my initial impression was that rotate or transform would rotate or move a shape (it's the shapes that move in MS Paint or other simple drawing programs I've used.)  Trying to rotate the images last night met with miserable failure.

My o'dark-hundred realization was that it's the canvas that moving, not the image.  The other realization was that if I defined a new "center" for the canvas (using context.translate), then started drawing my triangle a suitable distance "above" the center, then rotate the canvas by a suitable portion of a circle, the triangles would all be tangent at the corners...by magic!  Or math.

(image from draw.io)

The "base" is the base width of the triangles (which I can specify).  The circle represents the center of the canvas as defined by me.  The arrow represents the "offset" distance - the distance "above" the center where the triangle should be drawn.  It's found using the tangent of angle A, where the opposite side is length base/2.  

My next realization is that I could draw a stellate polygon of n sides by setting A = 180/n and the offset as base/(2*tan(A)).  The context rotation between each triangle would be 360/n...or rather 2*PI/n since javascript loves radians.

My code right now is a god-awful spaghetti mess, so I'm not going to put it up yet.  I really need to improve on writing long programs in neat packets.  I got pretty good with python, but javascript has knocked me back into my reflexive write-everything-in-one-big-blob mode.

For now, I'll just say that I thoroughly enjoyed this once I figured out what I was doing.   It feels so good to stretch my math muscles again, however small the stretch may be.

I still would like to make the output more interactive, so the stars rotate when clicked, and display an image of the day's weather.  I know how I'd like to do it (just redraw the stars starting at one place further along the array).

The other project for the day was expanding a javascript version of blackjack that one of the instructors had started.  The base version had a rudimentary game engine, and the cards and deck defined (with images).  There was a game clock that monitored the state and redrew cards and printed text as needed.  

Reading expert code may be my greatest joy here, other than my own occasional breakthroughs.  It's so interesting to see how people break a problem into pieces and make them work.

The really fun part was unravelling what was going on since there was no documentation provided.  I'm happy to say that by the end of the day (about 4 hours), I'd managed to implement a scoring system, and add hit and stand capability.  I can see the way to finish the project.  

I have so much to learn still.  I'm definitely at the "OMFG" stage of learning, or the "desert of despair" from the Viking Code School blog. I'm at the point where I'd felt like I kind of knew what was going on with python...and now I'm realizing just how deep the well is, with sql, django, javascript, ajax, server management...it's dispiriting. It's not that I don't want to learn all that (I love learning, after all).  It's more that there's just so much before I can do anything useful.

Unless you count n-sided rainbow stellate polygons as useful, of course. :)