Undergrads! – PhotoGeometry

Optimization for Diffraction Grating Groove Data

sarahhowellAugust 1, 2019Leave a comment

Utilizing the scripts outlined in previous blog posts by Grady and I, we were able to create an optimization function to solve for the groove distance and orientation of the diffraction gratings on our holographic glitter sheet.

Using the data gathered from the plot, pictured above, we were able to get the angles needed for our diffraction grating formula

With the left half of the equation solved using the angles we got from our graph, and the wavelength solved using the method outlined in last week’s blog post about wavelengths, we were able to create an optimization function to solve for our missing information: the distance between the grooves and the orientation of the grooves.

However, there seems to be multiple combinations of orientations and groove distances that can produce the same angle combinations, therefore we need to use more information for our optimization function.

We decided to use all of the lines from a lit square on the monitor to one glitter board to see if acquiring more data to run our optimization on would provide more specific results.

However, there are still multiple combinations of distance and orientation for the grooves that result in near-zero values for the error value. To combat this, we are looking for more parameters for our optimization function that would add constraints to the answers we receive, such as a minimum or maximum value for the groove distance. We have begun working on a script that will look at all pixels and their angles to light squares on the monitor, rather than just one pixels’. Hopefully this large amount of data will produce more specific results from our optimization function.

Colors of One Holographic Glitter Pixel

sarahhowellJuly 25, 2019Leave a comment

Grady and I have been working on a script that takes in a large set of pictures of a sheet of holographic glitter, where a picture is taken at every location of a small white box moving along a monitor across from the glitter sheet. The graph we created shows the color of a single glitter pixel when the white box is at that location on the monitor. From this data, I combined the graph with the script I wrote a few weeks ago and created a graph that shows the same thing, just displaying the closest monochromatic wavelength to the RGB value shown and recording that value.

Any photo where the pixel we selected was not lit was not used for this process, as well as any photo where the pixel was fully saturated, or not saturated to at least 15%.

We plan to use these lines to compute a various set of angles that we need, along with the wavelength, to determine the groove spacing of the diffraction grating on our specific holographic glitter sheet.

Some Project rePhoto Updates!

skleingartnerJuly 25, 2019Leave a comment

I spent a good amount of time in the last week planning and fine-tuning what would be best for the models (database tables) and how they need to be able to interact with each other. I also have been working on getting the project set up on the server and the web app moved over onto the server.

Below shows one of my diagrams of keys (primary and foreign) that I will be using in the models- this took a couple of re-dos in order for me to consider it efficient, but I learned some really important lessons about relations between tables and had to seriously think through how I will want to be accessing and changing these databases through the web app.

Having never worked with a database before, let alone a framework like Django, I really have had to dedicate a great deal of time to understanding how different elements work with one another, and how to configure the whole set-up so it actually works. Not that I have completely mastered anything, or that I’ve done every Django and SQL tutorial out there (just... most of them), but I really am feeling more confident about using Django and creating a back-end for the web app that I ever anticipated.

A couple successes & updates in the last week:

Dr. Pless suggested an addition to the web app (for right now, as the database is not up yet) - storing the photo that was last taken on the app as the overlay for the next photo. I used the LocalStorage JavaScript API, which I had already used to display the photo after it was taken on the next page, to make this happen and while it will only show the last photo you took using the web app, it’s still a pretty cool thing until we can get the database up and running!

The photo on the left shows the overlay in action- the blurriness is due to my shaking hand in trying to take the screenshot, but it was really great to see a sneak peek of what the web app will eventually become. The photo on the left shows the page after the photo is taken, displaying the photo with the choice to hold down to save to your camera roll along with sharing on social media and other options that I haven’t worked on yet.

I’ve been working on the plans for the flow as well as appearance of the site and have been drawing out my ideas, so there is a plan for the front-end! I’m currently more focused on the back-end, but when the time comes, I’m really excited to start working on giving Project rePhoto the eye-catching, modern look it deserves!

We’re still trying to come up with a Project rePhoto tagline- a couple of strong ones so far have been “chronicle change in your world” and “track what you care about”, and I’d love to hear more! There’s sticky notes by my desk if you ever think of an idea!

I would really appreciate any feedback that anyone has on what I’ve done so far; thanks for reading and happy Thursday!

Approximating Wavelengths from RGB Values

sarahhowellJuly 3, 2019Leave a comment

In the past few days I've been working on finding away to approximate the wavelengths of lights in photos. This isn't technically possible to do with complete accuracy, but I was able to write a script that can approximate the wavelength with some exceptions (mentioned later in the post).

Using data from a chart that has the XYZ colorspace of all monochromatic wavelengths of visible light, I was able to convert my RGB image to XYZ colorspace and find the closest wavelength value to the XYZ values of the pixel. I've produced a histogram of this script in action on an entire image, however I also produced a smaller version of the script that can be used as a function and convert one pixel at a time from an RGB value to wavelength value.

Here is the histogram of wavelengths of light found in the photo:
Here is a histogram of the hues present in the same photo:

As you can see, there are a few issues with the wavelength approximations. Some color hues (most notably pinks and purples) cannot be produced using a single wavelength. My script matches the RGB value of the pixel to the closest monochromatic wavelength, and therefore colors that cannot be created with one wavelength are approximated to the closest component wave, which can explain why there are larger amounts of blue and red light shown in the wavelength histogram.

This behavior seems fairly consistent across images, but in the future I might have to figure out a way to approximate these more complex colors if we find that more accuracy is needed. For now this script gives us an accurate enough idea of the wavelengths of light produced in the image in order to begin using holographic light diffraction grating equations to approximate other information such as the angle of the incoming wavelength before the diffraction occurred.

Planning the Project rePhoto Web App

skleingartnerJune 21, 2019July 26, 2019Leave a comment

The last week has seen a few dozen changes and improvements to the flow and functionality of the web app for Project rePhoto. Some changes of note: I added the ability to create new projects and subjects (requiring one subject with one entry to be created when the project is), the overlay will always be the first photo that was uploaded to the subject, and you can upload photos to a subject as a URL, which will then be converted to an .png image by a Python script I wrote this week.

Here’s a diagram I made to map out the flow of the web app, which is reflected in the current version of the web app (though it is a skeleton version without any backend functionality): rePhoto-flow

I also spent a decent amount of time earlier this week taking a deeper look at and getting more comfortable with Django. This is my current plan for the file management for the functionality of the web app: rePhoto-file-mgmt

I’ve also been working, without too much luck, on saving the photos locally on the devices they are taken on, and am working today on getting the “hold down to save” functionality working (which would be clunky but users are relatively familiar with doing this on their phones).

A few questions that I’ve been working towards answering:

When making the choice to require each new project to have at least one subject, with one entry, I thought about projects like RinkWatch (a collection of ice rinks around North America) and Scenic Overlooks (a compilation of views from various hikes), which have had a lot of subjects, but very few entries. Should we require the user to add an entry to each subject when it is created, even though we would no longer be able to allow for projects like RinkWatch?
Instead of providing a list of subjects, which would require our users to know the exact name of the subject they’re looking to add to, I thought it might be useful to use an Open Street Maps API, like the one that already exists in the View Projects section of the rePhoto website, to allow users to be able to view the subjects on a map before choosing one to contribute to. Kartograph (https://kartograph.org/) also seems to be another option, so I’m going to do some research in the next week about which one fits the goals of our project better, and would love any input on which might be best or why Open Street Maps was chosen for the Project rePhoto site a few years ago.
Is it best to keep the Projects -> Subjects -> Entries pipeline? Otherwise, we could use tags to connect subjects to each other, which would serve the organizational purpose of projects. Why I’m excited about tags- many people, across geographic boundaries, can see each other’s work; for example, with a “home-improvement” tag, someone in Alaska and another in Brazil could both be building outdoor fireplaces, and get ideas from watching the evolution of each other’s projects. My only concerns- if a user is not required to put any tags on their subjects, we could have a lot of subjects that wouldn’t be tied to anything, making the process of organization and re-finding subjects a bit more tedious.

I’d love any feedback on what I’ve done and what should be done next, and happy Friday! 🙂

Amber Alert: Is This Working?

devinkoppMay 9, 2019May 9, 2019Leave a comment

We corrected our experiment from last week.

This time, we selected a subset of the 8x8 grid of the section we wanted (the truck). We then zero'd out the vectors for the rest of the 8x8. We then ran the similarity again, against another image from the same class. Here are the results (left is vector zero'd out besides for those squares determined to be on the truck, right is just another image from the class):

image_0 -> image_15 within the class

image_7 -> image_15 within the class (so we can capture the back of the truck)

So.. kinda? It *seems* like its paying attention to the text, or the top of the truck, but it doesn't seem to care about the back of the truck, which was surprising to us because we thought the back would stay in frame for the longest time.

We thought it might do better on small cars, so we tried this experiment as well:

We've been a little worried that the model is paying attention to the road more than the cars themselves, which this image corroborates?

This prompted another experiment, to see what it thinks about some generic piece of road. We just selected 1 tile of just road, and we wanted to see where it mapped.

Interestingly, the most similar portion was the road right around it, which was very confusing to us as well. Our hypothesis was that this generic piece of road would either: map to itself only (both are uncovered in these 2 pictures), or map to most other pieces of open road. A little miffed by this.

Amber Alert: Cropped Similarity

devinkoppMay 2, 2019May 2, 2019Leave a comment

This week, we're still trying to see if the network is really learning something about the vehicles in our images.

We cropped one image with a big white truck:

We then ran the heatmap on this, against a different image in the same class (so it's LITERALLY the same truck).

Our hypothesis was that the ONLY possible thing in the image that could be similar between the two would be the trucks.

We ran this test a couple times, moving the cropped truck around and here were our results:

You can see... not great.

Some of our theories on why the model might not be so great at tracking cars is that it really only needs to pay attention to some things in the scene, not necessarily every single vehicle.

We're also thinking that, because our classes have frames that are very close together, the model always has a nearly identical image to look at. If we skip more frames between images in our classes, this could help this problem.

Our plans for next week are to:

Spread out the frames within our classes, so the model will have to keep track of cars over longer distances/ won't have another image that looks nearly identical
Get new data, with less traffic jams
Create long video of the highway

Amber Alert: “One of the coolest things I’ve ever seen” – Abby 1

devinkoppApril 25, 2019April 25, 20191 Comment

With Hong and Abby's help, we were able to get the heatmap up and running on our model. Here you can see the similarity visualization across a couple of our classes as time passes (Left image is always the first image in the class, the right image advances timewise):

class80:

class120:

It appears as if these blobs are following groups of cars/ specific cars. Which would mean that it must be learning something about cars themselves. Its kinda hard to see how specifically it is paying attention to certain cars when they are close together, due to the fact that the resolution of the heatmap is 8x8.

One concern I have is that it only seems to care about some cars and not others. Would anyone have any insight as to why? This could mean that, if we were to use the embedding to identify cars for our amber-alert problem, we could get 'unlucky' and miss the car we actually care about.

Next week:

Do this visualization on our training data too
Figure out our next steps

Amber Alert: Image Similarity 1

devinkoppApril 18, 2019April 18, 20191 Comment

In order to accomplish our goal of figuring out which pictures were being misplaced by our embedding network, we took some time to introspect on Hong's model accuracy calculation function. In doing this we learned/ confirmed some things about how the linear algebra behind embedding works!

We saw how, by multiplying the high dimensional space vector by its transpose, will give us the distance from each vector to every other vector. We then sort to find the closest vector to each other vector. We set the prediction of the given vector to whatever the class is of the closest other vector to it.

Here is an example image that the network misclassified (raw image on left, highlighted similarities on right):

Test Image (from ~1min from video start):

Closest image in High dimensional space (from ~30min from video start):

Here are the images again with some highlighted similarities:

Test Image:

Predicted image (closest in high-dimensional space):

Another similarity we saw was the truck in the far right lane, and far left lane. They look fairly similar, and it would be reasonable for them to be this far apart within the time period of one of our classes.

The first glaring issue we saw were the trucks/ lack of trucks in the bottom left. We then thought it may be fine, because it makes sense that, within a class it is likely for new vehicles to enter the scene from this point, so the model might allow for a situation like this.

We attempted to use the heatmap, we have most of it set up, we gave it a first attempt, but we just think we are plugging in the wrong vectors as input. (we can talk to Hong/ Abby more about this). Here is our first attempt (lol):

Next week:

Find the right vectors to use for our heatmap to confirm the car-tracking ability of the model

Amber Alert: Improvements

devinkoppApril 11, 2019April 11, 2019Leave a comment

This week we focused on making small improvements to our process/ model.

We fixed our training/test set so there is a time gap between the training and test classes. Before, the last image in each training class would have been directly adjacent, time-wise, to the first image in one of the testing classes. While validating, this would mean that some of the images would be almost identical to images it has already seen in training.

We also did some of the other tests, like the overfitting test to see that the loss went to 0, and it did.

Another issue, which we posted about before, was that we were getting our training accuracy to be 0. We found this .......

acc_tra = 0#recallAcc(Fvec_tra, dsets_tra.idx_to_class) (oof)

Our goal is also to see some of the images that the network is getting wrong (what class did the network guess/ what class is it supposed to be). We're close to getting this done. Part of the code maps an arbitrary class name to a specific index ('dog' -> class 7). We were able to obtain this specific index of the expected/actual of the incorrectly labelled images. All that's left is to translate this index back into the original class names. Then we can see guessed/actual of the wrong images.

We also switched our model over to resnet50 (previously resnet18), and retrained over 10 epochs.

For next week:

Finish up checking out mislabelled images
Use Abby's heatmap to visualize similarity within a class (to confirm the model is 'tracking cars')