Skip to content

Course work I: COMP0160 Perception and Interfaces

Release date: 24th January 2022

Due date: 7th February 2022

Instructor: Kaan Akşit

Background

COMP0160: Perception and Interface Course offers students a gateway to get familiar with various aspects of perception and interfaces. This document is designed to explain to you the task that has to be conducted by students for delivering a successful work as an outcome of your assignment. Specifically, the documentation is for the first-course work in the perception and interfaces course. The topic of the first-course work is related to the human visual system. As the students compile their course work, they will have a strong understanding of how the human visual system works under different eye prescriptions (e.g., myopia, astigmatism). Having a detailed understanding of the given topic can help students to have insights towards solving problems in domains such as computational displays, perceptual graphics and computational imaging. The software tools used in this course are publicly available. They are shared across the forefront of various industries and academia (e.g., data science to computational approaches in physics, biology or chemistry).

Requirements

This assignment assumes that you have an understanding of programming with Python language and you are familiar with Torch library that provides access to linear algebra calculations. Within this assignment, you will be asked to deliver your solution in a Jupyter Notebook format. The students are expected to be familiar with working Jupyter Notebooks and know the details regarding saving notebooks so that they can deliver their work for evaluation in the required form. We typically use Matplotlib library for plotting purposes while using Jupyter Notebooks.

In our production machines, we use the Python distribution 3.9.7, Torch distribution 1.9.0, Matplotlib distribution 3.3.4 and Jupyter Notebook distribution 6.2.0. For successfully compiling the assignment, make sure to have these libraries installed on your computer properly. Given that you are going to compile your work with Torch, you can either choose to run your code on CPU or GPU by selecting the proper device in your code. However, at the time you deliver your code, please make sure that your code runs on CPU. As a practical observation, you can also get a sense of speed differences in between those two devices and report within your Jupyter notebook (optional). We typically run these on a Linux operating system. However, it is not a requirement for students to use the same operating system as these components also run on your favourite operating system (e.g., Windows, Android, Mac OS or alike). In your course work, make sure to add docstring type documentation for every function in your code and make sure to comment in between lines to explain your steps within a function.

Before starting with the tasks, we encourage students to attend the second lecture of the perception and interfaces course, namely Lecture 2: Visual Perception in Perceptual Graphics and Computational Displays (recording available on Moodle).

Special note from your instructor: We design this homework to make you be better in your next in your life. If you do not have the right background to use the tools proposed in this coursework or if you are a confused absolute beginner, please do not hesitate to reach out to us through Moodle. We are here to support you. Please carefully frame your questions as you approach us for support (e.g., what you want to ask and what you expect) so that we can support you at our best.

Problem description

Each and every one of us has a unique visual system. At the heart of our visual system lies our eyes. Our eyes can be simplified as an optical instrument that images a three-dimensional scene to our retinas, sensor alike cellular structure. In this assignment, your task is to develop a user interface on a Jupyter Notebook that simulates how our visions are being affected by various kinds of eye prescriptions. To simply put, how would you perceive a scene if you had a certain prescription. We expect you to have this simulator in live view, enabling the user to choose different eye prescriptions based on Zernike Polynomials. Before conducting any work, we suggest you go through the listed references below:

In addition, you can get a sense of the importance of prescription in next-generation display technologies by going through the survey paper below (actual industrial applications from learning of this course work):

These references can help you to find the required technical details for your subtasks.

Zernike Polynomial generator (10 points)

The first task is to derive a Pythonic class that can generate Zernike Polynomials on demand. These polynomials can help you to represent the point-spread functions of people with a prescription. A point-spread function can be best described as the system response of your eye to a given scene. In the way you will use point-spread functions, they can be described as kernels that can help you describe your eye as a linear transform or a system that is represented with a single convolution. Once you have fully compiled the zernike_polynomial_generator class in your Notebook, please proceed with the next task:

class zernike_polynomial_generator():

    def __init__(self):
          ...

Our expectation from you, in this case, is to have multiple functions in your zernike_polynomial_generator class that spits out various point-spread functions, read more from here (7.5 points). A person may be having an eye prescription composed of various point-spread functions. To support such a case, make sure to add a function in your class that outputs a weighted sum of chosen point-spread functions (2.5 points).

Forward model (10 points)

We will work with an assumption that our eyes are responding to every point on a given scene in the same way (e.g., stationary kernels, not spatially varying kernels). You have to have a function that is able to load images from a given path.

def load_image(filename):
     """
    Function to load an image.

    Parameters
    ------------
    filename            : str
                                Filename of the image.

    Returns
    --------
    image               : torch.tensor
                               Loaded image.
     """
     ....
    return image

Please do not hesitate to use images from Creative Commons for your experiments, and please make sure that these images are clean, meaning ethically good to work with. Please also make sure to work with images that has 1920x1080x3 resolution and please reduce the image into a single color image by taking the average across second axis (1920x1080 - Black and white). Make sure to provide the image that you use together with your Jupyter Notebook in a compressed file format (ZIP).

You will be using this specific image load definition to load images, and you will process these images with your forward model function. Here forward model corresponds to convolving the loaded image with a combination of Zernike polynomials to simulate various kinds of eye prescriptions. In the simplest form, your forward model should look like below:

def forward(image, psf):
     """
    Forward model, convolving the given image with a given point-spread function.

    Parameters
    ------------
    image              : torch.tensor
                               Image as a torch tensor (MxN).
    psf                   : torch.tensor
                              Point-spread function as a torch tensor (MxN).

    Returns
    --------
    result               : torch.tensor
                               Abberated image.
     """
     ....
     return result

You will receive your 3 points for loading images properly (3 points). The remaining 7 points will be dedicated to the forward model definition (7 points).

Hint for the forward model: torch.nn.Conv2d (Do not necessarilly have to use it but can help).

Visualizer (10 points)

The last bit we want you to add to the Jupyter Notebook is related to the way you will be visualizing the outcome of your forward model. We want your code to be interactive as much as you can make it to be on your given computer hardware. Make sure to visualize images of your forward model using Matplotlib. Make sure to provide buttons and controls for your users to choose different combinations of Zernike polynomial to formulate a point-spread function, and make sure to visualize the point-spread functions that you have generated. Note that we will heavily be relying on your visualizer to assess the outcome of your code; please pay attention to make sure that you have provided all the controls (either as variables to manipulate or buttons or sliders), and they are easy for a user to work with. Note that you are allowed to use other libraries beyond Matplotlib such as Pyplot or if you want to develop a user interface outside of the boundaries of a Jupyter Notebook, it is also ok. But if you do that, please make sure that you have communicated the change in a clear fashion and we are able to run your code.

If you can plot the outcome of the forward model, this plotting can guarantee you half of the points you can receive (5 points). The remaining points can be received as you introduce more sophistication to your visualizer, as explained above (5 points).

Problem and potential solutions (15 points)

We want you to add a text section to your notebook, where you will find an unsolved/partially solved scientific problem related to eye prescription and visuals (displays, graphics or any other form). The source of this problem can be from the existing literature, and please make sure to survey using your favourite search engines, academic ones and non-academic ones (e.g., Brave, Google Scholar, etc.). The problem can also rely on your practical observations as well as long as you describe it clearly. You also provide potential solutions to the problem that you have found from the literature and your own predictions towards new solutions in the future. The text can not be more than 500 words, no less than 250 words. Note that the length of your text is not an indicator of success, and most powerful writing happens in shorter forms.

You will receive half of the points from your problem description (7.5 points). The remaining half will be from your proposed solution (7.5 points).

Contacting Us

The prefered way of communication is through University College London's online lecture system, Moodle. Please use the Moodle forum for your questions related to the course work.