Kitchen of the Future
When: July 2016 - August 2016
Where: Koç University Design Lab, Istanbul
Role: Software Engineer

Language: Python
Technologies: OpenCV, Caffe, Kinect SDK
PhD project supervised by Dr. Asım Evren Yantaç, aimed at exploring everyday indoor applications of Projection Mapping and Spatial Augment Reality. Project Page: Spatial Augmented Reality for Indoor Use

Above: A frying pan being detected by the program in real time.

In the summer of 2016, I was selected as a Software Engineering Intern at the Koç University Arçelik Research (KUAR) design lab to begin development on a desktop application employing use of Computer Vision, Machine Learning and Spatial Augmented Reality.

As the sole engineer on this project, I accomplished the following milestones:
  • Calibrated a Projector and Kinect using Gene Kogan's Kinect Projection Toolkit.
  • Wrote a program to detect several kitchen utensils using the Kinect, PyOpenCV and Caffe.
  • Projected bounding boxes onto the utensils' real-world coordinates using the Projector and PyOpenCV.

  • Above: Project work setup including a table, Kinect and projector + System detecting an object and projecting onto it's real-world coordinates.
    The project required an immense amount of self study on my part as I familiarized myself with machine learning concepts and tools. Initially, I dealt with utensil detection using cascaded binary classifiers based on Haar-like features in PyOpenCV, and implemented a simple unsupervised confidence-based learning mechanism to reduce the number of false positives. To effectively track these detected utensils and surrounding regions, I implemented a pyramidal frame-to-frame optical flow procedure in Python, which was inspired by the classic Lucas-Kanade optical flow method.
    Moving forward with this approach, however, would have had me train separate classifiers for each utensil that I wanted to detect so I changed course looking for other solutions. I elevated to using a model convolutional network (CNN) that had been trained on the ImageNet dataset. ImageNet contains over 15 million high-resolution images belonging to roughly 22,000 categories so it could already detect most utensils on my list. I collected my own datasets for the utensils that were not included in ImageNet.

    Notable Experiences:
    • Developed a strong sense of self efficacy working as the sole programmer in a team of designers.
    • Improved at communicating ideas clearly to non-technical people without the use of jargon.
    • Became considerably experienced with the PyOpenCV and Caffe libraries for Python.
    • Learned how to structure and write code that is better for passing on to new owners indefinitely.