Above: A frying pan being detected by the program in real time.
In the summer of 2016, I was selected as a Software Engineering Intern at the Koç University Arçelik Research (KUAR) design lab to begin development on a desktop application employing use of Computer Vision, Machine Learning and Spatial Augmented Reality.
As the sole engineer on this project, I accomplished the following milestones:
Calibrated a Projector and Kinect using Gene Kogan's Kinect Projection Toolkit.
Wrote a program to detect several kitchen utensils using the Kinect, PyOpenCV and Caffe.
Projected bounding boxes onto the utensils' real-world coordinates using the Projector and PyOpenCV.
Above: Project work setup including a table, Kinect and projector + System detecting an object and projecting onto it's real-world coordinates. The project required an immense amount of self study on my part as I familiarized myself with machine learning concepts and tools. Initially, I dealt with utensil detection using cascaded binary classifiers based on Haar-like features in PyOpenCV, and implemented a simple unsupervised confidence-based learning mechanism to reduce the number of false positives. To effectively track these detected utensils and surrounding regions, I implemented a pyramidal frame-to-frame optical flow procedure in Python, which was inspired by the classic Lucas-Kanade optical flow method.
Moving forward with this approach, however, would have had me train separate classifiers for each utensil that I wanted to detect so I changed course looking for other solutions. I elevated to using a model convolutional network (CNN) that had been trained on the ImageNet dataset. ImageNet contains over 15 million high-resolution images belonging to roughly 22,000 categories so it could already detect most utensils on my list. I collected my own datasets for the utensils that were not included in ImageNet.
Notable Experiences:
Developed a strong sense of self efficacy working as the sole programmer in a team of designers.
Improved at communicating ideas clearly to non-technical people without the use of jargon.
Became considerably experienced with the PyOpenCV and Caffe libraries for Python.
Learned how to structure and write code that is better for passing on to new owners indefinitely.