Machine Vision Project

Abstract

So far, you have successfully programmed robots using reactive control strategies and taken a deep dive into the robot localization problem. Next, you will be learning about machine vision and the role it plays in robotics. In contrast to previous project, this time you will have a great deal of freedom to shape your project towards your interests and learning goals.

Learning Objectives

  • Self-directed learning of a new robotics algorithm.
  • Implementing a robotics system with minimal scaffolding.

Teaming

For this project, you can work with one other student. If you want to have a team of three, please talk to the instructor beforehand. Please add yourself to a Canvas group as soon as you have a team!

Project Topic

Your project should be about machine vision and its intersection with robotics. You will carry out that project either using a robot simulator, a computer vision dataset, the Neato, an external camera system, or some combination.

In your project proposal, you will outline your project motivation, scope, and implementation and testing plan. You have substantial latitude in shaping your project to focus on the parts of modern or classical machine vision topics you really want to learn (e.g., system design versus basic understanding of algorithms). We expect that you will do some exploration of algorithms as part of this project (which could include implementation or perhaps substantial learning about an algorithm or class of algorithms), and will need to be strategic in your implementation to balance learning with building.

Potential Algorithm Topics

  • Object tracking or detection
  • Image de-noising or correction; in-painting
  • Image segmentation; semantic labeling
  • Text recognition
  • Gesture recognition
  • Fiducial tracking and navigation
  • Visual odometry or SLAM
  • Structure from motion
  • Neural rendering (radiance fields; Gaussian splats)
  • Vision-Lidar or Vision-Inertial sensor fusion

(Note: there are some resources for these topics later in the document)

Robot Platform and Data Pipelines

One option is to continue to work with the Neato. In class, you’ll see the Neato camera setup and go through a simple demo of how to use the images to control a robot.

You may want to use an external dataset for your project. Here are some possible starting points.

  • Visual Data

    and Papers with Code has a nice collection of computer vision datasets and projects.

  • If you want a huge (but very cool) dataset for self-driving vehicles, consider using Waymo’s Open Dataset
  • If you’re interested in machine learning for robot control, you might consider building off some of the datasets (and code) from the 2018 Robot Learning project You also might want to check out the Google Slides presentation summarizing their results.
  • Pick any open dataset of imagery in a domain of interest to you (e.g., for ocean biology FathomNet is a nice option; for city landscapes Cityscapes could be good; for food check out Food-101…and so on!)
  • A lot these datasets are big and fancy, but don’t be afraid to start with smaller, more classic datasets. That’s a great way to learn without having to deal with the greater complexity that comes with some of this data.

Finally, you could go all-in on sensor/perception only (abstracting away a robot) if you’d like to get deep in the weeds on a particular vision-based processing algorithm or fundamental piece of theory (note: the connection to robotics MUST be clear, however). For this, we have one ZED 2i stereo cameras available, and you can use your computer webcam, your phone camera/lidar, or any other visual sensor you may have access to.

Deliverables

There are four deliverables for this project.

Project Proposal (due 10/21 9PM)

At a minimum, please include the answers to the following questions. You should include enough detail for us to be able to give you useful feedback. Submit your proposal on canvas.

  • Who is on your team?
  • What is the main idea of your project?
  • What are your learning goals for this project?
  • What algorithms or computer vision areas will you be exploring?
  • What real-world applications will you be examining related to your proposed project?
  • What components of the algorithm will you implement yourself, which will you use pre-built code for?  Why?
  • What is your MVP? What is a stretch goal?
  • How will you test and evaluate your code?
  • What do you view as the biggest risks to you being successful (where success means achieving your learning goals) on this project?
  • What might you need from the teaching team for you to be successful on this project?

In-class Presentation / Demo (11/09 in class)

We’d like each team to spend about 10 minutes presenting what they did for this project. You can structure the presentation in whatever manner you’d like, however, you should try to meet these goals:

  • Explain your project and motivation
  • Describe your technical approach, including key design decisions
  • Demonstrate your system in action (either in a video [recommended] or live). If your system doesn’t work completely yet, that is fine, try to show at least one component of your system in action.
  • Note limitations of the implementation
  • Give real-life uses of the topics covered in your project
  • Highlight lessons learned
  • This presentation / demo should be very informal. This presentation will be assessed in a purely binary fashion (basically did you do the things above).

Please submit your presentation on canvas.

Code (Due 11/10 at 7PM)

Writeup (Due 11/10 at 7PM)

In your code package create a README.md file to hold your project writeup. We expect this writeup to be done in such a way that you are proud to include it as part of your professional portfolio. As such, please make sure to write the report so that it is understandable to an external audience. Make sure to add pictures to your report, links to Youtube videos, embedded animated Gifs (these can be recorded with the tool peek). Your writeup should touch on the following topics:

  • What was the goal of your project? Since everyone is doing a different project, you will have to spend some time setting this context.
  • How did you solve the problem (i.e., what methods / algorithms did you use and how do they work)? As above, since not everyone will be familiar with the algorithms you have chosen, you will need to spend some time explaining what you did and how everything works.
  • Describe a design decision you had to make when working on your project and what you ultimately did (and why)? These design decisions could be particular choices for how you implemented some part of an algorithm or perhaps a decision regarding which of two external packages to use in your project.
  • What if any challenges did you face along the way?
  • What would you do to improve your project if you had more time?
  • Did you learn any interesting lessons for future robotic programming projects? These could relate to working on robotics projects in teams, working on more open-ended (and longer term) problems, or any other relevant topic.
  • How does your project relate to real-life robotic systems?

Resources and Potential Project Directions

Resources

Some picks (good starting points for machine learning flavored computer vision projects):

  • UZH FPV Drone Racing Dataset: Get OpenVins or GTSAM running to do some pose estimation. Try to implement or compute the optical flow between image frames.
  • KITTI Autonomous Car Dataset. Implement or use YOLO or other CNN to do 2D object detection for AV environment (e.g., this tutorial). Some form of 360 video to BEV model similar to this tutorial.
  • Some Foundation Model (e.g., Segment Anything). Try to run and use the SAM model for some structured navigation task with the neatos (e.g., detect stop signs…)
  • Try to run VINT on the Neato

In-Class Activities from Past CompRobo Offerings and Other Tutorials

  • Object Tracking
  • 3D Structure from Motion
  • Image Filtering
  • Object Recognition

    (the instructions on running the code are out of date, but the ideas might help).

  • Convert ROS image messages to OpenCV using CvBridge
  • Visual Navigation for Flying Robots

    is a course on said topic. The linked page includes lectures and even some bag files.

  • Connecting a webcam to ROS and OpenCV
  • An overview of methods for object tracking
  • Slides from the CompRobo Learning Project 2018 Class Report Out
  • Mastering OpenCV with Practical Computer Vision Projects
  • Visual odometry resources (one example)
  • Canny edge detection
  • Template matching
  • Hough line transform
  • Basics of histograms

    and histogram equalization

  • Basic Numpy tutorials
  • GUI Features in OpenCV
  • Basic Operations on Images
  • Arithmetic Operations on Images
  • Corner Detection
  • Numpy Examples List

Past projects to draw from