[GSoC] Interest in Add OpenVINO support to Google Mediapipe and Add Video Anomaly Detection Inference to Gradio

[sahilpmehra]

Introduction

My name is Sahil Mehra, and I am a Master's student at University of California, San Diego specializing in Image Processing and Computer Vision. I have also worked at Oracle as a Software Engineer for a couple of years. Prior to that, I received my Bachelor's degree from Birla Institute of Technology and Science (BITS), Pilani in India.

I fell in love with Computer Vision in my sophomore year and haven't looked back ever since. In the last 6 years, I have been part of many research projects related to Computer Vision and have had extensive experience in implementing the latest Computer Vision algorithms, with a particular focus on 3D Vision algorithms. During my thesis on Weakly Supervised Human Geometry Estimation from Monocular Images, I collaborated with the Video Analytics group at the Indian Institute of Sciences. In this role, I was responsible for optimizing and developing the model pipeline, and automating visualizations. Our paper was accepted as an oral presentation at the prestigious ECCV 2020 conference.

Lately, I have been taking up graduate level courses related to GPU programming and Deep Learning, and working on a project using C++ for improving performance of models on GPUs.

I am very excited to make a meaningful contribution to OpenVINO and help others like me in the future.

NOTE: I am almost done with my prerequisite task and would be grateful if the mentors could help me find a suitable initial task and/or issue according to the projects I am interested in.

Interested Projects

For GSoC'23 I am particularly interested in the following two projects:

1. Add OpenVINO support to Google Mediapipe

Mentored by: @ngaloppo and Radwan Ibrahim
I have excellent C++ skills and have also been using C++ in my latest DL related research project at UC San Diego. I have also used MediaPipe and am willing to learn and contribute to this project to show my capability. Although challenging, I believe I can do justice to this project idea with my skills and passion.

7. Add Video Anomaly Detection Inference to Gradio

Mentored by: Samet Akcay and Dick Ameln
I have been using Python for all my Deep Learning related research projects so far. So, I am pretty confident with my Python skills. I also have experience using Gradio for building a Chess Engine web app for myself. The optional addition of a new algorithm for detecting video anomalies to Anomalib makes the project interesting and challenging.

Thank you for all your efforts. Hope to hear from you soon!

[adrianboguszewski]

@ngaloppo, @Radwan-Ibrahim
@samet-akcay @djdameln

Guys, please look at this

[ngaloppo]

Hello Sahil, thanks for your interest!

For the MediaPipe project, here's an initial task you could perform: implement a new calculator (and test) based on the following requirements:

// Calculator for normalizing and converting an ImageFrame or Matrix
// into a ov::Tensor or a GpuBuffer to a ov::RemoteTensor
//
// This calculator is designed to be used with the OpenVINOInferenceCalculator,
// as a pre-processing step for calculator inputs.
//
// IMAGE and IMAGE_GPU inputs are normalized to [-1,1] (default) or [0,1],
// specified by options (unless outputting a quantized tensor).
//
// Input:
//  One of the following tags:
//  IMAGE - ImageFrame (assumed to be 8-bit or 32-bit data).
//  IMAGE_GPU - GpuBuffer (assumed to be RGBA or RGB GL texture).
//
// Output:
//  One of the following tags:
//  TENSORS - Vector of ov::Tensors
//  TENSORS_REMOTE - Vector of ov::RemoteTensors
//
// Example use:
// node {
//   calculator: "OpenVINOConverterCalculator"
//   input_stream: "IMAGE:input_image"
//   output_stream: "TENSORS:image_tensor"
//   options: {
//     [mediapipe.OpenVINOConverterCalculatorOptions.ext] {
//       zero_center: true
//     }
//   }
// }

Feel free to implement in a private repo/fork and share with us.

[sahilpmehra]

Hi @ngaloppo, thanks for this. Will get back to you soon.