Rendering Physically-Based Lens Flares
Current methods for producing lens flare distortions are scene-independent, in the sense that they render a flare distortion without regard to the camera viewpoint or position of the sun, and synthetically overlay the distortion onto a scene that had been previously rendered or captured. In this project, we modified a ray-tracing engine to capture infinite light sources in a scene (like the sun), and render the physics-based lens flare distortion that would appear as if the image was taken by a camera. Half of this implementation is based on a Fourier method, and the other half is based on heuristics of light. Unlike some approaches, a rendered lens flare in our approach completely depends on the viewpoint and the location of the light source.
NumS - Open Source Contribution
NumS is a Python library that translates n-dimensional matrix operations across a distributed network of computers by using Ray as a backend for scheduling tasks, and managing the memory and resources of a node cluster. For my final project and as an open source contribution, I implemented an algorithm (based on Bientenisi et al.) that performs a matrix inversion on upper-triangular matrices, specifically the R matrix from QR factorization. We can use this algorithm for least squares regression on extremely large data with any number of features, and subsequently show that this algorithm performs better than naive matrix inversion.
RoBob The Builder
Built a robotic simulation upon the idea that a user can specify a unique structure composed of blocks and a robot will process the structure and attempt to recreate it autonomously. We used the PAL Robotics TiAGo robot model to perform 2 primary tasks: (1) in an initial world with a desired structure created by a user, use TiAGo’s frontal cameras to capture the structure and extract the 3D coordinate points of the blocks that make up the structure, and (2) autonomously navigate to and pick up randomly placed blocks in a new world, and place them in in a way that replicates the desired structure. For this project, we used OpenCV to process the strutural images and extract coordinates, and used ROSPy and ArUco to actuate TiAGo autonomously to perform pick up/place down actions.
Video Summarization Using REINFORCE
With inspiration from Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward, Kaiyang Zhou et al., we built a paired unsupervised and supervised pipeline to train a network to select key frames of interest from videos using deep reinforcement learning. Also, we added two custom reward functions: (1) a time-delta reward that yields high reward for clustering frames in a video summary, and (2) an object-ness reward based on the YoloV3 that encourages the policy to choose frames that may contain an abundance of objects. Our network, coined FLOCKAS, was tested on the SumMe and TVSum video summarization datasets and achieved an F-Score of 45.5% and 58.9% respectively, beating the previous benchmark by 4.1% and 1.3% respectively.
(Code is not released as we are currently still performing further research in this domain)
Yelp Me Rate This (Yelp Rating Prediction)
Experimented with various custom and state-of-the-art NLP models to predict the star rating of any given Yelp review for any business. Used PyTorch to build a Bi-LSTM with a decoder and a base transformer, and used HuggingFace to load pre-trained large transformers such as RoBERTa and XLNet. After training and evaluation on the Yelp dataset, our team achieved a peak performance or near 80% accuracy on the full dataset and 60% on the challenge datasets.
This-or-That (Web and iOS App)
ThisOrThat is an application made to allow a community of users to vote between two choices. Users can create a poll with two options and the rest of the entire user base can vote on either one. By the end of the voting period, users can see the final results and choose an option if they wish to. Used Swift to develop the iOS application and Ruby on Rails for the web application and iOS API.
nxtBook
nxtBook is a native iOS mobile app that allows students to engage in peer to peer textbook exchange instead of using Facebook and/or other general e-commerce sites. Used Swift to create the mobile app and Ruby on Rails for the API backend.