I added micro-gestures (head nods, wink, eyebrow movement) to animated avatars.Synthesia is a platform where users can generate videos of their own avatars. I was part of a team of four Research Engineers that worked on adding new capabilities for the avatars.

In a team of two people I worked on the implementation of a Motion Matching algorithm. The goal of the project was to create realistic human animations of different locomotion styles (running, walking, jumping, crawling). My task was to design appropriate features that can be extracted from recordings and used to train neural networks to generate new motion which allows a character to follow a given path. The machine learning part was done using Python (scikit-learn and PyTorch) and the visualization in Unity.In another project I worked on the generation of layouts using Generative Adversarial Networks (GANs).

I did research on facial landmark detection using Convolutional Neural Networks (CNN) as a follow-up work on my master's thesis. In this part of the internship I mainly used Python and PyTorch.I contributed to the creation of the CMU-MOSEAS dataset.At the end of the internship I developed web-based tools for multi-modal dataset creation and annotation using JavaScript, TypeScript, React, Redux, HTML and CSS.

As an exchange student I worked on my master's thesis on facial landmark detection and generative shape models.The goal was to improve the accuracy of facial landmark detection. Given an image of a human face, the system predicts the location of 68 landmarks. I implemented a system based on Stacked Hourglass Networks which are a special type of Convolutional Neural Networks (CNNs) that encode and decode an image multiple times in order to produce heatmaps of landmarks. These heatmaps are then converted into 2D numerical coordinates using a Differentiable Spatial to Numerical Transformation (DSNT). The network was trained using a special loss function called WingLoss. In order to further improve the results, an additional shape model was trained that can refine the predictions of the Stacked Hourglass Network based on the confidence of each landmark.I used multiple AWS EC2 instances to run a big hyper-parameter search and find the optimal number of Stacked Hourglass Networks, the number of features in each layer, the optimal learning rate, batch size, etc.More details can be found in my thesis: https://simonhessner.de/wp-content/uploads/2019/09/masters_thesis_simon_hessner.pdfTechnologies: Python, PyTorch, Deep Learning, Neural Networks, Computer Vision, Machine Learning, CNNs

At Philips Research I worked on detecting emotions in human conversations using different machine learning techniques. I used word embeddings such as Word2Vec, Doc2Vec and GloVe to represent transcripts of human conversations. These representations were processed and analyzed using recurrent neural networks (LSTMs).I pre-processed the conversations using tools from NLTK and the embeddings were generated using the gensim library. The code was written in Python and the neural networks implemented using PyTorch.

I was part of the Lecture Translator team at the Interactive Systems Lab (ISL). The Lecture Translator is a tool that allows foreign students to follow lectures in German without speaking German. The lectures are recorded, the speech converted to text and translated to the target language in real-time.In order to make it easy to use the system, I implemented iOS and Android apps that professors can use on their phone in arbitrary lecture halls without having to rely on a stationary recording system.The apps record the speech and send it to the server via an API that handles the real-time speech recognition and translation. The same API can then be used by the student client which shows the lecture content in their native language on the screen.I used Swift for the iOS app and Java for the Android app.

After finishing my bachelor's thesis I was hired as an assistant to continue and improve the project. The thesis was about implementing a reliable multicast communication protocol that used unreliable UDP broadcast under the hood.The code was written in Python and tested with mininet.

This was a greenfield project where I was the first hire. The project was a social network where dancing enthusiasts can find training partners. My responsibilities included designing the database, implementing front-end and back-end and administrating the web server.In the back-end I used PHP with the Sympfony framework and MySQL as the database. On the front-end JavaScript with jQuery, CSS with Bootstrap and HTML was used. The nginx webserver was running on a Ubuntu virtual machine that I set up on a rented root server.At a later stage of the project I added a REST API to enable mobile apps to be integrated in the system.

I developed a log file viewer and analyzer using Python and Django. The tool included features to filter, sort and highlight different events from log files and log streams.