I design (and implement and evangelize) MediaPipe solutions, which are easily consumable, bite-sized, augmented reality components that developers can use on virtually any device. These solutions include things like pose estimation, finely tuned face meshes, human segmentation, and 3d object tracking. All of these are neural nets bolstered by easy-to-use, customer facing APIs. You want to blur the background during a meeting? Attach signage to everyday objects? Overlay make-up effects on a human face? Do puppetry? This is where it happens.I've spent the last several years integrating Google research into various Google products as well as extending our outreach to 3P open source communities. If you've gotten a chance to ride in a WayMo vehicle, or you have a Google Home product, or you use Lens, or even YouTube, you've probably been running some of my code. If you've been in Google Meet and you've turned on any of the visual effects, then you've seen my work there as well.I've worked on our underlying MediaPipe architecture (C++) that runs just about anywhere (Android / iOS / Windows / Mac / Stadia / Coral devices / et al), see mediapipe.dev. I've invested a good bit of time getting all of our neural network models and code running inside of WASM on the browser (via WASM and JavaScript/TypeScript). I maintain a lot of our demos at code.mediapipe.dev, so if you have a camera on your laptop, go check it out.My passion has been to get our tech into the hands of creatives, and I've recently moved from Research to Google's CoreML team, which allows me to focus on a lot of our 1P friends and get them using a standard set of AR components. I've exposed some simple editors in the past, and these were geared towards our engineer friends (viz.mediapipe.dev), but I'm now focusing on feature rich IDEs that can be used by creatives to generate content directly into some of Google's popular services. Stay tuned!

Have you ever had to click on that little button that says, "I am not a robot?" Have you ever had to then find all of the cross walks, cars, turkeys, cats? Well... You're welcome. I hear what you are saying -- what a hassle, right? Well, hang on:Google's reCaptcha system gets about a billion or so clicks a month, and I supplied the challenge screen that happens occasionally. Why? Well, beyond the obvious reason that solving and typing answers to text puzzles on a phone is not great, I work in Deep Learning research on the Image Understanding team, and I want to help our applied scientists train our models on real data, we need _lots_ of data. We could pay annotators to help us fine tune our image recognition algorithms. Or... We could leverage the world to help us tune our models. This idea isn't new -- remember when puzzles used to be street signs and excerpts from books? Every wonder why Google has some of the best OCR and street view recognition? It's that data. If even a small percentage of those "I am not a robot" images gets a slightly annoying challenge screen, we can get millions of data points a month.Google has one of the best image recognition systems in the world (the best? Maybe. I'm biased). My interest has been to allow us humans to augment reality. But a computer can't augment what it can't understand. Unless we're happy doing edge filters, zooms, and saturation filters, we need to understand what's in front of the camera. This project was an outreach between image understanding and Google's reCaptcha team.This work contributed to models used by Google Lens, Google Image Search, WayMo, and many more.

I worked on the Core team for Google Play. I created "buttery smooth" animations for the Android Play application -- specifically Mobile. I also worked on making the store a more useful experience. I became interested in wearable technology, including Google Glass and Google Watches, but spent my time making the store more aesthetic. I studied up on graphics acceleration.During this time I also began mentoring and teaching architecture to interns, and I started championing a more artistic approach to application design. I worked with artists to bring SVG to the play store (strange that it wasn't already a thing). I worked on being faithful to typesetting standards and evangelizing the aesthetics of the store.After a couple of years, I felt strong in mobile development, but kept coming back to my passion in AR. I was writing personal demos in Unreal Studio and decided I wanted to do that full time. I had been making friends in Google Daydream, which was Google's VR arm. I had friends on the image understanding research team during my time with Search and I realized that augmented reality would soon be a thing. I positioned myself to transfer to Google Research in order to facilitate moving our research tech into customer facing applications.

Google search had need of a visual developer that could make attractive content for Google search. While on Google Search, I slid into a more visible position providing "smart cards" for the search page, as well as full page experiences on the web. My tech worked across a large spectrum of browsers, including mobile device browsers.I worked on a voting application that would pop up during certain live shows "The Voice." Lots of data, huge spikes of usage on the Google servers. Interesting visual component. I also worked on real time contextual news. A neural net would watch for trends in the news and would supply contextual information and create a timeline of the event by auto-curating content. I created a magazine layout for this content, as well as a few interactive 3D demos.I learned a lot about making user animations smooth within the browser, which became invaluable later when I added animations to Google Play.During this time, "Mobile First" was becoming a buzzword. Smart phones were becoming _much_ more popular, and Google Glass was still a thing. I moved teams in order to get closer to handheld device tech and to learn about graphical optimization for battery operated personal devices. I would soon join Google Play, as is seemed to be the team releasing and creating the most code for the Android phones and Android wearables (Watch, Google Glass, Android TV).

I had been working in Ontological reasoning (A kind of AI) and had previously worked for a finance company with their due diligence systems. This was an opportunity to be part owner in a company doing this on a larger scale for government facilities. Auditors would walk through government sites, checking for compliance. They have iPads in their hands, running through checklists and all of that data is uploaded and aggregated in real time. From a central hub, we can watch the animations and work unfold with graphical charts.The satellite software is web based, but can run offline. The central system actually ran inside of Unity, and was meant to be interesting to look at, and to show well at our booth. I was a business partner, but it was my job to architect and role out the software.While doing this job, I was also going to graduate school (again). I realized that my experience in AI was great, but that my passion was for a world more visual. Using Unity to build auditing systems was interesting, but felt shoehorned. As 3D capabilities on video cards became better, I realized I needed a change to something closer to information understanding, and, with luck, I could change my career to be more graphically oriented (see Google).

My background was in Ontological reasoning. Big data analysis and language understanding. Modus Operandi gathered SBIR contracts that revolved around understanding data and allowing people to make decisions or analyze data that was too complex for a single human to understand.A patent I own from here was on Ontological Federation. The inspiration was Jack Bauer from the show "24." He would ask his researchers to cross reference all known associates of some bad buy who travelled on a particular airline and were allergic to fish. The researchers would spin around in their seats and start typing. I laughed -- what were they typing? Were they firing off an email to their friends to complain about the latest impossible assignment?But I came up with the language they could speak. I designed a cross domain querying system that would use ontological relationships to federate those queries. I had experience with large scale data mining, and with semantic reasoning (for the technical -- a SPARQL style query across a triples data store). This allowed the machines to make surprising inferences.I eventually left for an opportunity to run my own company working with ontologies. Around this time, my passion for mathematics and graphics grew, and I started working on ray tracers and shaders. It would still be some time before I was able to put this into practice, but the idea of large data visualizations started becoming more important to me.

My background in applied artificial intelligence helped me create a rules based inference engine to help defend users against predatory lending and to help perform due diligence on mortgage lending. We analyzed documents for patterns that suggested improper handling, and our pipeline was efficient.The president of Lydian had told me that the success of companies like theirs was to create a toll booth that was cheaper than going it on one's own, and this stuck with me. Although I would add that the toll booth analogy never really sat well with me. In addition, I believe that a system should make a person feel relieved not to go a different way. There should be delight.I began getting involved in "emotional design." A design should be elegant, and it should be aesthetic. This was a huge evolution in my thought pattern.As for the job -- It was interesting building an inference engine from scratch. It was .NET and SQL Server and a large pub-sub system. It was a short lived experience, because the housing market in 2004 was toxic. It was interesting and I'm still friends with many of the developers. Would not recommend.

e-Security did real time analysis of potential security threats to an organization, whether through some faulty configuration or through an active intruder. We would boil down information with a correlation engine to report to an operator concisely and with recommendations. It was C++ and Java using Oracle and SQL Server (and eventually MySQL). When I started, I was to assist with the correlation engine -- a system which I redesigned and submitted to be rewritten by colleagues of mine. I ended up redesigning the communications layer, moving several antiquated communication technologies (including Corba) into a Publish-Subscription style layer. I also redesigned their e-Wizard product, eventually rewriting most of it to work with modern design patterns, providing out of the box functionality that simply wasn't possible before. I also rewrote the interpreter with a custom just-in-time parser. The system was eventually bought by Novell, and it's with some pride that i can still locate my documentation and API's on their site. This was an exciting job that started a side vision that I began calling "Code Zero", which was a visual intensive re-imagining of how one might run a large real time system, and how one might program for such a network. I would eventually get the chance to revisit this technology, albeit briefly, with NetBoss many years later.