TreeHacks

IMG_6946.jpg

TreeHacks was a blast. It was definitely one of the best (if not the best) hackathon I’ve ever attended. Everything was well-organized and there were plenty of fun involved. The venue – spanning over 3 floors at Stanford University’s Jen-Hsun Huang Engineering Center – was comfortable and spacious, and the food were delicious. With hundreds of participants from over half a dozen countries, TreeHacks was a relatively big hackathon, but the team behind it, along with Major League Hacking, did a fantastic job in maintaining diversity and inclusivity. Best of all, my hack, Clow, was awarded the “Best Big Data Hack”, and we got to bring back a 23andme kit sponsored by Oscar Health.

IMG_6947.jpg

Introducing Clow #

To give you a bit of heads up, Clow is a clinic-facing app that uses emotion analysis and machine learning to predict patients’ illnesses. To the patient, Clow acts as an electronic clinic check-in system for them to sign in before their session; during the sign-in process, Clow will capture an image of the patient’s face and will run it through an emotion analysis algorithm and generate a set of “scores” across multiple categories, including fear, sadness and happiness. After that, Dr. Clow – our machine learning service – will predict the patient’s illness using its predictive analysis algorithm, which is trained and improved by modeling the actual diagnosis result and the patient’s emotion “scores”.

Over a period of two hours on Valentine’s Day, my team and I enthusiastically presented our hack to dozens of judges, investors, mentors, and interested individuals. Throughout the demo, two questions were constantly asked; one of them, interestingly, was how we came up with the name; the other one, unsurprisingly, was how we built it. Both are thought-provoking questions that required us to sum up 36 hours of brainstorming and work into just a few sentences, but the answers get more concise as we continued to respond to both questions.
IMG_6955.jpg Me pitching to the Microsoft judges. Credit: Leon

Idea + Name #

It was past midnight, and I had never gone this far at a hackathon without a concrete idea. I was about to work on one of the projects TreeHacks proposed until one of the organizers came into our room and informed us about Microsoft’s tech talk regarding Internet of Things (IoT). Perhaps they were feeling generous, but they gave away complimentary Raspberry Pis and Microsoft IoT packs to the first 70 attendees. The presenter was talking about the Pi and the sensors that came along with it until he demonstrated some of the fascinating services offered by Microsoft Azure. He showed us how to deploy machine learning on their cloud service, all the way from setting up the model to setting up API endpoints for our newly trained service. When he was scrolling the list of services that we can integrate with, my eyes came across a facial recognition API and an emotion analysis API, both offered by Microsoft. And it was then I realized that both APIs, combined with machine learning and healthcare (the theme of this hackathon), could form a great idea. For some reason, the term “predictive analysis” appeared on my mind, filling in the final piece of the puzzle. Now that the schema formed in my mind, I went back to our room and pitched the newfound idea to my new friends whom I met at the mixer. My teammates, Dominick and Leon, liked the idea, and so we formed a team to work on it.

Of course, name was always a non-issue until we have to set up GitHub repositories and other relevant services. We brainstormed for a while and we put together ‘Clinic’ and ‘Flow’ to make it ‘Clow’. Or as my fellow teammate Dominick put it, “we put the flow back to the clinic.”

Technology + APIs #

Now here’s the interesting part. Clow relies heavily on machine learning, which we had zero knowledge of. Lucky for us, Microsoft Azure simplified the process of deploying machine learning to just a few drag-and-drops and clicks, and they actually provided us with $200 in virtual credits to test out their service. To analyze a patient’s face, we used Emotion API developed as part of Microsoft’s Project Oxford. For the app itself, we used Ionic Framework, partly due to our proficiency in web development and experience in the framework itself, and also partly due to its ease of deployment. Since we also have a companion app for patients to track their status and wait time, we used Firebase to share data between both apps in real-time.

The biggest challenge we had was actually training the model. We had to find real pictures of real patients experiencing illnesses and run them through the emotion analysis algorithm in order to get back the “scores”. Those “scores” were subsequently inserted into a spreadsheet, along with the name of the actual illness, so that we could export it as a csv file. These information were then uploaded to Azure so that we can train Dr. Clow to recognize these patterns and associate them with illnesses. As it was hard to find such images, our initial model for training Dr. Clow was relatively small, but enough to demonstrate its accuracy for a short demo.

Emotion + Big Data #

I’m new to big data and I do not have a lot of experience in the field. However, reflecting my 48 hours spent at Stanford, there was one particular moment that struck out. I had just finished building Clow, and had only slept for 2 hours in the past 48 hours when a mentor strongly disagreed with what we had built. Citing my lack of experience in the medical field, the lack of research in the correlation between emotion and sickness and the shortcoming of emotional recognition technology, she maintained that this idea was inviable. Granted, I first met her at another hackathon, and I feel that she was doing this only to help me. Well, in a way she did, as we altered our pitch from ‘matching the patient with a doctor who’s more skillful in the area of the sickness’ to ‘focused more on visually identifiable illnesses such as rashes and mental illnesses’.

We switched our focus to big data as Dr. Clow has the potential to be really accurate after training it with thousands or millions of data. Emotion analysis improves significantly over time as engineers develop new ways to recognize facial expressions, and it could be used with pattern recognition to detect visually identifiable illnesses or scars too. To quote a gentleman whom listened to our demo, we could store the images and reanalyze them when newer, more accurate facial recognition technology has developed, making the results more consistent and accurate.
treehacks-video.jpg My cameo appearance in the TreeHacks video!

Tidbits #

Concerned that the original venue might not be enough to accommodate all hackers, the organizers set up another hacking space at the Packard Electrical Engineering Building. With every room in the original venue occupied, my team relocated to the building in the middle of the night, but apparently it was only accessible by select Stanford students on its 24-hour list. Coincidentally, someone left the building (yes, at 3am), so we were able to sneak in and hack with little to no distraction. However, one of us had to stay in the building every time we wanted to leave, in order to open the doors from the inside. Sure, it’s a little inconvenient, and we might have missed some fun, but we were able to concentrate fully, which in turn allowed us to finish our hack with 12 hours remaining on the clock. We spent some of the time polishing Clow and made it attractive, and I slept for 7 hours (the most I’ve gotten at a hackathon so far), went for a morning walk around the campus the following morning, and pitched to the judges with a fresh, clear mind.

Oh yeah, I won a onesie at a raffle too. Coincidentally, it happened just moments after I told my teammates about how I never had the luck to win a raffle. I had never worn a onesie in my life, and thanks to Leon’s help, I was able to put one on before I turn 21.

Final Thoughts #

As ambitious as Clow might seem, this technology might take years before it can finally take off. Privacy remains an issue, and the correlation between emotion and sickness has to be studied thoroughly, so appropriate technologies can be developed. Nonetheless, it was fun playing with this idea, and it has definitely kick off my interest in healthcare. Hope to be there at TreeHacks 2017!

Code #

Our codes are publicly available on my GitHub account. We have separated our code base into two separate repositories, one for the clinic-facing app, and another for the patient’s companion app, and they can be found here:

Special Kudos #

 
18
Kudos
 
18
Kudos

Now read this

Hiatus

It’s been more than two years since my last (and also first) post here on my blog. Plenty of things have changed tremendously over the past two years: I’m now based in Silicon Valley, pursuing a computer science degree at one of the most... Continue →