This is a proof-of-concept application that transcribes an audio stream, detects the transcript's medical keywords, and automatically fetches supplementary content (definitions, pictures, drug data) when the user taps/clicks on a keyword.
It is meant to enhance the e-Shadowing experience by streamlining the process of finding good reference material without diverting one's attention away from the physician. When e-shadowing, I sometimes find myself searching Google for anatomy cartoons to refresh my memory, which is relatively cumbersome when compared to this tool.
This project serves more as an archive for an idea that I envisioned, rather than a sincere effort at producing a commercially viable product. It does exactly what I intended it to do, and I still strongly believe the concept has merit. I love medical education and seek to further our understanding of how technology may assist students and professors alike.
The application is packaged as a Flutter front-end and a helm chart backend for kubernetes. The application should scale horizontally to accomodate hundreds (or even thousands) of concurrent users, without any further optimization.
- Automatic recognition of keywords
- Diagnoses
- Anatomy
- Medications (both generic and brand name)
- Context-sensitive image search results
- Images are curated for relevance
- Anatomical cartoons & radiographs for anatomy
- Histology images for diseases
- Automatic retrieval of drug data
- Scrapes DrugBank with a headless web browser
- Interactive 3D visualization of chemical structure
- Accurate transcription of medical language
- Made possible with Amazon Transcribe Medical
- English only
- Simple and versatile architecture
- Amazon Transcribe Medical
- Amazon Comprehend Medical
- Bing Web Search
- Bing Image Search
- ChatGPT (OpenAI)
| Platform | Supported |
|---|---|
| Windows 10/11 | ✔️ |
| MacOS | ✔️ |
| Linux | ✔️ |
| iOS | ✔️ |
| Android | ✔️ |
| Chrome | ✔️ |
Running this tool can be financially expensive. If you leave it running for an hour, don't be surprised if your AWS bill is in the hundreds of dollars. For this reason, I'm apprehensive to believe such a tool has any commercial viability at all.
- Neurosurgery Video Dataset: brain & spine surgery videos curated with the intent of enhancing neurosurgical e-Shadowing. This dataset was to enrich e-Shadowing Transcriber by giving students the ability to click on the pixels of a live video stream to identify anatomical structures underneath the cursor.
- thavlik portfolio: my showcase of medical software projects, of which this repository is a part
All code in this repository is released under MIT / Apache 2.0 dual license, which is extremely permissive. Please open an issue if somehow these terms are insufficient.