A guide to deep learning in healthcare

Here we present deep-learning techniques for healthcare, centering our discussion on deep learning in computer vision, natural language processing, reinforcement learning, and generalized methods. We describe how these computational techniques can impact a few key areas of medicine and explore how to build end-to-end systems. Our discussion of computer vision focuses largely on medical imaging, and we describe the application of natural language processing to domains such as electronic health record data. Similarly, reinforcement learning is discussed in the context of robotic-assisted surgery, and generalized deep-learning methods for genomics are reviewed.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

cancel any time

Subscribe to this journal

Receive 12 print issues and online access

206,07 € per year

only 17,17 € per issue

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Developing robust benchmarks for driving forward AI innovation in healthcare

Article 15 November 2022

AI in health and medicine

Article 20 January 2022

Shortcut learning in deep neural networks

Article 10 November 2020

References

  1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature521, 436–444 (2015).
  2. Russakovsky, O.et al. Imagenet large scale visual recognition challenge. Int. J. Compute. Vis.115, 211–252 (2015).
  3. Hirschberg, J. & Manning, C. D. Advances in natural language processing. Science349, 261–266 2015). ArticleCASGoogle Scholar
  4. Geoffrey Hinton, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag.29, 82–97 (2012).
  5. Stanford Health. Harnessing the power of data in health. Stanford Medicine 2017 Health Trends Report (2017).
  6. Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nature529, 484–489 (2016). ArticleCASGoogle Scholar
  7. Yohannes Kassahun, et al. Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. Int. J. Comput. Assist. Radio. Surg.11, 553–568 (2016). ArticleGoogle Scholar
  8. Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal.42, 60–88 (2017). ArticleGoogle Scholar
  9. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature542, 115–118 (2017).
  10. Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol.29, 1836–1842 (2018).
  11. Cheng, J.-Z. et al. Computer aided diagnosis with deep learning architecture: applications to breast lesions in us images and pulmonary nodules in CT scans. Sci. Rep.6, 24454 (2016).
  12. Cicero, M. et al. Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs. Invest. Radiol.52, 281–287 (2017). ArticleGoogle Scholar
  13. Kooi, T. et al. Large scale deep learning for computer aided detection of mammographic lesions. Med. Image Anal.35, 303–312 (2017). ArticleGoogle Scholar
  14. Barreira, C. M. et al. Abstract WP61: Automated large artery occlusion detection in st roke imaging-paladin study. Stroke49, AWP61 (2018).
  15. Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA316, 2402–2410 (2016). ArticleGoogle Scholar
  16. Poplin, R. et al. Webster. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng.2, 158–164 (2018).
  17. Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med.24, 1342 (2018). ArticleGoogle Scholar
  18. Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Mitosis detection in breast cancer histology images with deep neural networks. In International Conference on Medical Image Computing and Computer-assisted Intervention 411–418 (Springer, 2013).
  19. Liu, Y. et al. Detecting cancer metastases on gigapixel pathology images. Preprint at https://arxiv.org/abs/1703.02442 (2017).
  20. Charoentong, P. et al. Pan-cancer immunogenomic analyses reveal genotype–immunophenotype relationships and predictors of response to checkpoint blockade. Cell Rep.18, 248–262 (2017).
  21. Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med.3, 108ra113 (2011). ArticleGoogle Scholar
  22. Yosinski, J., Clune, J., Bengio, Y. and Lipson, L. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 3320–3328 (2014).
  23. Jamaludin, A., Kadir, T. and Zisserman, A. Spinenet: automatically pinpointing classification evidence in spinal mris. In International Conference on Medical Image Computing and Computer-Assisted Intervention 166–175 (Springer, 2016).
  24. Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell172, 1122–1131 (2018). ArticleCASGoogle Scholar
  25. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (2015).
  26. Goodfellow, I. et al. Generative adversarial nets. In Advances in Neural Information Processing Systems 2672–2680 (2014).
  27. Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 3104–3112 (2014).
  28. Wu, Y. et al. Google’s neural machine translation system: bridging the gap between human and machine translation. Preprint at https://arxiv.org/abs/1609.08144 (2016).
  29. Kannan, A. et al. Smart reply: automated response suggestion for email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, 2016).
  30. Vinyals, O., Toshev, A., Bengio, S. & Erhan, D. Show and tell: a neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3156–3164 (2015).
  31. The Office of the National Coordinator for Health Information Technology. Quick stats: health IT dashboard. https://dashboard.healthit.gov/quickstats/quickstats.php (2017).
  32. Shickel, B., Tighe, P. J., Bihorac, A. & Rashidi, P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J. Biomed. Health Inform.22, 1589–1604 (2017).
  33. Rajkomar, A. et al. Scalable and accurate deep learning with electronic health records. NPJ Digit. Med.1, 18 (2018). ArticleGoogle Scholar
  34. Miotto, R. et al. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci. Rep.6, 26094 (2016).
  35. Liu, V., Kipnis, P., Gould, M. K. & Escobar, G. J. Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables. Med. Care48, 739–744 (2010). ArticleGoogle Scholar
  36. Choi, E. et al. Doctor AI: predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare 301–318 (2016).
  37. Che, Z. et al. Recurrent neural networks for multivariate time series with missing values. Rep.8, 1–12 (2018). Google Scholar
  38. Suresh, H. et al. Clinical intervention prediction and understanding with deep neural networks. PMLR68, 322–377 (2017).
  39. Johnson, A. E. W.et al. Mimic-iii, a freely accessible critical care database. Sci. Data3, 160035 (2016). ArticleCASGoogle Scholar
  40. Mao, Q.et al. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and icu. BMJ Open8, e017833 (2018). ArticleGoogle Scholar
  41. Abbeel, P. & Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning 1 (ACM, 2004).
  42. Ratliff, N. D., Silver, D. & Bagnell, J. A. Learning to search: functional gradient techniques for imitation learning. Autonomous Robots27, 25–53 (2009). ArticleGoogle Scholar
  43. Schulman, J. et al. A case study of trajectory transfer through non-rigid registration for a simplified suturing scenario. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 4111–4117 (IEEE, 2013).
  44. Mayer, H.et al. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Adv. Robot22, 1521–1537 (2008). ArticleGoogle Scholar
  45. Shvets, A., Rakhlin, A., Kalinin, A. A. and Iglovikov, V. Automatic instrument segmentation in robot-assisted surgery using deep learning. Preprint at https://arxiv.org/abs/1803.01207 (2018).
  46. Jin, A. et al. Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. Preprint at https://arxiv.org/abs/1802.08774 (2018).
  47. Loh, P.-R. et al. Efficient bayesian mixed-model analysis increases association power in large cohorts. Nat. Genet.47, 284 (2015). ArticleCASGoogle Scholar
  48. Abadi, M. et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. http://download.tensorflow.org/paper/whitepaper2015.pdf (2015).
  49. Tran, D. and Blei, D. M. Implicit causal models for genome-wide association studies. In International Conference on Learning Representations (2018).
  50. Lee, S.-I. et al. Learning a prior on regulatory potential from eqtl data. PLoS Genet.5, e1000358 (2009). ArticleGoogle Scholar
  51. Kircher, M. et al. A general framework for estimating the relative pathogenicity of human genetic variants. Nat. Genet.46, 310–315 (2014). ArticleCASGoogle Scholar
  52. Quang, D., Chen, Y. & Xie, X. Dann: a deep learning approach for annotating the pathogenicity of genetic variants. Bioinformatics31, 761–3 (2015). ArticleCASGoogle Scholar
  53. Dudley, J. T. et al. Personalized medicine: from genotypes, molecular phenotypes and the quantified self, towards improved medicine. In Pacific Symposium on Biocomputing 342–346 (2014).
  54. Leung, M. K. K., Delong, A., Alipanahi, B. & Frey, B. J. Machine learning in genomic medicine: a review of computational problems and data sets. Proc. IEEE104, 176–197 (2016).
  55. Xiong, H. Y. et al. The human splicing code reveals new insights into the genetic determinants of disease. Science347, 1254806 (2015). ArticleGoogle Scholar
  56. Alipanahi, B. et al. Predicting the sequence specificities of dna-and rna-binding proteins by deep learning. Nature Biotechnol.33, 831–838 (2015). ArticleCASGoogle Scholar
  57. Snyder, T. M., Khush, K. K., Valantine, H. A. & Quake, S. R. Universal noninvasive detection of solid organ transplant rejection. Proc. Natl Acad. Sci. USA108, 6229–6234 (2011). ArticleCASGoogle Scholar
  58. Abril, M. K. et al. Diagnosis of capnocytophaga canimorsus sepsis by whole-genome next-generation sequencing. In Open Forum Infectious Diseases Vol. 3, ofw144 (Oxford University Press, 2016).
  59. Forshew, T. et al. Noninvasive identification and monitoring of cancer mutations by targeted deep sequencing of plasma dna. Sci. Transl. Med.4, 136ra68–136ra68 (2012).
  60. Fan, H. C. et al. Non-invasive prenatal measurement of the fetal genome. Nature487, 320–324 (2012). ArticleCASGoogle Scholar
  61. Ching, T. et al. Opportunities and obstacles for deep learning in biology and medicine. Preprint at https://doi.org/10.1101/142760 (2017).
  62. Angermueller, C., Lee, H. J., Reik, W. & Stegle, O. Deepcpg: accurate prediction of single-cell dna methylation states using deep learning. Genome Biol.18, 67 (2017). ArticleGoogle Scholar
  63. Chen, Y. et al. Gene expression inference with deep learning. Bioinformatics32, 1832–1839 (2016). ArticleCASGoogle Scholar
  64. Koh, P. W., Pierson, E. & Kundaje, A. Denoising genome-wide histone chip-seq with convolutional neural networks. Bioinformatics33, i225–i233 (2017). ArticleCASGoogle Scholar

Acknowledgements

The authors would like to thank D. Wang, E. Dorfman, and A. Rajkomar for the visual design of the figures in this paper and P. Nejad for insightful conversation and ideas.

Author information

  1. These authors contributed equally: Andre Esteva, Alexandre Robicquet.

Authors and Affiliations

  1. Stanford University, Stanford, CA, USA Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov & Sebastian Thrun
  2. Google Research, San Jose, CA, USA Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado & Jeff Dean
  1. Andre Esteva