Improving Deep Reinforcement Learning with Transitional Variational Autoencoders: A Healthcare Application

Matthew Baucum; Anahita Khojandi; Rama Vasudevan

doi:10.1109/JBHI.2020.3027443

Improving Deep Reinforcement Learning with Transitional Variational Autoencoders: A Healthcare Application

Matthew Baucum, Anahita Khojandi, Rama Vasudevan

Data Nanoanalytics

Research output: Contribution to journal › Article › peer-review

22 Scopus citations

Abstract

Reinforcement learning is a powerful tool for developing personalized treatment regimens from healthcare data. Yet training reinforcement learning agents through direct interactions with patients is often impractical for ethical reasons. One solution is to train reinforcement learning agents using an 'environment model,' which is learned from retrospective patient data, and can simulate realistic patient trajectories. In this study, we propose transitional variational autoencoders (tVAE), a generative neural network architecture that learns a direct mapping between distributions over clinical measurements at adjacent time points. Unlike other models, the tVAE requires few distributional assumptions, and benefits from identical training, and testing architectures. This model produces more realistic patient trajectories than state-of-the-art sequential decision-making models, and generative neural networks, and can be used to learn effective treatment policies.

Original language	English
Article number	9209034
Pages (from-to)	2273-2280
Number of pages	8
Journal	IEEE Journal of Biomedical and Health Informatics
Volume	25
Issue number	6
DOIs	https://doi.org/10.1109/JBHI.2020.3027443
State	Published - Jun 2021

Funding

Manuscript received April 21, 2020; revised August 4, 2020 and September 21, 2020; accepted September 22, 2020. Date of publication September 29, 2020; date of current version June 4, 2021. This work was supported in part by Science Alliance, The University of Tennessee and in part by the Laboratory Directed Research, and Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC, for the U.S. Department of Energy. (Corresponding author: Anahita Khojandi.) Matthew Baucum and Anahita Khojandi are with the Department of Industrial, and Systems Engineering, University of Tennessee, Knoxville, TN 37996 USA (e-mail: [email protected]; [email protected]).

Funders	Funder number
Laboratory Directed Research
Science Alliance
U.S. Department of Energy
Oak Ridge National Laboratory
University of Tennessee

Keywords

Reinforcement learning
generative adversarial networks
hidden Markov models
long short-term memory networks
variational autoencoders

Access to Document

10.1109/JBHI.2020.3027443

Cite this

@article{e92d4eaf106a408ebfae11953cc17933,

title = "Improving Deep Reinforcement Learning with Transitional Variational Autoencoders: A Healthcare Application",

abstract = "Reinforcement learning is a powerful tool for developing personalized treatment regimens from healthcare data. Yet training reinforcement learning agents through direct interactions with patients is often impractical for ethical reasons. One solution is to train reinforcement learning agents using an 'environment model,' which is learned from retrospective patient data, and can simulate realistic patient trajectories. In this study, we propose transitional variational autoencoders (tVAE), a generative neural network architecture that learns a direct mapping between distributions over clinical measurements at adjacent time points. Unlike other models, the tVAE requires few distributional assumptions, and benefits from identical training, and testing architectures. This model produces more realistic patient trajectories than state-of-the-art sequential decision-making models, and generative neural networks, and can be used to learn effective treatment policies.",

keywords = "Reinforcement learning, generative adversarial networks, hidden Markov models, long short-term memory networks, variational autoencoders",

author = "Matthew Baucum and Anahita Khojandi and Rama Vasudevan",

note = "Publisher Copyright: {\textcopyright} 2013 IEEE.",

year = "2021",

month = jun,

doi = "10.1109/JBHI.2020.3027443",

language = "English",

volume = "25",

pages = "2273--2280",

journal = "IEEE Journal of Biomedical and Health Informatics",

issn = "2168-2194",

publisher = "Institute of Electrical and Electronics Engineers",

number = "6",

}

TY - JOUR

T1 - Improving Deep Reinforcement Learning with Transitional Variational Autoencoders

T2 - A Healthcare Application

AU - Baucum, Matthew

AU - Khojandi, Anahita

AU - Vasudevan, Rama

PY - 2021/6

Y1 - 2021/6

N2 - Reinforcement learning is a powerful tool for developing personalized treatment regimens from healthcare data. Yet training reinforcement learning agents through direct interactions with patients is often impractical for ethical reasons. One solution is to train reinforcement learning agents using an 'environment model,' which is learned from retrospective patient data, and can simulate realistic patient trajectories. In this study, we propose transitional variational autoencoders (tVAE), a generative neural network architecture that learns a direct mapping between distributions over clinical measurements at adjacent time points. Unlike other models, the tVAE requires few distributional assumptions, and benefits from identical training, and testing architectures. This model produces more realistic patient trajectories than state-of-the-art sequential decision-making models, and generative neural networks, and can be used to learn effective treatment policies.

AB - Reinforcement learning is a powerful tool for developing personalized treatment regimens from healthcare data. Yet training reinforcement learning agents through direct interactions with patients is often impractical for ethical reasons. One solution is to train reinforcement learning agents using an 'environment model,' which is learned from retrospective patient data, and can simulate realistic patient trajectories. In this study, we propose transitional variational autoencoders (tVAE), a generative neural network architecture that learns a direct mapping between distributions over clinical measurements at adjacent time points. Unlike other models, the tVAE requires few distributional assumptions, and benefits from identical training, and testing architectures. This model produces more realistic patient trajectories than state-of-the-art sequential decision-making models, and generative neural networks, and can be used to learn effective treatment policies.

KW - Reinforcement learning

KW - generative adversarial networks

KW - hidden Markov models

KW - long short-term memory networks

KW - variational autoencoders

UR - http://www.scopus.com/inward/record.url?scp=85107472488&partnerID=8YFLogxK

U2 - 10.1109/JBHI.2020.3027443

DO - 10.1109/JBHI.2020.3027443

M3 - Article

C2 - 32991294

AN - SCOPUS:85107472488

SN - 2168-2194

VL - 25

SP - 2273

EP - 2280

JO - IEEE Journal of Biomedical and Health Informatics

JF - IEEE Journal of Biomedical and Health Informatics

IS - 6

M1 - 9209034

ER -

Improving Deep Reinforcement Learning with Transitional Variational Autoencoders: A Healthcare Application

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this