Health Care > EXAM > CS 234 assignment 2-ALL ANSWERS 100% CORRECT (All)
CS 234 Winter 2020: Assignment #2 Introduction In this assignment we will implement deep Q-learning, following DeepMind’s paper ([mnih2015human] and [mnih-atari-2013]) that learns to play At... ari games from raw pixels. The purpose is to demonstrate the effectiveness of deep neural networks as well as some of the techniques used in practice to stabilize training and achieve better performance. In the process, you’ll become familiar with TensorFlow. We will train our networks on the Pong-v0 environment from OpenAI gym, but the code can easily be applied to any other environment. In Pong, one player scores if the ball passes by the other player. An episode is over when one of the players reaches 21 points. Thus, the total return of an episode is between −21 (lost every point) and +21 (won every point). Our agent plays against a decent hard-coded AI player. Average human performance is −3 (reported in [mnih-atari-2013]). In this assignment, you will train an AI agent with super-human performance, reaching at least +10 (hopefully more!). 1 0 Test Environment (6 pts) Before running our code on Pong, it is crucial to test our code on a test environment. In this problem, you will reason about optimality in the provided test environment by hand; later, to sanity-check your code, you will verify that your implementation is able to achieve this optimality. You should be able to run your models on CPU in no more than a few minutes on the following environment: • 4 states: 0, 1, 2, 3 • 5 actions: 0, 1, 2, 3, 4. Action 0 ≤ i ≤ 3 goes to state i, while action 4 makes the agent stay in the same state. • Rewards: Going to state i from states 0, 1, and 3 gives a reward R(i), where R(0) = 0.1, R(1) = −0.2, R(2) = 0, R(3) = −0.1. If we start in state 2, then the rewards defind above are multiplied by −10. See Table 1 for the full transition and reward structure. • One episode lasts 5 time steps (for a total of 5 actions) and always starts in state 0 (no rewards at the initial state). State (s) Action (a) Next State (s′) Reward (R) 0 0 0 0.1 0 1 1 -0.2 0 2 2 0.0 0 3 3 -0.1 0 4 0 0.1 1 0 0 0.1 1 1 1 -0.2 1 2 2 0.0 1 3 3 -0.1 1 4 1 -0.2 2 0 0 -1.0 2 1 1 2.0 2 2 2 0.0 2 3 3 1.0 2 4 2 0.0 3 0 0 0.1 3 1 1 -0.2 3 2 2 0.0 3 3 3 -0.1 3 4 3 -0.1 Table 1: Transition table for the Test Environment An example of a trajectory (or episode) in the test environment is shown in Figure 5, and the trajectory can be represented in terms of st, at, Rt as: s0 = 0, a0 = 1, R0 = −0.2, s1 = 1, a1 = 2, R1 = 0, s2 = 2, a2 = 4, R2 = 0, s3 = 2, a3 = 3, R3 = (−0.1) • (−10) = 1, s4 = 3, a4 = 0, R4 = 0.1, s5 = 0. Figure 1: Example of a trajectory in the Test Environment 1. (written 6 pts) What is the maximum sum of rewards that can be achieved in a single trajectory in the test environment, assuming γ = 1? Show first that this value is attainable in a single trajectory, and then briefly argue why no other trajectory can achieve greater cumulative reward. Solution: The optimal reward of the Test environment is 4.1 To prove this, let’s prove an upper bound of 4.1 with 3 key observations • first, the maximum reward we can achieve is 2 when we do 2 → 1. • second, after having performed this optimal transition, we have to wait at least one step to execute it again. As we have 5 steps, we can execute 2 optimal moves. Executing less than 2 would yield a strictly smaller result. We need to go to 2 twice, which gives us 0 reward on 2 steps. Thus, we know that 4 steps gives us a max of 4. Then, the best reward we can achieve that is not an optimal move (starting from state 1) is 0.1, which yields an upper bound of 4.1. Considering the path 0 → 2 → 1 → 2 → 1 → 0 proves that we can achieve this upper bound. 1 Q-Learning (24 pts) Tabular setting If the state and action spaces are sufficiently small, we can simply maintain a table containing the value of Q(s, a) – an estimate of Q∗(s, a) – for every (s, a) pair. In this tabular setting, given an experience sample (s, a, r, s′), the update rule is Q(s, a) Q(s, a) + α r + γ max Q (s′, a′) Q (s, a) (1) a'∈A where α > 0 is the learning rate, γ ∈ [0, 1) the discount factor. Approximation setting Due to the scale of Atari environments, we cannot reasonably learn and store a Q value for each state-action tuple. We will instead represent our Q values as a function qˆ(s, a; w) where w are parameters of [Show More]
Last updated: 1 year ago
Preview 1 out of 13 pages
Statistics> EXAM > CS 234 assignment 2-ALL ANSWERS 100% CORRECT Study Guide (All)
CS 234 assignment 2-ALL ANSWERS 100% CORRECT Study Guide
By MARKALLAN , Uploaded: Aug 01, 2022
$5
*NURSING> EXAM > Relias Dysrhythmia Basic Test Answers 2023 Dysrhythmia – Basic A ( A+ GRADED 100% VERIFIED) (All)
Relias Dysrhythmia Basic Test Answers 2023 Dysrhythmia – Basic A ( A+ GRADED 100% VERIFIED)
By kian biden , Uploaded: Jan 29, 2024
$9.5
*NURSING> EXAM > ATI PN comprehensive predictor 2020/2021 180 Questions And Answers.(SCORE A GUARANTEED) (All)
Ati PN comprehensive predictor 2020/2021 180 Questions And Answers.(SCORE A GUARANTEED)
By Rixx Dennis , Uploaded: Nov 17, 2021
$46
*NURSING> EXAM > NURS 6512 Advanced Health Assessment and Diagnostic Reasoning Final Exam- Walden University (All)
NURS6512 - Advanced Health Assessment & Diagnostic Reasoning Advanced Health Assessment Final Exam Q & A what is the leading cause of mortality in teenagers? - motor vehicle accidents ho...
By quiz_bit , Uploaded: Apr 30, 2021
$13
Anatomy> EXAM > BSC 2346 A & P Module 5 Mid Term Exam; Latest Complete Solution Guide; Rasmussen College. (All)
BSC 2346 A & P Module 5 Mid Term Exam Mid Term- Module 5 Which choice best describes the function of the haversian (central) canals within bone? What statement is NOT true regarding synovial...
By Expert#1 , Uploaded: Feb 11, 2020
$11
*NURSING> EXAM > ATI Comprehensive Predictor (Study This One!) (All)
ATI Comprehensive Predictor Quizlet/ATI Comprehensive Predictor Quizlet/ATI Comprehensive Predictor Quizlet/ATI Comprehensive Predictor Quizlet
By Rixx Dennis , Uploaded: Mar 07, 2021
$15
*NURSING> EXAM > NR 501 Nursing Theory Week 8 Quiz (All)
NR501 Week 8: Nursing Theory Quiz Week 8: Nursing Theory Quiz Question 1 A statement of beliefs and values about human beings and their world is: An assumption A concept A hypothesis A philosophy Ques...
By QUIZ BIT , Uploaded: Jun 23, 2021
$10
*NURSING> EXAM > Nutrition Proctored-Questions and Answers (2020) (All)
Nursing 160 Nutrition Proctored-Questions and Answers/Nursing 160 Nutrition Proctored-Questions and Answers /Nursing 160 Nutrition Proctored-Questions and Answers
By Rixx Dennis , Uploaded: Jan 20, 2021
$12
*NURSING> EXAM > HESI Med Surg Exam 2023, Med-Surg HESI Exam (All)
HESI Med Surg Exam (CHECK THE LAST PAGE FOR MULTIPLE VERSIONS OF THE EXAM AND OTHER HESI EXAMS) A nurse is assisting with the care of a client who has a femur fracture and is in skeletal traction....
By Maxquizer , Uploaded: Jul 18, 2021
$9
*NURSING> EXAM > RN ATI Capstone Comprehensive Assessment A: The Actual Exam. Individual Performance Profile. (All)
RN ATI Capstone Comprehensive Assessment A: The Actual exam. Individual Performance Profile AREAS OF REVIEW STUDY Management of care Safety and infection control Health promotion and maintance Ps...
By QuizMaster , Uploaded: Aug 10, 2020
$9
Connected school, study & course
About the document
Uploaded On
Dec 03, 2022
Number of pages
13
Written in
This document has been written for:
Uploaded
Dec 03, 2022
Downloads
0
Views
56
Avoid resits and achieve higher grades with the best study guides, textbook notes, and class notes written by your fellow students
Your fellow students know the appropriate material to use to deliver high quality content. With this great service and assistance from fellow students, you can become well prepared and avoid having to resits exams.
Your fellow student knows the best materials to research on and use. This guarantee you the best grades in your examination. Your fellow students use high quality materials, textbooks and notes to ensure high quality
Get paid by selling your notes and study materials to other students. Earn alot of cash and help other students in study by providing them with appropriate and high quality study materials.
In Browsegrades, a student can earn by offering help to other student. Students can help other students with materials by upploading their notes and earn money.
We're available through e-mail, Twitter, Facebook, and live chat.
FAQ
Questions? Leave a message!
Copyright © Browsegrades · High quality services·