Vyhodnocení agentů hlubokého Q-učení pomocí testu algoritmického IQ

Deep Q-learning connects two branches of artificial intelligence: feed-forward neural networks and reinforcement learning. Reinforcement learning is a method in which an agent that can be artificial intelligence learns by solving a given problem. The problem is an environment that communicates with the agent and gives him information about the state of the environment. The agent reacts to this information by taking action, for which the environment can reward him. Agent’s goal is to learn a behavior that allows him to collect the biggest sum of cumulated rewards. Deep Q-learning is a superstructure of a classical method for solving reinforcement learning problems called Q-learning. This method uses the Q-function to learn its behavior. Deep Q-learning approximates the Q-function by a neural network. Thanks to that, the agent can work with environments that have large state spaces. In this thesis, the agent was implemented into an algorithmic IQ (AIQ) test, which builds upon the concept of reinforcement learning and uses the concept of Universal intelligence. The very simplified thought of the AIQ test is to test an agent in the most environments possible, measure its accumulated reward, and, by taking into account the difficulty of that environment, count the final AIQ score. Two agents were implemented as a result of later research on Deep Q-learning. The first agent comes out of the first research paper about Deep Q-learning and consists of a mechanism for gathering the agent’s experiences from interactions with the environment within the test, called Replay memory. He also uses a single neural network that learns the agent’s behavior. The second agent comes from a follow-up research paper about DQL, in which the second neural network is added to the agent’s architecture. This second neural network helps to create learning data for the main neural network, which helps to stabilize the learning of the main neural network that still handles its behavior. AIQ test and implemented agents use Python as a programming language with popular libraries such as NumPy and PyTorch. The test was extended by a search algorithm based on evolutionary algorithms because of the large hyperparameter spaces of both agents, and evolutionary algorithms help to search this space heuristically and more efficiently. With these evolutionary algorithms, agent configurations were found, and the configuration of the first agent didn’t get better results than the configuration from the research paper, but for the second agent, a configuration was found that was statistically significantly better than the configuration from the research paper. The results of the DQL agents were compared with the results of the basic Q-learning agent because it was interesting to test the significance of the approximation of Q-function by the neural network. The results of this experiment don’t prove that DQL is any better than Q-learning for the current AIQ setting. There weren’t any significant differences between the two DQL architectures themselves either. Agents got the best ranks by final scores when tested against the best configurations of all other agents in the AIQ test. All results were gathered using the algorithmic IQ test, which was set up for each agent in the same way. The results of the agents were tested using the Student’s t-test.