An Analysis of the Presumptions of Intelligent Behaviour of Computer Systems

Artificial General Intelligence seeks to create an artificial system capable of tackling many different and possibly unforeseen tasks or problems thus being comparable in its intelligence to that of a human. Such an endeavour, however, requiressuitable methods that can evaluate whether an artificial system is intelligent, andto what extent. This doctoral thesis searches for such evaluation methods thusanalyzing the presumptions of intelligent behaviour of computer systems.The thesis has three goals: 1) To choose a suitable intelligence test for artificialsystems; 2) To evaluate the chosen intelligence test empirically; and 3) To improvethe test based on its evaluation.Based on an extensive literature overview covering both philosophical andcognitive presumptions of intelligence as well as formal definitions and practical tests of intelligence grounded in Algorithmic Information Theory, this thesisconcludes that the Algorithmic Intelligence Quotient test, derived from UniversalIntelligence definition, is currently the most suitable candidate for a practical intelligence evaluation method of artificial systems. The test, however, has severalknown limitations. Universal Intelligence is dependent on the choice of the reference Turing machine. Among the environments used by the test, there are somethat have no discriminative power and as such do not aid the actual evaluation ofan agent’s intelligence. Some aspects of intelligence, concerning time and computational effectiveness, are not included in the overall score.Based on an empirical assessment of the Algorithmic Intelligence Quotienttest, comprised of experiments conducted using both the default settings and others in which the action and observation spaces were modified, and also a semanticanalysis of the environment programs use, the thesis confirmed and further specified the known limitations of the test. The results of the Algorithmic IntelligenceQuotient test are also dependent on its reference machine settings. Around 17 % ofthe programs within the environment programs sample feature no discriminativepower, returning random rewards to the tested agents. The empirical evaluationalso discovered other limitations of the test. Some relevant aspects of intelligencethat are missing in the final AIQ score are captured by its convergence process.At least 74 % of environment programs contain some form of pointless code thatis together with other problems concerning the environment programs causedby unoptimized random sampling. The test is highly computationally demandingpartially due to an ineffective implementation of its testing procedure. The default settings of the test may be too simple thus failing to sufficiently bring out thedifferences among tested agents.Based on a synthesis of the theoretical and practical findings concerning thelimitations of the Algorithmic Intelligence Quotient test, the thesis proposed andthoroughly discussed several possibilities that eliminate or reduce the observedlimitations. The thesis managed to further advance the Algorithmic IntelligenceQuotient test in the direction of a suitable general intelligence evaluation methodfor artificial systems since it implemented the chosen proposals in the followingways. Measures of convergence dynamics and agents’ reaction times can differentiate among agents with otherwise similar score. Multi-round methods for sufficiently precise estimates and sufficiently converged scores can reduce the test’sconsumption of computational resources. An extension of the test that saves theintermediate results reduces the time required to perform the test by 30 %. An extension of the test that implements multi-round convergence of the total accumulated rewards enables further time reduction by more than 40 % while maintaining the precision of the results estimates. Extensions to the sampling procedureof the test reduce the proportion of the pointless code more consistently and significantly reduce the proportion of non-discriminative environment programs. Anextension to the sampling procedure of the test enables the definition of a minimum length for environment programs in the sample that reduces the dependenceof the test results on the reference machine and increases the difficulty of the testsetting. The implemented extensions were empirically validated by a series of experiments and in relevant cases also by a thorough analysis of sampled programs.The evaluation proved the validity of the conducted changes.