Background: Prof Paul Cohen visited AIAI on 23/24th October 2000. Since Paul is very well known for his work on evaluation and empirical methods in AI, we thought it would be a good opportunity to discuss possible ideas for evaluation of the Coalition TIE. 1. Two Sorts of Evaluation in CoAX In CoAX, there are two sort of evaluation we should be doing. One is to view CoAX as an application of the underlying agent technology (such as the Grid) and use the data generated by CoAX as a way of testing the underlying technology in a real application. We regard this sort of evalution as very valuable and something we can offer to provide to the CoABS program. The other evaluation we should do is to evaluate CoAX and the ideas underlying CoAX as entities in their own right. We spent most of our time discussing the latter. 2. The CoAX Hypothesis In CoAX, the central hypothesis we are trying to test is: The agent-based computing paradigm is a very good fit for the kind of computational support needed for coalition operations. Additionally, we want to show that the CoAX approach gives us agility, inter-operability and robustness. 3. Assumptions We will assume that coalition members will bring software support tools with them and that those tools will need to be connected in some way in order to send information to each other. We can further assume that, although integration may be a high cost at the moment, this can be ignored - if it turns out that the CoAX approach is the correct one, then the amount of work we do at the moment to integrate systems will be much greater than the amount of work that will be needed if the CoAX method becomes "the norm". 4. Three Evaluation Ideas for CoAX A. Paul showed us a simple model of connecting agents, based on either point-to-point connections or using bridging agents. Given the costs of connecting point-to-point or to a bridging agent, the model can quickly tell you which approach is optimal, and how many bridge agents should be used. B. For CoAX we considered a model of agents which are bounded by functional domains, with each functional domain being characterised by a set of axioms. These functional domains are groupings of agents that perform a task, such that the amount of information they have to explain to each other can be minimised. The optimisation problem is, given a set of tasks T and a set of agents A (where size_of(A) > size_of(T)), to assign agents to domains so that the resulting system achieves all the tasks with the minimum amount of information exchange. C. We also discussed the idea of a game-theoretic formulation of the agent system we are trying to build, based on either an agent model of connection or point-to-point. The idea here is to show that the agent model is like an auction, where everyone can see everyone - in game theoretic terms, this is regarded as a fair system. Point-to-point connections where only certain other agents are "visible" can be shown to breed distrust. 5. Brief Discussion It is not clear to me (John L) that any of these ideas are the right way to evaluate the CoAX hypothesis, though they all contain elements that we may be able to use. If we accept our assumption that the systems must be connected somehow, then 4A might be able to prove that an agent arrangement is best, given which systems need to talk to which. If we have functional domains in which "teams" of agents can cooperate to achieve a task, 4B will be useful - but I'm not sure that CoAX can be characterised in this way (yet). Given that our agent model has bounded domains in which an agent may be restricted, I'm not sure we can get much leverage out of 4C. Perhaps, given our needs for security and control of (possibily rogue) agents, the CoAX arrangement of agents is the fairest we can get? Our central concerns are for agility, inter-operability and robustness. We need to define what each of these mean - how do you distinguish an agile approach from a non-agile approach? - and try to design experiments (which could be pen-and-paper, such as the ones above, or actual runs of the CoAX system) to show these apply. Please send me your comments on these notes. Thanks! John Levine (J.Levine@ed.ac.uk) 6th November 2000