Abstract Multi-Agent Systems (MAS) are used for a wide range of applications. Goals and plans are the key premise to achieve MAS targets. Correct and proper execution and coverage of plans and achievement of goals ensures confidence in MAS. Proper identification of all possible faults in MAS working plays its role towards gaining such confidence. In this paper, we devise a model based approach which ensures goals and plans coverage. A Fault model has been defined covering faults in MAS related to goal and plan execution and interactions. We have created a test model using Prometheus design artifacts, i.e., Goal overview diagram, Scenario overview, Agent and Capability overview diagrams. New coverage criteria have been defined for fault identification. Test Paths have been identified from test model. Test cases have been generated from test paths. Our technique is then evaluated on actual implementation of MAS in JACK Intelligent Agents is a framework in Java for multi-agent system development (JACK) by executing more than 100 different test cases. Code has been instrumented for coverage analysis and faults have been injected in MAS. This approach successfully finds the injected faults by applying test cases for coverage criteria paths on MAS execution. ‘Goal plan coverage’ criterion has been more effective with respect to fault detection while scenario, capability and agent coverage criteria have relatively less scope in fault identification.

References

[1] Duff S., Thangarajah J., and Harland J., “Maintenance Goals in Intelligent Agents” Computational Intelligence, vol. 30, no. 1, pp. 71-114, 2014.

[2] Dam K., Evaluating and Comparing Agent- Oriented Software Engineering Methodologies, Ph.D. Thesis, RMIT University, 2003.

[3] Elammari M. and Issa Z., “Using Model Driven Architecture to Develop Multi-Agent Systems,” The International Arab Journal of Information Technology, vol. 10, no. 4, pp. 349-355, 2013.

[4] IEEE. Standard for Software Test Documentation. IEEE STD 829, 1998. URL http://standards.ieee.org/findstds/standard/829- 1998.html, Last Visited, 2016.

[5] Jack intelligent agents, http://aosgrp.com/products/jack/, Last Visited, 2015.

[6] Padgham L., Zhang Z., Thangarajah J., and Miller T., “Model-based Test Oracle Generation for Automated Unit Testing of Agent Systems,” IEEE Transactions on Software Engineering, vol. 39, no. 9, pp. 1230-1244, 2013.

[7] Padgham L. and Winikoff M., Developing Intelligent Agent Systems: A Practical Guide, Wiley Series in Agent Technology. John Wiley and Sons, 2004.

[8] Padgham L. and Winikoff M., “Prometheus: A Methodology for Developing Intelligent Agents,” in Proceedings of International Workshop on Agent-Oriented Software Engineering, Bologna, pp. 174-185, 2003.

[9] Riemsdijk M. and Yorke-Smith N., “Towards Reasoning with Partial Goal Satisfaction in Intelligent Agents,” in Proceedings of International Workshop on Programming Multi- Agent Systems, Toronto, pp. 41-59, 2010.

[10] Shaw P. and Bordini R., “An Alternative Approach for Reasoning about the Goal-Plan Tree Problem,” in Proceedings of Languages, Methodologies, and Development Tools for Multi-Agent Systems, 3rd International Conference, Lyon, pp. 115-135, 2010. 0 2 4 6 8 10 12 14 All goals Coverage Scenario Coverage Agent and capability Coverage Plan Coverage Goal Plan Coverage Loop Coverage Detected Faults by Coverage Criteriawith Minimum Test CasesTest casesNo of Faults Detected 64 The International Arab Journal of Information Technology, Vol. 16, No. 1, January 2019

[11] Thangarajah J., Harland J., Morley D., and Yorke-Smith N., “Towards Quantifying the Completeness of BDI Goals,” in Proceedings of The International Conference on Autonomous Agents and Multi-Agent Systems, Paris, pp. 1369- 1370, 2014.

[12] Thangarajah J., Harland J., Morley D., and Yorke-Smith N., “Quantifying the Completeness of Goals in BDI Agent Systems,” in Proceedings of 21st European Conference on Artificial Intelligence, Prague, pp. 879-884, 2014.

[13] Thangarajah J., Sardina S., and Padgham L., “Measuring Plan Coverage and Overlap for Agent Reasoning,” in Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, Valencia, pp. 1049- 1056, 2012.

[14] Thangarajah J. and Padgham L., “Computationally Effective Reasoning about Goal Interactions,” Journal of Automated Reasoning, vol. 47, no. 1, pp. 17-56, 2011.

[15] Thangarajah J., Jayatilleke G., and Padgham L., “Scenarios for System Requirements Traceability and Testing,” in Proceedings of The 10th International Conference on Autonomous Agents and Multiagent Systems, Taipei, pp. 285-292, 2011.

[16] Thangarajah J., Harland J., and Yorke-Smith N., “A Soft COP Model for Goal Deliberation in a BDI Agent,” in Proceedings of the 6th International Workshop on Constraint Modelling and Reformation, Rhode Island, pp. 61-75, 2007.

[17] Thangarajah J., Padgham L., and Winikoff M., “Prometheus Design Tool,” in Proceedings of the 4th International Conference on Autonomous Agents and Multi Agent Systems, Utrecht, pp. 127-128, 2005.

[18] Thangarajah J., Padgham L., and Winikoff M., “Detecting and Avoiding Interference between Goals in Intelligent Agents,” in Proceedings of the 18th International Joint Conference on Artificial Intelligence, Acapulco, pp. 721-726, 2003.

[19] Tian J., “Quality Assurance Alternatives and Techniques: A Defect Based Survey and Analysis,” Software Quality Professional, vol. 3, no. 3, pp. 6-18, 2001.

[20] Winikoff M., in Multi-Agent Programming, Springer, 2005.

[21] Wooldridge M., an Introduction to Multi-Agent Systems, John Wiley and Sons, 2002.

[22] Zhang Z., Thangarajah J., and Padgham L., “Automated Testing for Intelligent Agent Systems,” in Proceedings of the 10th International Conference on Agent-Oriented Software Engineering, Budapest, pp. 66-79, 2011.

[23] Zhou Y., Torre L., and Zhang Y., “Partial Goal Satisfaction and Goal Change: Weak and Strong Partial Implication, Logical Properties, Complexity,” in Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, Estoril, pp. 413- 420, 2008. Towards Automated Testing of Multi-Agent Systems Using Prometheus Design Models 65 Shafiq Ur Rehman is PhD (CS) candidate at Capital University of Science and Technology, Islambad. He is member of Center for Software Dependability research group. His research focuses on software quality assurance and testing, spcifically model based testing of multi-agent systems. In this research area he has published journals and conferences papers as well. Besides his research activities he is working as a software test engineer as well. Aamer Nadeem is an Associate Professor in the Department of Computer Science at Capital University of Science and Technology, Islamabad. He is also Head of the Center for Software Dependability - a research group working in the areas of software reliability, software fault tolerance, formal methods and safety-critical systems. He received his MSc in computer science from QAU, MS in software engineering from NUST, and PhD from Mohammad Ali Jinnah University, Islamabad. He did part of his PhD research work at the Chinese University of Hong Kong (CUHK) under a research collaboration. He is a professional member of the Association for Computing Machinery (ACM). Muddassar Sindhu received his PhD from Royal Institute of Technology (KTH), Stockholm, Sweden. Currently, he is an Assistant Professor of Computer Science at Quaid-i-Azam University, Islamabad, Pakistan. His research interests include software testing, learning- based testing, formal methods and formalization of informal software requirements.