SUO: Testing conformance to an ontology
Jim Schoening wrote:
>> All,
>>
>> Here's a question from a potential SUO user:
>>
>> How would one do conformance testing? If the SUO WG
>> produced an IEEE standard, and a program mandated it for new
>> systems/applications, how could one measure whether a new
>> system conformed to the standard?
>>
>> The question comes from a senior technical leader on
>> the Future Combat System (FCS), which has plenty of content
>> on the Web.
>>
>> Jim Schoening
>
>
When you test conformance of software to specs or to a standard, a
common way to do it is a giant test suite. There are test suites for
Java conformance, etc. Testing that a document conforms to standards of
gender-neutral prose (or whatever) involves looking for instances of
what you are trying to rule out. What does it mean for an application
to conform to an ontology? Perhaps the testing depends on what you use
the ontology for.
Only use it as a source of standardized terms
Use it as knowledge representation background, and expect to do
reasoning with it
Use it as source of constraints, "type checking"
Could we turn the question back? This is too open for a concrete answer
as is. We need some help to answer this question in a satisfying way.
What sort of conformance checking have they done other than by test
suites? I assume this exercise is more than checking off a box, that
they want to determine that the ontology was correctly used and that
they have not introduced errors of interpretation. If this checking
finds non-conformant spots, does that indicate an error or that the
ontology was incomplete? Sounds like a case-by-case decision.
The conformance testing will have to be very pragmatic and will miss
many theoretical niceties. Do they really want to determine if the
program embodies only concepts from the ontology or would they be OK on
determining something much more syntactic? It would help if they could
provide an example or two of what they plan to do with an ontology, and
what they are concerned about ensuring.
Allan Terry