Here are some use
cases.
Assumptions:
1)
There
may be a reasoner in the loop, or not.
2)
If there
is a reasoner, the interaction between reasoner and tester may be very
dynamic, or the test application may be constrained to following a TPS and
only use the reasoner at infrequent choice
points.
3)
The
logic and flow of the TPS may be simple to understand a priori, or it may be
very difficult to know what happens without an actual trace
log.
4)
TestResults is THE
data structure for recording and communicating (e.g. to a reasoner)
information gathered at run-time, including what the test application actually
did, what information was gathered, what UUT was actually tested, etc.
Use cases and
scenarios
1)
Debugging: Developers
perform offline analysis of recorded TestResults to verify that
TestDescription and the actual TPS are in agreement. This requires that
a fairly complete action trace be recorded including
setups/adjustments/etc.
2)
Hidden
variables: There can be non-test data gathered by a test station that that can
significantly alter the subsequent flow control of the TPS (e.g. user inputs,
tester state, UUT state). The reasoner would like to know this
information so that it can make optimal recommended actions. The
information would also be useful for offline
analysis.
3)
Hidden
actions: We have observed TPSs with non-test actions within a test procedure,
such as adjustments. The adjustment alters the state of the UUT, and
might even intentionally fix the fault. This action should be recorded
in TestResults. One could argue that you can deduce whether of not the
adjustment took place by analysis of the sequences in the TestDescription. But
real world TPSs can be difficult to decipher and the flow can be based on user
choice or a complex set of global variables (neither of which are currently in
test results). The safest bet is to simply record the
action.
4)
Reasoner-guided
testing: if the tests sequences are guided by a reasoner at a fine grain (e.g.
through a ranked list of recommended next tests/adjustment/R&R/etc), then
the test application needs to be very explicit when it tells the reasoner what
it actually did. You should not assume that there is a well defined
sequence in a testDescription that allows one to know what actions occurred
between tests.
Additional
points
- I don’t like the alternate idea
of having a sister document to TestResults for the remaining non-test
information. Partitioning things into smaller interlinked XML
documents can be a good thing if it is done wisely and systematically.
But we are already down the path of having monolithic
documents.
- I think that the core argument
that TestResults can only contain results of tests is overly strict (that’s
not a test, it’s a …). If we were to develop an information
architecture, we would find many relationships between tests and other
entities. Those relationships need to be expressed somewhere in the
XML, and the descendants of <testResult> should not be
restricted to relationships of the form “ ____ is a test”. TestResults
already contains many such relationships: {actual tests, performed by,
actual operator} , {actual repairs, performed prior to, actual tests}
{actual tests, performed on, actual UUT} etc. Why not also allow
{actual setup, performed prior to, actual test} and {actual adjustment,
performed within sequence of, actual tests} etc?
- Reliance on the TestDescription
to determine what actions took place instead of a recorded action trace is
risky. Reasoner-guided diagnostics and even legacy TPSs are capable of
traces that are not easy to express a priori.
- The core argument in bullet 2
that is preventing us from putting setups, adjustments etc in TestResults
should apply equally well to TestDescription. What will we do if
TestDescription can’t contain any non-test actions either? Does the
argument apply to everything in ATML?
- The requirements and use cases
for TestResults are what’s important. If the name of the root element
“TestResults” is preventing us from meeting the use cases or requirements,
then change the root element.
Darryl
From:
stds-ai-estate@ieee.org [mailto:stds-ai-estate@ieee.org] On Behalf Of Sheppard, John (JSHEPPAR)
Sent: Monday, September 12, 2005 8:43
AM
To:
stds-ai-estate@IEEE.ORG
Subject: RE: Proposed change to 1636.1
test results XML schema
Thank you
for the concise and cogent summary of the discussion. I think you captured the
issues very well. That said, I suspect we may need to defer this specific
issue until the meeting at AUTOTESTCON where we will be able to address the
issue in an online, face-to-face setting. My experience is that, when faced
with difficult issues such as this, these face-to-face meetings are
invaluable.
Also, I
would like to point out that some additional information was requested but
never provided. Perhaps the most important additional information was better
explanation of the underlying use case. Since I was not part of the original
discussion, I did not have the benefit of seeing this use case. Therefore,
while I tend to agree with Brit, I do not feel I am able to argue the case
adequately without understanding that use
case.
The second
bit of information that was requested (by me) was some kind of "pictorial"
representation of the schema with and without the added information. My
preference is an EXPRESS model, but I am willing to entertain other
representations. Note that I find the XML Spy representation to be very
difficult to follow since it does not provide a "bird's-eye) view of the
schema (especially since I do not have XML Spy to browse the schema and have
to rely on exported documentation).
Bottom line:
While I appreciate and agree with the desire to "git 'r dun," I
don't think we have the necessary information to make the required decision in
the coming week. Therefore, if we absolutely have to make a decision this
week, I have to go with direction in which I am leaning and oppose to the
addition.
-----
Dr. John W.
Sheppard
-- Fellow
and Assistant Research Professor
-- ARINC and
Johns
Hopkins University
--
jsheppar@arinc.com and jsheppa2@jhu.edu
-----Original
Message-----
From:
stds-ai-estate@ieee.org [mailto:stds-ai-estate@ieee.org] On Behalf Of Ralph, John (Mission Systems)
Sent: Monday, September 12, 2005 9:25
AM
To:
stds-ai-estate@IEEE.ORG
Subject: Proposed change to 1636.1 test
results XML schema
There
was a good bit of discussion and a variety of opinions expressed regarding
the proposal to add a "non-test action" element to the 1636.1 Test Results
schema. Some of the discussion revolved around the definition of terms.
While I agree that we should be clear on what we mean when we use certain
words, I don't believe we have the luxury of debating definitions for the
next 12 months; we need to take our "best shot" and move forward so that
Draft 2 may be submitted.
It was suggested by Brit Frank
that Test Results is not the proper place to capture non-result events or
actions that occur during the course of a test. Rather, he suggested that
Test Description might be a better home for this information. A
clarification on the purpose of Test Description has shown that this won't
work.
Another suggestion was the
creation of a new schema for the specific purpose of recording such
non-result information. The down-side of this approach is that it fragments
the serial flow of data by requiring references to and linking in external
files when it becomes necessary to reconstruct the entire sequence of events
that occured during the course of a test.
I believe we have two options
regarding the suggested modifications to Test Results:
1. Accept the proposed change,
and modify the schema/documentation accordingly, or
2.
Reject the proposal and require users with a desire/need to capture such
information to institute their own use-case specific extension to the Test
Results schema.
Obviously this can't be a
unilateral decision, but a decision must be made soon - preferrably within
the next week.
I solicit your feedback, and
specifically request that potential users of the schema weigh in with their
preference.
John Ralph