RE: Proposed change to 1636.1 test results XML schema
Ok, I've had some time to take all this in and hopefully I can get my
point across. I think we are trying to "band-aid" the problem. The
following is some verbiage take from the latest draft of P1636.1.
"In the ATML context, a test is a procedure for evaluating or
quantifying the operation of some device or system. The Test Results
schema provides a standard format for the transport or storage of both
quantitative (measured values) and qualitative (pass/fail determination)
test results. The schema design is such that ancillary information such
as environmental conditions and system/operator messages may also be
stored in an instance document. This information, while not specifically
"results", is intended to permit use of an instance document for a
variety of purposes, including statistical analysis and diagnostics.
Some examples of this ancillary information includes identifying
information for the UUT, the test station, and the test program; ambient
environmental conditions at the time of the test; test equipment
calibration data; test program input data and ancillary textual
comments. The schema establishes a hierarchical structure for results
data to permit the grouping of a series of related test results in a
single instance document."
This document centers around things called tests and the results of
them. I think everyone can agree on that (or then again maybe not). It
also contains things not related to tests, i.e., UUT information,
operator, MOS, calibration data, etc. These non-test things are
required in order to characterize the object (the test performed).
Again, I think everyone can agree that we need these things. So far, I
don't think I have precluded any use cases presented by Darryl Busch and
don't go against any of his assumptions. Now, here are my points.
1) I don't think a name change or a PAR revision is required.
2) Just because we are going down a path (see Additional point #1 in
Darryl's e-mail) doesn't mean it is the right path. ABBET went down a
lot of the wrongs paths, which ultimately lead to its demise.
3) Procedures do NOT belong in this dot standard. They belong
somewhere else.
4) I keep referring to an "architecture" (not like the one that Tim
sent out for reference) for ATML. This is what is needed and I will
elaborate on this below.
I agree with what Mike Seavey sent out earlier. I have included a
portion of it here for reference.
"1) ATML is not defining, nor is a Architecture, ATML is not defining
Services (as ABBET tried to do) ATML at the top level is a Framework.
This Framework definition is a present being worked by the TII
subcommittee under IEEE P1671. The latest Draft of the Framework
document (IEEE P1671) is on the TII web-site. Chris Gorringe and Myself
are the co-chairs of TII presently.
2)The ATML framework consists of the elements within the ATS, that have
been defined to date, where a standardized description would allow
interoperability of the information between elements. An Example would
be an instruments capability, the XYY companies DMM capabilities
description, if described in a consistent non proprietary mannor,
allowing the development of tools/utilities permitting things like
resource allocation (where the resource allocator is an
implimentation)and if two different vendors provided ATML complient
resource allocators, they could be interchanged ATML is defining what
information needs to be included in the component, NOT what to do with
the information. What to do with the information is an implimentation
(by either the ATS integrator and/or tool developers)"
The "architecture" is am referring to is a bottoms-up, top-down look at
all the information required, i.e., the software interfaces that were
deemed critical in the ATS Critical Interfaces Report. These interfaces
are: the Adapter Function and Parametric Data - the information and
formats used to define to the Application Development Environments the
capabilities of the test fixture, how the capabilities are accessed, and
the associated performance parameters; the Instrument Function and
Parametric Data - the information and formats used to define to the
Application Development Environments the load, sense, and drive
capabilities of the Instruments, how these capabilities are accessed,
and the associated performance parameters; Switch Function and
Parametric Data - the information and formats used to define to the
Application Development Environments the interconnect capabilities of
the Switch Matrix, how these capabilities are accessed, and the
associated performance parameters; the UUT Test Requirements - the
information and formats used to define to the Application Development
Environments the load, sense, and drive capabilities that must be
applied to the UUT to test it, including the minimum performance
required for a successful test; the Test Program Documentation - the
human-understandable representations of information about the TPS for
use by the TPS maintainer; the Run Time Services - the run time services
needed by Test Programs not handled by the services supplied by the
instrument driver API, the framework, the resource adapter interface and
the network interfaces (e.g., error reporting, data logging).
We need to see what information is common among all the component
standards, what information is common among certain component standards,
and what unique information is required for each component standard, and
have definitions of what each piece of information is. Then it should
be fairly evident where everything belongs and where it does not. I
think the ATML standards are very important and the XML data format is a
great way to store/retrieve the information we need. I just don't want
to see the same problems arise as with the ABBET standards.
Again, I don't think I have precluded any use cases presented by Darryl
Busch and don't go against any of his assumptions.
Now, here's where I deviate (other than my point 2 above; Darryl's
Additional points #1) from Darryl. I am not stating that TestResults
contain only results of tests. Again, non-test things are required in
order to characterize the object (the test performed). I also believe I
agree with Darryl's Additional points #2 (unless he is putting in
procedurally-oriented information). No where and at no time have I
stated that non-test information can't or shouldn't be put in the ATML
standards (Darryl's Additional points #4). I am just stating that
"procedures" don't belong in THIS dot standard and they need to go
somewhere, thus the need for the "architecture" pointed out above.
I hope in all this rambling that I have cleared some things up.
Thanks,
Brit
________________________________
From: stds-ai-estate@IEEE.ORG [mailto:stds-ai-estate@IEEE.ORG] On Behalf
Of Sheppard, John (JSHEPPAR)
Sent: Wednesday, September 14, 2005 12:27 PM
To: stds-ai-estate@IEEE.ORG
Subject: RE: Proposed change to 1636.1 test results XML schema
Tim,
At this point, I am expressing an opinion, so take it for what it is
worth.
Assuming we as a working group can agree that a) this satisfies the
concern and b) this is what we want to do, we should be able to create a
revised PAR at the Orlando meeting and get it approved by Steering at
the same time. This can then be handled by the Standards Board under
continuous processing with the P1232 PAR. In the worst case, should we
need to delay the PAR for December SB approval, or even March SB
approval (i.e., outside of continuous processing), then I believe we
could still go to ballot in March. The only time the actual PAR will
matter is when we submit the approved draft to the standards board.
Therefore, in my opinion, changing the PAR should not have any impact on
schedule.
John
-----
Dr. John W. Sheppard
-- Fellow and Assistant Research Professor
-- ARINC and Johns Hopkins University
-- jsheppar@arinc.com and jsheppa2@jhu.edu
-----Original Message-----
From: Wilmering, Timothy J
[mailto:timothy.j.wilmering@boeing.com]
Sent: Wednesday, September 14, 2005 11:31 AM
To: Sheppard, John (JSHEPPAR); stds-ai-estate@IEEE.ORG
Subject: RE: Proposed change to 1636.1 test results XML schema
John Ralph and I discussed this possibility a couple of weeks
ago during a flare-up of these discussions (I like this idea - the real
problem is that the name is too restrictive, as Darryl well points
out), but changing the PAR seemed like a show stopper at the time (there
are folks that really want this out there soon). The roadmap has us
producing the final draft by January and going to ballot in March. Can
we make this schedule if we modify the PAR?
More importantly, I think we need to know if modifying the name
and literal scope of the PAR (and underlying schema) will satisfy the
persons who are arguing that "non-test" events do not belong in the Test
Results schema?
Tim
_________________________________
Timothy J. Wilmering
'314.234.6781
-<mailto:timothy.j.wilmering@boeing.com>
________________________________
From: Sheppard, John (JSHEPPAR)
[mailto:jsheppar@ARINC.COM]
Sent: Tuesday, September 13, 2005 3:29 PM
To: stds-ai-estate@IEEE.ORG
Subject: RE: Proposed change to 1636.1 test results XML
schema
Of course, doing this requires changing the PAR. Oh
well.
John
-----
Dr. John W. Sheppard
-- Fellow and Assistant Research Professor
-- ARINC and Johns Hopkins University
-- jsheppar@arinc.com and jsheppa2@jhu.edu
-----Original Message-----
From: stds-ai-estate@ieee.org
[mailto:stds-ai-estate@ieee.org] On Behalf Of Sheppard, John (JSHEPPAR)
Sent: Tuesday, September 13, 2005 4:20 PM
To: stds-ai-estate@IEEE.ORG
Subject: RE: Proposed change to 1636.1 test
results XML schema
I had a long response prepared to send to this
group that probably would have gotten a flame war going, and then I
stopped to rethink. Rather than get people all rialed up, I would like
to see if we can reach some level of consensus.
One of the suggestions Darryl made was to rename
the top-level entity (and thereby the name of the schema I hope) to
something other than TestResults. My initial reaction was to want to
jump all over it as a band aid solution, but as I put together my
response, my opinion shifted.
As you all know, AI-ESTATE is organized into
several models. These models can be classified along three lines: common
static information (CEM), reasoning technique-specific information (FTM,
DIM, EDIM, BM), and dynamic reasoner-based information (DCM). In many
respects TestResults is just like the DCM. It is an attempt to capture
the "dynamic" information from the tester to provide to the reasoner (or
elsewhere) for additional processing. Therefore, might I suggest that we
not only change the name but the semantic intent of this schema to
reflect (for example) RunTimeResults as opposed to TestResults. That way
the schema now subsumes TestResults, legitimately allows for repair
actions, adjustments, calibrations, etc., and opens the way for
including other "non-test" related information collected during testing.
By the way, we might also find that it fits better both with AI-ESTATE
and with the goals of SIMICA.
John
-----
Dr. John W. Sheppard
-- Fellow and Assistant Research Professor
-- ARINC and Johns Hopkins University
-- jsheppar@arinc.com and jsheppa2@jhu.edu
-----Original Message-----
From: Busch, Darryl G (MN65)
[mailto:darryl.busch@honeywell.com]
Sent: Tuesday, September 13, 2005 1:56
PM
To: Sheppard, John (JSHEPPAR);
stds-ai-estate@IEEE.ORG
Subject: RE: Proposed change to 1636.1
test results XML schema
Here are some use cases.
Assumptions:
1) There may be a reasoner in the
loop, or not.
2) If there is a reasoner, the
interaction between reasoner and tester may be very dynamic, or the test
application may be constrained to following a TPS and only use the
reasoner at infrequent choice points.
3) The logic and flow of the TPS
may be simple to understand a priori, or it may be very difficult to
know what happens without an actual trace log.
4) TestResults is THE data
structure for recording and communicating (e.g. to a reasoner)
information gathered at run-time, including what the test application
actually did, what information was gathered, what UUT was actually
tested, etc.
Use cases and scenarios
1) Debugging: Developers perform
offline analysis of recorded TestResults to verify that TestDescription
and the actual TPS are in agreement. This requires that a fairly
complete action trace be recorded including setups/adjustments/etc.
2) Hidden variables: There can be
non-test data gathered by a test station that that can significantly
alter the subsequent flow control of the TPS (e.g. user inputs, tester
state, UUT state). The reasoner would like to know this information so
that it can make optimal recommended actions. The information would
also be useful for offline analysis.
3) Hidden actions: We have
observed TPSs with non-test actions within a test procedure, such as
adjustments. The adjustment alters the state of the UUT, and might even
intentionally fix the fault. This action should be recorded in
TestResults. One could argue that you can deduce whether of not the
adjustment took place by analysis of the sequences in the
TestDescription. But real world TPSs can be difficult to decipher and
the flow can be based on user choice or a complex set of global
variables (neither of which are currently in test results). The safest
bet is to simply record the action.
4) Reasoner-guided testing: if the
tests sequences are guided by a reasoner at a fine grain (e.g. through a
ranked list of recommended next tests/adjustment/R&R/etc), then the test
application needs to be very explicit when it tells the reasoner what it
actually did. You should not assume that there is a well defined
sequence in a testDescription that allows one to know what actions
occurred between tests.
Additional points
1. I don't like the alternate idea
of having a sister document to TestResults for the remaining non-test
information. Partitioning things into smaller interlinked XML documents
can be a good thing if it is done wisely and systematically. But we are
already down the path of having monolithic documents.
2. I think that the core argument
that TestResults can only contain results of tests is overly strict
(that's not a test, it's a ...). If we were to develop an information
architecture, we would find many relationships between tests and other
entities. Those relationships need to be expressed somewhere in the
XML, and the descendants of <testResult> should not be restricted to
relationships of the form " ____ is a test". TestResults already
contains many such relationships: {actual tests, performed by, actual
operator} , {actual repairs, performed prior to, actual tests} {actual
tests, performed on, actual UUT} etc. Why not also allow {actual setup,
performed prior to, actual test} and {actual adjustment, performed
within sequence of, actual tests} etc?
3. Reliance on the TestDescription
to determine what actions took place instead of a recorded action trace
is risky. Reasoner-guided diagnostics and even legacy TPSs are capable
of traces that are not easy to express a priori.
4. The core argument in bullet 2
that is preventing us from putting setups, adjustments etc in
TestResults should apply equally well to TestDescription. What will we
do if TestDescription can't contain any non-test actions either? Does
the argument apply to everything in ATML?
5. The requirements and use cases
for TestResults are what's important. If the name of the root element
"TestResults" is preventing us from meeting the use cases or
requirements, then change the root element.
Darryl
________________________________
From:
stds-ai-estate@ieee.org [mailto:stds-ai-estate@ieee.org] On Behalf Of
Sheppard, John (JSHEPPAR)
Sent: Monday, September 12, 2005 8:43 AM
To: stds-ai-estate@IEEE.ORG
Subject: RE: Proposed change to 1636.1
test results XML schema
John,
Thank you for the concise and cogent
summary of the discussion. I think you captured the issues very well.
That said, I suspect we may need to defer this specific issue until the
meeting at AUTOTESTCON where we will be able to address the issue in an
online, face-to-face setting. My experience is that, when faced with
difficult issues such as this, these face-to-face meetings are
invaluable.
Also, I would like to point out that
some additional information was requested but never provided. Perhaps
the most important additional information was better explanation of the
underlying use case. Since I was not part of the original discussion, I
did not have the benefit of seeing this use case. Therefore, while I
tend to agree with Brit, I do not feel I am able to argue the case
adequately without understanding that use case.
The second bit of information that was
requested (by me) was some kind of "pictorial" representation of the
schema with and without the added information. My preference is an
EXPRESS model, but I am willing to entertain other representations. Note
that I find the XML Spy representation to be very difficult to follow
since it does not provide a "bird's-eye) view of the schema (especially
since I do not have XML Spy to browse the schema and have to rely on
exported documentation).
Bottom line: While I appreciate and
agree with the desire to "git 'r dun," I don't think we have the
necessary information to make the required decision in the coming week.
Therefore, if we absolutely have to make a decision this week, I have to
go with direction in which I am leaning and oppose to the addition.
John
-----
Dr. John W. Sheppard
-- Fellow and Assistant Research
Professor
-- ARINC and Johns Hopkins University
-- jsheppar@arinc.com and
jsheppa2@jhu.edu
-----Original Message-----
From: stds-ai-estate@ieee.org
[mailto:stds-ai-estate@ieee.org] On Behalf Of Ralph, John (Mission
Systems)
Sent: Monday, September 12, 2005
9:25 AM
To: stds-ai-estate@IEEE.ORG
Subject: Proposed change to
1636.1 test results XML schema
There was a good bit of
discussion and a variety of opinions expressed regarding the proposal to
add a "non-test action" element to the 1636.1 Test Results schema. Some
of the discussion revolved around the definition of terms. While I agree
that we should be clear on what we mean when we use certain words, I
don't believe we have the luxury of debating definitions for the next 12
months; we need to take our "best shot" and move forward so that Draft 2
may be submitted.
It was suggested by Brit Frank
that Test Results is not the proper place to capture non-result events
or actions that occur during the course of a test. Rather, he suggested
that Test Description might be a better home for this information. A
clarification on the purpose of Test Description has shown that this
won't work.
Another suggestion was the
creation of a new schema for the specific purpose of recording such
non-result information. The down-side of this approach is that it
fragments the serial flow of data by requiring references to and linking
in external files when it becomes necessary to reconstruct the entire
sequence of events that occured during the course of a test.
I believe we have two options
regarding the suggested modifications to Test Results:
1. Accept the proposed change,
and modify the schema/documentation accordingly, or
2. Reject the proposal and
require users with a desire/need to capture such information to
institute their own use-case specific extension to the Test Results
schema.
Obviously this can't be a
unilateral decision, but a decision must be made soon - preferrably
within the next week.
I solicit your feedback, and
specifically request that potential users of the schema weigh in with
their preference.
John Ralph