|
On 4/29/13 9:20 PM, Dmitry Nadezhin
wrote:
Richard,
I like the idea of Michel Hack about preprocessor from my format
to your format.
I think that what you are talking about with a Reference Checker is
a program
that generates a test suite input and output. It is (part of) a
pre-processor that takes, for example,
rational input and a type designation and produces type-specific
"correct" results.
The Reference Checker, if complete and correct, sounds to me like it
is a reference implementation.
I shall try to write it in Java. I'm Sorry :). You can
write it as separate phase before Lisp.
So I answer your questions and I do not argue about which format
is better.
> > 1) I would like to use in test suite symbolic names
like "add" instead of numbers.
> I would personally prefer this too.
> I was trying to make it perfectly unambiguous and not have
> to worry about character sets, capitalization, Add vs
ADD vs unicode vs ASCII.
I suggest to use exacty the names that are in Draft of the
standard.
All of them are ASCII. The names must match exactly. "add"
can't be replaced by "Add" or "ADD".
> > 2) Is it necessary to put name on every line ?
> > There may be a header line
with operation name before a few lines with operation
instances.
>
> I think each line should stand alone without reference to
any other line, so that the
> line can be run through a program all by itself.
Ok. Preprocessor will put operation name on every line.
> > 3) The number can of inputs in operation instance is
fixed for most operations.
>
> Yes
>
> > It is variable for such operations like "min",
"max".
> > There are also overloaded reverse operations
invRev(c) and invRev(c,x).
> > Nevertheless, reader can count number of inputs in a
line for such operations.
>
> I am trying to make the access entirely uniform and not
make the reader program
> any more complicated than necessary.
>
> > It is not necessary to write the counter explicitly
>
> For the same reason as above, I think each line should
have a sequence number.
> There might be gaps in the sequence, but I think it is
good for each line
> to be referred to by a single identifier
> so you can say "my program gets a different result for
line xxxxx.
> Is my program wrong or is the test suite wrong?"
Ok. Preprocessor will put number of inputs and number of
outputs on every line.
> > 4) The outputs may be optional for some tools.
> > For example Reference Checker reads input, computes
tightest interval internally
> > and checks that it implementation under test returns
result that contains the tightest.
>
> I don't have "reference checker" in my list..
> But if it computes something like "Yes this result is
checked"
> then I suppose it could be added to the test suite.
> If a program has no output then it computes nothing and
can be discarded :)
Checker permits to reuse test suite line for different
datatypes.
So it treats inputs as exact rational intervals.
Intervals like [0.1,1/3] are allowed in inputs.
The interval datatype T is the parameter of test run.
Checker reads operation name and rational inputs.
read(f);
read(x).
It convert them to specified datatype by T-hull operation if
necessary.
xt = hull_T(x).
Then it found the tigtest result
zt = hull_T (f_exact (xt)).
The it runs operation from library under test
zi = f_impl(xt).
Thet it checks that (zt \subseteq zi) and reports result.
This is the reason why outputs are optional in my format.
> > I suggest to separate inputs and outputs (if
present) by some mark, for example "->".
>
> why? A program to read and format the test suite,
> or to parse some other input and produce the
> test suite format, could be written .
> I wanted to make the format so simple
> that any programming language could easily read it,
> and not have to check to see if the next field is a
number or a mark.
Ok. Preprocessor will not put any mark between inputs and
outputs.
> > 5) I suggest to use "hexadecimal-significand" form
as described in section 13.4 of Draft 7.1 instead of tripples.
>
> I disagree.
> Why should each person have to write a parser for such
elaborate stuff.
> Furthermore, it doesn't even cover everything the triples
cover,
> because it deals only with finite numbers not infinities
etc etc.
> Furthermore, the representations are subject to rounding
to get to the actual numbers.
> So, unless I am misunderstanding something here, it looks
unsuitable.
How do you want to represent infinitives by triples ?
the IEEE 754 standard has unambiguous representations for +-oo for
each base type.
We are told in section 9.6 that empty = [oo,-oo] by the definition
of inf(empty) and sup(empty).
The representation for NaI is specified (so far as I can tell) in 2
conflicting ways, in section 9.6 .
If it is empty with decoration ILL, then as a bare interval it
would be indistinguishable from
empty with some other decoration. But section 9.6 ALSO suggests
using naN perhaps with a payload.
My suggestion is to use a particular NaN for the lower limit and
zero for the upper limit, at least for
the test suite.
How do you want to represent empty interval ?
> > 6) Is test sequence number necessary ? Shouldn't it
be optional
>
> No, but if you wish to provide a test, I will be happy to
provide a test sequence number for it!
Can it be test name that is not necessary a number ?
There will be, I think a few thousand lines. I proposed numbers to
make it easy. If
you want to provide a grammar for alpha-numeric acceptable
identifiers, you are making
more work. For Lisp, it makes no difference to read a symbol or a
number or for that
matter, a triple of numbers. I am trying to make it easier for
someone who might be
writing in Matlab, Ruby, PHP, Java, C, Haskell, Fortran, _javascript_,
Visual Basic, assembly language.
>> 7) Are comments allowed
>
> I'm agreeable to anything that is convenient.
> What would you like?
> For Lisp, the convention is that any line beginning with
semicolon ";" is a comment.
> If you want to use some convention for comments at the
end of a line, that's OK too.
> If you wish to use // or .... # that's OK with me.
I think that most people will prefer C++ style comments
starting with // .
> alternatively, since each line has a sequence number, a
separate file with sequence numbers and comments
> may be useful; this may keep the test file relatively
compact.
Ok. Preprocessor can strip out all comments.
> Frankly, I don't know what you mean by [1,2].
> Is the input widened? is the output? how do you express
oo or -oo?
+oo and -oo are expressed by +infinity and -infinity .
these are triples in my notation.
Intervals bounds are widened only if they do not fit in
format.
Unacceptable. (to me, at least)
If an input interval cannot be expressed exactly in the format
being tested, then the line should be rejected as a test in that
format.
Rounding up/down/widening/ computing (tightest) hull is the essence
of the computation being tested. If you allow the program to
"interpret"
the test suite by choosing some roundings and hull computations you
are invalidating the purpose of the test.
Input [1,2] remains as
[0x1.0000000000000p1,0x1.0000000000000p2]
Input [0.1,1] is widend to
[0x1.9999999999999p-4,0x1.0000000000000ap1]
Preprocessor will write widened..
If the preprocessor writes out the test suite so the post-processed
text has
no ambiguity, no hidden widening or rounding, and can be run through
EVERY
proposed reference standard (not just yours), then it makes sense.
If the
post-processor text is again subjected to widening in some secret
way,
and tests for the right answers are not explicit exact comparisons,
there
seem to be issues. I am not sure how one can specify numbers that
need
to be merely accurate and not necessarily tight. I was hoping
someone
would figure out a good approach.
> I have not put notation in for decorations.
> I would prefer that stuff not be alphabetic.
> I suppose that each interval could be (triple)(triple)
DECORATION
> where DECORATION is 0,1,2,3,4
> Though illegal and empty can be detected by special
triples, I think.
> com, dac, def may be all that is necessary.
Please give me example how you want to represent empty
intervals.
The standard implicitly says [oo,-oo] is how to represent empty.
|