Greetings from Amsterdam
Steven Pemberton, CWI Amsterdam
Version: 2020-07-30.
Submission, the process of sending data to a server and dealing with the response, is probably the hardest part of XForms to implement, and certainly involves the XForms element with the most attributes. This is largely due to legacy: XForms was designed to work with existing standards, and HTTP submission was designed before XML existed: the data representations are several, and on occasion byzantine.
Part of the process of producing a standard such as XForms is a test suite to check implementability of the specification. The original XForms test suite consisted of a large collection of XForms, one XForm per feature to be tested. These had to be run by hand, and the output inspected to determine if the test had passed.
As a part of the XForms 2.0 effort, a new test suite is being designed and built. This tests features by introspection, without user intervention, so that the XForm itself can report if it has passed or not. Current work within the test suite is on submission.
This paper gives an overview of how the test suite works, and discusses the issues involved with submission, the XForms approach to it, and how to go about introspecting something that has left the client before you can cast your eyes on it.
A declarative programming language. (Designed to be accessible out of the box ;-) )
Recently celebrated 10 years of XForms 1.1
XForms 2.0 now in preparation.
XForms 1.0 and XForms 1.1 both had a Testsuite:
XForms 2.0 is in progress, and with it Testsuite 2:
One large XForm with XML description of the tests.
Loads and runs tests one by one, or allows individual tests to be run.
The suite consists of chapters of tests containing:
<testsuite spec="XForms 2.0" version="0.4">
<chapter name="about" title="About XForms"/>
<chapter name="introduction" title="Introduction to XForms"/>
<chapter name="structure" title="Document Structure">
<test name="namespace" spec="structure-namespace">XForms namespace</test>
<test name="3.2.1.a" spec="structure-attrs-common">id attribute</test>
<test name="3.2.1.b" spec="structure-attrs-common">foreign attributes</test>
...
The suite consists of chapters of tests containing:
<testsuite spec="XForms 2.0" version="0.4">
<chapter number="1" name="about" title="About XForms"/>
<chapter number="2" name="introduction" title="Introduction to XForms"/>
<chapter number="3" name="structure" title="Document Structure">
<test name="namespace" spec="structure-namespace">XForms namespace</test>
<test name="3.2.1.a" spec="structure-attrs-common">id attribute</test>
<test name="3.2.1.b" spec="structure-attrs-common">foreign attributes</test>
...
The suite consists of chapters of tests containing:
<testsuite spec="XForms 2.0" version="0.4">
<chapter number="1" name="about" title="About XForms"/>
<chapter number="2" name="introduction" title="Introduction to XForms"/>
<chapter number="3" name="structure" title="Document Structure">
<test name="namespace" spec="structure-namespace">XForms namespace</test>
<test name="3.2.1.a" spec="structure-attrs-common">id attribute</test>
<test name="3.2.1.b" spec="structure-attrs-common">foreign attributes</test>
...
The suite consists of chapters of tests containing:
<testsuite spec="XForms 2.0" version="0.4">
<chapter number="1" name="about" title="About XForms"/>
<chapter number="2" name="introduction" title="Introduction to XForms"/>
<chapter number="3" name="structure" title="Document Structure">
<test name="namespace" spec="structure-namespace">XForms namespace</test>
<test name="3.2.1.a" spec="structure-attrs-common">id attribute</test>
<test name="3.2.1.b" spec="structure-attrs-common">foreign attributes</test>
...
Each test is a separate XForm containing any number of test cases, largely using a standard template.
The template includes an instance containing title, description and test cases.
<tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
Each test is a separate XForm containing any number of test cases, largely using a standard template.
The template includes an instance containing title, description and test cases.
<tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
Each test is a separate XForm containing any number of test cases, largely using a standard template.
The template includes an instance containing title, description and test cases.
<tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
Each test case has (optional) values in its content used as parameters to the test, as well as attributes:
req: for the required result, res: for the actual result, pass: recording if the test has passed. <tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
Each test case has (optional) values in its content used as parameters to the test, as well as attributes:
req: for the required result, res: for the actual result, pass: recording if the test has passed. <tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
Each test case has (optional) values in its content used as parameters to the test, as well as attributes:
req: for the required result, res: for the actual result, pass: recording if the test has passed. <tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
Each test case has (optional) values in its content used as parameters to the test, as well as attributes:
req: for the required result, res: for the actual result, pass: recording if the test has passed. <tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
Each test case has (optional) values in its content used as parameters to the test, as well as attributes:
req: for the required result, res: for the actual result, pass: recording if the test has passed.
<tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
Each test case has (optional) values in its content used as parameters to the test, as well as attributes:
The top-level tests element has an attribute
pass that records if all cases have passed.
.
<tests pass="" xmlns="">
<description title="days-from-date()">days-from-date() function</description>
<test pass="" res="" req="-1">1969-12-31</test>
<test pass="" res="" req="0">1970-01-01</test>
<test pass="" res="" req="1">1970-01-02</test>
<test pass="" res="" req="4">1970-01-05T01:01:01.01Z</test>
...
bind elements set up the tests. Calculating the results:
<bind ref="test/@res" calculate="days-from-date(..)"/>
(Note that this single bind sets up all the tests.)
Calculating whether each test has succeeded:
<bind ref="test/@pass" calculate="if(../@res = ../@req, 'yes', 'no')"/>
and whether all tests have succeeded:
<bind ref="@pass" calculate="if(//test[@pass!='yes'], 'FAIL', 'PASS')"/>
bind elements set up the tests. Calculating the
results:
<bind ref="test/@res" calculate="days-from-date(..)"/>
(Note that this single bind sets up all the
tests.)
Calculating whether each test has succeeded:
<bind ref="test/@pass" calculate="if(../@res = ../@req, 'yes', 'no')"/>
and whether all tests have succeeded:
<bind ref="@pass" calculate="if(//test[@pass!='yes'], 'FAIL', 'PASS')"/>
bind elements set up the tests. Calculating the results:
<bind ref="test/@res" calculate="days-from-date(..)"/>
(Note that this single bind sets up all the tests.)
Calculating whether each test has succeeded:
<bind ref="test/@pass" calculate="if(../@res = ../@req, 'yes', 'no')"/>
and whether all tests have succeeded:
<bind ref="@pass" calculate="if(//test[@pass!='yes'], 'FAIL', 'PASS')"/>
bind elements set up the tests. Calculating the results:
<bind ref="test/@res" calculate="days-from-date(..)"/>
(Note that this single bind sets up all the tests.)
Calculating whether each test has succeeded:
<bind ref="test/@pass" calculate="if(../@res = ../@req, 'yes', 'no')"/>
and whether all tests have succeeded:
<bind ref="@pass" calculate="if(//test[@pass!='yes'], 'FAIL', 'PASS')"/>
<group>
<label class="title" ref="description/@title"/>
<output class="block" ref="description"/>
<output class="{@pass}" ref="@pass"/>
<repeat ref="test">
<output value="."/> → <output ref="@res"/>
<output class="wrong" ref="@req[.!=../@res]"/>
</repeat>
</group>
<group>
<label class="title" ref="description/@title"/>
<output class="block" ref="description"/>
<output class="{@pass}" ref="@pass"/>
<repeat ref="test">
<output value="."/> → <output ref="@res"/>
<output class="wrong" ref="@req[.!=../@res]"/>
</repeat>
</group>
<group>
<label class="title" ref="description/@title"/>
<output class="block" ref="description"/>
<output class="{@pass}" ref="@pass"/>
<repeat ref="test">
<output value="."/> → <output ref="@res"/>
<output class="wrong" ref="@req[.!=../@res]"/>
</repeat>
</group>
<group>
<label class="title" ref="description/@title"/>
<output class="block" ref="description"/>
<output class="{@pass}" ref="@pass"/>
<repeat ref="test">
<output value="."/> → <output ref="@res"/>
<output class="wrong" ref="@req[.!=../@res]"/>
</repeat>
</group>
<group>
<label class="title" ref="description/@title"/>
<output class="block" ref="description"/>
<output class="{@pass}" ref="@pass"/>
<repeat ref="test">
<output value="."/> → <output ref="@res"/>
<output class="wrong" ref="@req[.!=../@res]"/>
</repeat>
</group>
These examples test a function, and @res gets set with the
value of the function.
Other tests set the @res attribute in different ways, and you
will shortly see lots of examples of this.
The process of sending data to a server and dealing with the response.
One of the harder parts of XForms to implement:
Submission is also hard to test, because the very thing you want to test has left the client before you get a chance to introspect.
Submission in XForms has a number of steps performed by the implementation:
Since the serialization leaves the client without the possibility of introspection, you need to get it back into the client for checking.
The original testsuite used a server with a special echo CGI
URI, that caused the submitted serialization just to be returned unchanged to
the client:
action="http://xformstest.org/cgi-bin/echo.sh"
so that submission tests submitted the data, and then displayed the result, which the tester would have to inspect.
The CGI solved part of the problem, but very few webservers actually implement all of HTTP.
Whilst not testing servers, we do need to test that XForms implementations correctly talk to servers that do implement all of HTTP.
Our solution has been to write a server [XContents] that supports all of HTTP, and can be run locally.
No need for an echo facility: you can now just do a PUT followed by a GET.
The server is a fairly straightforward:
The only interesting case is POST:
<post>...</post> . (Therefore, if you don't like the tags
<post>...</post> when the file is created, you can
first PUT an empty file, or one containing only the open and
closing tags, e.g. <data></data>, before you
POST to it.)
The general template for submission tests includes three instances (though not all tests need all three):
<instance id="tests">...</instance> <instance id="data">...</instance> <instance id="result"> <data xmlns=""/> </instance>
To ensure we are really getting the right result back, on
start-up we initialise a req attribute to a random value,
before submitting the data:
<action ev:event="xforms-ready">
<setvalue ref="instance('data')/test[1]/@req" value="random()"/>
<send submission="put"/>
</action>
The submission called put is responsible for the submission:
<submission id="put" ref="instance('data')"
resource="test.xml" method="put" replace="none"/>
To ensure we are really getting the right result back, on start-up we
initialise a req attribute to a random value,
before submitting the data:
<action ev:event="xforms-ready">
<setvalue ref="instance('data')/test[1]/@req" value="random()"/>
<send submission="put"/>
</action>
The submission called put is responsible for the submission:
<submission id="put" ref="instance('data')"
resource="test.xml" method="put" replace="none"/>
To ensure we are really getting the right result back, on start-up we
initialise a req attribute to a random value, before
submitting the data:
<action ev:event="xforms-ready">
<setvalue ref="instance('data')/test[1]/@req" value="random()"/>
<send submission="put"/>
</action>
The submission called put is responsible for the submission:
<submission id="put" ref="instance('data')"
resource="test.xml" method="put" replace="none"/>
There are two things that can happen in response to this.
1) the submission succeeds, in which case we initiate retrieving the instance we just sent:
<action ev:event="xforms-submit-done"> <send submission="get"/> </action>
There are two things that can happen in response to this.
1) the submission succeeds, in which case we initiate retrieving the instance we just sent:
<action ev:event="xforms-submit-done"> <send submission="get"/> </action>
There are two things that can happen in response to this.
2) the submission fails, where we just set the result to an error message, and leave it at that:
<action ev:event="xforms-submit-error">
<setvalue ref="test/@res"
value="concat('xforms-submit-error on PUT: ',
event('response-reason-phrase'))"/>
</action>
There are two things that can happen in response to this.
2) the submission fails, where we just set the result to an error message, and leave it at that:
<action ev:event="xforms-submit-error">
<setvalue ref="test/@res"
value="concat('xforms-submit-error on PUT: ',
event('response-reason-phrase'))"/>
</action>
These two actions are placed in the body of the put
submission:
<submission id="put" resource="test.xml" method="put" replace="none">
<action ev:event="xforms-submit-done">
<send submission="get"/>
</action>
<action ev:event="xforms-submit-error">
<setvalue ref="test/@res"
value="concat('xforms-submit-error on PUT: ',
event('response-reason-phrase'))"/>
</action>
</submission>
An action element that contains a single sub-action can be
contracted.
This is equivalent and shorter:
<submission id="put" resource="test.xml" method="put" replace="none">
<send ev:event="xforms-submit-done" submission="get"/>
<setvalue ev:event="xforms-submit-error" ref="test/@res"
value="concat('xforms-submit-error on PUT: ',
event('response-reason-phrase'))"/>
</submission>
In the case of success, the submission called get is initiated,
which GETs the resource, and stores it in the result instance:
<send ev:event="xforms-submit-done" submission="get"/>
<submission id="get" resource="test.xml" method="get" serialization="none"
replace="instance" instance="result">
...
</submission>
This too has two event listeners in its body -- in the case of success, the random number we generated earlier gets copied to the result attribute; in the case of a failure, an error message:
<submission id="get" resource="test.xml" method="get" serialization="none"
replace="instance" instance="result">
<setvalue ev:event="xforms-submit-done" ref="test/@res"
value="instance('result')/test/@req"/>
<setvalue ev:event="xforms-submit-error" ref="test/@res"
value="concat('xforms-submit-error on GET: ',
event('response-reason-phrase'))"/>
</submission>
This too has two event listeners in its body -- in the case of success, the random number we generated earlier gets copied to the result attribute; in the case of a failure, an error message:
<submission id="get" resource="test.xml" method="get" serialization="none"
replace="instance" instance="result">
<setvalue ev:event="xforms-submit-done" ref="test/@res"
value="instance('result')/test/@req"/>
<setvalue ev:event="xforms-submit-error" ref="test/@res"
value="concat('xforms-submit-error on GET: ',
event('response-reason-phrase'))"/>
</submission>
Here are examples of output. Note that this uses exactly the same output code as the earlier template.
(Unlike most other examples in this talk, this is not live, but an image, since the server for these slides doesn't implement PUT)
A failure case of a server not supporting PUT. (This one is live)
(Earlier tests had already tested features such as the correct events being sent on completion).
Trying to submit with no data should generate an
xforms-submit-error event with an error-type of
no-data.
ref="nodata")resource="doesntmatter")replace="none"):<submission ref="nodata" resource="doesntmatter" replace="none">
<setvalue ev:event="xforms-submit-error"
ref="test/@res" value="event('error-type')"/>
<setvalue ev:event="xforms-submit-done"
ref="test/@res">submission incorrectly succeeded</setvalue>
</submission>
ref="nodata")resource="doesntmatter")replace="none"):<submission ref="nodata" resource="doesntmatter" replace="none">
<setvalue ev:event="xforms-submit-error"
ref="test/@res" value="event('error-type')"/>
<setvalue ev:event="xforms-submit-done"
ref="test/@res">submission/ incorrectly succeeded</setvalue>
</submission>
ref="nodata")resource="doesntmatter")replace="none"):<submission ref="nodata" resource="doesntmatter" replace="none">
<setvalue ev:event="xforms-submit-error"
ref="test/@res" value="event('error-type')"/>
<setvalue ev:event="xforms-submit-done"
ref="test/@res">submission incorrectly succeeded</setvalue>
</submission>
ref="nodata")resource="doesntmatter")replace="none"):<submission ref="nodata" resource="doesntmatter" replace="none">
<setvalue ev:event="xforms-submit-error"
ref="test/@res" value="event('error-type')"/>
<setvalue ev:event="xforms-submit-done"
ref="test/@res">submission incorrectly succeeded</setvalue>
</submission>
ref="nodata")resource="doesntmatter")replace="none"):<submission ref="nodata" resource="doesntmatter" replace="none">
<setvalue ev:event="xforms-submit-error"
ref="test/@res" value="event('error-type')"/>
<setvalue ev:event="xforms-submit-done"
ref="test/@res">submission incorrectly succeeded</setvalue>
</submission>
If you try to submit invalid data, submission should also fail, so this example works similarly.
By default, non-relevant data is elided. We have data to submit:
<instance id="data">
<tests pass="" xmlns="">
<description title="Submission with relevance">
Check that only relevant data is submitted by default.</description>
<test pass="" res="" req=""/>
<test pass="" res="" req="nonrelevant"/>
<test pass="" res="" req="nonrelevant"/>
</tests>
</instance>
We mark some elements as non-relevant:
<bind ref="instance('data')/test" relevant="@req!='nonrelevant'"/>
Initialise with a random number as before, submit, and GET it back, replacing the main instance:
<submission id="get" resource="test.xml" method="get" serialization="none"
replace="instance" instance="tests">
<setvalue ev:event="xforms-submit-done" ref="test[1]/@res"
value="instance('data')/test[1]/@req"/>
<setvalue ev:event="xforms-submit-error" ref="test/@res"
value="concat('GET xforms-submit-error: ',
event('response-reason-phrase'))"/>
</submission>
Values that were not relevant in the source data should not appear in the result.
One of the possible ways of serialising data to the server is as part of the URI, and this is often the case with the GET method.
test.xml?q=word&n=99
Introspecting this is easier, because the events that signal success or failure contain the initiating URI.
So we prepare some data.
<data xmlns=""> <value attr="">one</value> <text attr="attr" ibute="ibute">ual</text> <number>3.14159</number> </data>
The testcases instance contains the serialization we expect:
<tests pass="" xmlns="">
<description title="Serialization on GET with three values">
With GET, serialization is done in the URI in a simplified fashion. This tests several facets of values.
</description>
<test pass="" res=""
req="test.xml?value=one&text=ual&number=3.14159"/>
</tests>
We submit in the usual way, not caring if it succeeds or fails, since both events contain the URI we want:
<submission id="get" ref="instance('data')" resource="test.xml"
method="get" replace="none">
<setvalue ev:event="xforms-submit-error" ref="test/@res"
value="event('resource-uri')"/>
<setvalue ev:event="xforms-submit-done" ref="test/@res"
value="event('resource-uri')"/>
</submission>
Of course, we can package lots of these up in a single instance:
<instance id="tests">
<tests name="Serialization on GET" pass="" xmlns="">
<description>With GET, serialization is done in the URI in a simplified fashion.</description>
<test pass="" res="" req="test.xml?data=value"><data attr="ignored">value</data></test>
<test pass="" res="" req="test.xml?value=one&text=text"><data><value this="is ignored">one</value><text>text</text></data></test>
<test pass="" res="" req="test.xml?value=one&text=ual&number=3.14159">
<data>
<value attr="">one</value>
<text attr="attr" ibute="ibute">ual</text>
<number>3.14159</number>
</data>
</test>
<test pass="" res="" req="test.xml?one=1&two=2%266%3B&three=3">
...
Work in progress: many tests have still to be added, which is a top priority.
Work will be done on the test infrastructure, in particular storing the results of tests, (easier now with PUT).
One aspect that will have to be treated differently is the generation of HTTP headers.
These are hidden in the protocol, and not part of the content and therefore hard to test. While headers returned from the server are accessible to an XForm, headers sent by the implementation are not.
An addition to the server will probably have to be added in order to be able to gain access to those parts of the protocol.
Self-testing through introspection makes life much easier for testers.
The new test infrastructure makes it much easier to add and update tests.
Although testing submission has challenges compared with other parts of XForms, even these can be overcome using a fairly uniform approach, without too much overhead.
One surprise:
Thanks @Steven Pemberton for your https://homepages.cwi.nl/~steven/forms/TestSuite/index.xhtml
In the desert of learning xforms this is a valuable and helpful contribution. Not only do I get to learn about the definitions but there is also a test suite download to play around with as well as right clicking on the page for the source and seeing how xforms are put together. Thanks.
-- John Reed on xmlcom slack channel
