Wiki
javarosa / OpenRosa Workgroup Skype Chat 18 Nov 2011 4.24pm EST to 6:59 pm EST
 [4:23:53 PM] Anton de Winter: Taking a snapshot of the conversation so far. [4:25:19 PM | Removed 4:25:25 PM] Anton de Winter: This message has been removed. [4:25:44 PM] Jonathan Jackson: @Cory - would you ever recommend we not use just an int? [4:27:53 PM] Cory Zue: yes, i think if the formdesigner was smart enought to generate "[version].[uiVersion]" that would be an improvement [4:28:09 PM] Jonathan Jackson: so you would actually spend the time to implement that in vellum? [4:28:22 PM] Anton de Winter: That would be a 30 minute fix [4:28:26 PM] Anton de Winter: (or less ) [4:28:30 PM] Clayton Sims: ... [4:28:31 PM] Clayton Sims: "nope" [4:28:32 PM] Jonathan Jackson: to know if something constitutes a model change? [4:28:34 PM] Jonathan Jackson: not at all. [4:28:38 PM] Jonathan Jackson: how about if i change branching logic? [4:29:01 PM] Jonathan Jackson: or logic the nulls something out [4:29:05 PM] Anton de Winter: vellum inherently knows when you alter the model, or just text [4:29:08 PM] Anton de Winter: for example. [4:29:15 PM] Cory Zue: i can also imagine a world where you just want uids and want them to be a reference to all kinds of other metadata [4:29:20 PM] Clayton Sims: So I hve a question WRT uiVerison/Version [4:29:34 PM] Mitch Sundt: I would recommend staying with a single number. The critical factor is what the server (aggregator) can do with the multiple versions. [4:29:36 PM] Clayton Sims: Can you guys give me an idea of a pure "ui change"? [4:29:39 PM] Cory Zue: and uids are easier to deal with than incrementing numbers, particularly in disconnected environments [4:29:51 PM] Matt Adams: That's true. [4:30:00 PM] Clayton Sims: Because if I change the text of a question from "Are you sick" to "Are you not sick" that's not a ui change [4:30:03 PM] Clayton Sims: that's a model change [4:30:04 PM] Mitch Sundt: Each form builder will not be able to understand the limitations of the various aggregators and their abilities to combine datasets. [4:30:19 PM] Matt Adams: Also true. [4:30:23 PM] Mitch Sundt: That's why it should be either input by people or kept as a simple integer. [4:30:31 PM] Cory Zue: clayton, i think the defintition being considered is "something that would cause a sql database to change it's schema" [4:30:37 PM] Cory Zue: i agree it's not a very useful definition [4:30:43 PM] Jonathan Jackson: its not useful at all. [4:30:44 PM] Clayton Sims: So building an automated tool that "detects" model changes seems irrelevant, since we can't merge them correclty anyway [4:30:49 PM] Clayton Sims: QED: Single integer [4:31:00 PM] Cory Zue: here's what i consider a valid use case [4:31:12 PM] Cory Zue: i make a form and deploy it for my pilot [4:31:19 PM] Jonathan Jackson: the way you would test if there was a schema change is by regening the schema code off the xform, regardless of thre id, and checking if it matches [4:31:25 PM] Cory Zue: then i iterate on it for a while and go through a bunch of versions [4:31:26 PM] Drew Roos: a UI change: "formatting or phrasing change that does not meaningfully alter the question" [4:31:32 PM] Cory Zue: then i deploy a totally new app [4:31:35 PM] Clayton Sims: Drew: Undetectable [4:31:38 PM] Cory Zue: and i want a simple way to say "was this a pilot form" [4:31:46 PM] Clayton Sims: change the xmlns [4:31:51 PM] Jonathan Jackson: or change teh version [4:31:56 PM | Edited 4:32:00 PM] Cory Zue: that is readable to a human who doesn't have to remember the version number the change happened [4:32:02 PM] Drew Roos: think of it like the 'minor change' checkbox on confluence [4:32:15 PM] Clayton Sims: Have you _ever_ checked the "minor change button"? [4:32:15 PM] Cory Zue: it's the same xmlns [4:32:20 PM] Drew Roos: all the time [4:32:23 PM] Clayton Sims: ... [4:32:24 PM] Cory Zue: i want to see the data together in ~all places [4:32:28 PM] Drew Roos: i make a lot of minor changes [4:32:29 PM | Edited 4:32:31 PM] Cory Zue: but it's "importantly different" [4:32:45 PM] Jonathan Jackson: but its only importantly different to you because of context [4:32:51 PM] Anton de Winter: @Cory, in that example you'd still not know if 2.3 is the latest version (was there a 2.4?) [4:32:56 PM] Cory Zue: i just don't understand when a less forgiving spec for an identifier was an advantage [4:33:07 PM] Mitch Sundt: Cory: if you change 3 things, would you have "pilot+changes to foobar questions to fix spelling and changes to blah to add new items and changes to bobo to look for images from set" ? [4:33:09 PM] Cory Zue: @anton i don't care [4:33:09 PM] Drew Roos: seriously [4:33:10 PM] Jonathan Jackson: less choices == easier for anyone coming along next [4:33:14 PM] Cory Zue: i only care about 2/3 [4:33:20 PM] Mitch Sundt: How long would the string get? [4:33:24 PM] Drew Roos: server interpret the version attr best the can, and degrade gracefully [4:33:34 PM] Cory Zue: who cares? [4:33:35 PM] Mitch Sundt: At some point, you need to have a short-hand representation of the change [4:33:39 PM] Drew Roos: if it's an int, great, but if not, all bets are off and do a string compare [4:33:41 PM] Mitch Sundt: Integer == shortest possible [4:33:55 PM] Cory Zue: what is the argument for short? [4:34:16 PM] Cory Zue: readable >> short [4:34:20 PM] Cory Zue: freedom >> restrictions [4:34:23 PM] Drew Roos: i think it's funny that the ODK guys, who want to be able to add custom attributes to everything, are so insistent on locking down this particular attribute [4:34:48 PM] Mitch Sundt: And having to allocate 65,000 characters to hold a version string -- where does it stop [4:34:50 PM] Drew Roos: if you don't understand a weird version format, ignore it [4:34:57 PM] Jonathan Jackson: cory, the whole point of this is to restrict freedom [4:34:59 PM] Jonathan Jackson: so I don't agree [4:35:04 PM] Cory Zue: not for this attribute [4:35:09 PM] Jonathan Jackson: yes, [4:35:12 PM] Jonathan Jackson: for this attribute [4:35:12 PM] Mitch Sundt: how long is long enough [4:35:18 PM] Jonathan Jackson: we are saying no unicdoe, certainly [4:35:18 PM] Cory Zue: _who cares_? [4:35:24 PM] Yaw Anokwa: the odk guys want things that are simple. version=1/2/3 is simple. attributes to change appearance without whitelisting is simple. [4:35:29 PM] Cory Zue: are you worried about running out of disk space storing your version ids? [4:35:46 PM] Cory Zue: "put something unique here" is also simple [4:35:54 PM] Mitch Sundt: MySQL has a 65536 byte limit on the size of any data row [4:35:56 PM] Jonathan Jackson: put something here that is in ascii you mean? [4:36:10 PM] Cory Zue: but your implementation can use ints [4:36:14 PM] Mitch Sundt: That's 16,000 UTF-8 characters. [4:36:21 PM | Edited 4:36:26 PM] Cory Zue: i don't see why you'd feel the need to force that on everyone else [4:36:38 PM] Mitch Sundt: The standard is for cross-operation [4:36:41 PM] Jonathan Jackson: we're only forcing the solution everyone is already going to go with [4:36:48 PM] Mitch Sundt: The standard defines the minimal common subset. [4:36:53 PM] Drew Roos: pretty much the only operation we'd ever do on this value is 'determine if one version is newer than another', right? [4:36:58 PM] Jonathan Jackson: yes [4:37:04 PM] Mitch Sundt: Your tools can support more, but they need to support at least this. [4:37:07 PM] Jonathan Jackson: and as far as any of our programmatic capabilites are concerned [4:37:12 PM] Jonathan Jackson: that means "did anything change anywhere on this form" [4:37:26 PM] Cory Zue: (switch to postgres?) [4:37:30 PM] Mitch Sundt: The impact is on the form creation tool. [4:37:31 PM] Cory Zue: (kidding) [4:37:42 PM] Mitch Sundt: PostgreSQL is definitely better. [4:37:49 PM] Drew Roos: why not use timestamps then [4:38:17 PM] Mitch Sundt: Timestamps would work. But I think they aren't as user friendly as an integer. [4:38:33 PM] Jonathan Jackson: correct [4:38:38 PM] Drew Roos: and no one will begrude you if you server throws an error if someone tries to encode lord of the rings as xsd:base64binary in the version attribute [4:38:51 PM] Drew Roos: monotonically increasing ints aren't really that user friendly! [4:38:55 PM] Drew Roos: they're for machines, not people [4:39:08 PM] Jonathan Jackson: that's not at all true. [4:39:14 PM] Jonathan Jackson: everyone understand this: [4:39:22 PM] Jonathan Jackson: "What version did the doc send us? 2" [4:39:27 PM] Jonathan Jackson: "crap, we have 1, can you send me 2" [4:39:40 PM] Cory Zue: that's my argument, jon [4:39:42 PM] Yaw Anokwa: moreover, timestamps don't work well across time zones. [4:39:42 PM] Mitch Sundt: If you want strings, spec a max size, but realize that 4 bytes for integers is way less burdensome on the backend storage than long strings [4:39:56 PM | Edited 4:40:44 PM] Drew Roos: "what's the meaningful difference between 562 and 563" [4:40:06 PM] Jonathan Jackson: NO ONE IS GETTING TO 562 [4:40:13 PM] Cory Zue: are you kidding me? of course they are? [4:40:16 PM] Drew Roos: that's absolutely not true [4:40:22 PM] Jonathan Jackson: who is getting to 562? [4:40:25 PM] Drew Roos: every punctuation change in vellum is a new version [4:40:31 PM] Cory Zue: anyone who uses a ui tool and presses save a lot [4:40:38 PM] Jonathan Jackson: that's the designers choice [4:40:40 PM] Mitch Sundt: You will always need a mapping between a short-hand and what was changed. [4:40:49 PM] Jonathan Jackson: and in that case, you better not ever think someone is going to look at the version [4:40:51 PM] Jonathan Jackson: so it doesn't matter [4:41:12 PM] Cory Zue: so is the only real argument against the "anything" plan the space limitations on disk? [4:41:25 PM] Jonathan Jackson: for me its just simplicity [4:42:46 PM] Mitch Sundt: simplicity. [4:43:34 PM] Mitch Sundt: If you want to describe what your form changes are, you can put it in the <descriptionText> of the FormListAPI. [4:43:56 PM] Mitch Sundt: It doesn't belong in the submission. [4:45:50 PM] Mitch Sundt: I need to get lunch -- signing out for now... [4:53:37 PM] Jonathan Jackson: did we settle the formid / xmlns issue? [4:55:41 PM] Jonathan Jackson: how about: "version is a string representation of the version of the XForm. It is strongly recommended that this is implemented as an integer-valued string that increments with changes to the XForm. The only requirement is that for any two forms of the same id and version, they are identical." [4:56:38 PM] Drew Roos: that would be a SHOULD in spec-speak? [4:57:12 PM] Anton de Winter: I'm fine with making it a string as well with a recommendation for ints [5:06:57 PM] Anton de Winter: This is the current text for xlmns/id: [5:06:58 PM] Anton de Winter: Either id or xmlns identifying the form id. These values should be in the form of scheme:value. For the id, the implementor's registered domain name should be used as part of the scheme (e.g., opendatakit.org:widgetForm). If specified, the id value takes precedence over any explicit xmlns declaration. [5:09:30 PM] Jonathan Jackson: that's more confusing than saying nothing [5:11:08 PM] Anton de Winter: wanna take a crack at re-wording? [5:11:15 PM] Anton de Winter: I just copied and pasted... [5:14:06 PM] Jonathan Jackson: it should be one or the other. [5:14:09 PM] Jonathan Jackson: that was my question [5:20:56 PM] Drew Roos: yaw's whole argument for <foo:DeviceId/> <foo:Image/> <foo:Location/> <foo:Description/> </foo:geotagger> </instance> [6:04:07 PM] Mitch Sundt: So if you have a tool that generates namespaced, named values, how do you know what the identity fo the form is? [6:04:28 PM] Mitch Sundt: Note that the meta data fields were to use orx. [6:04:48 PM] Drew Roos: but the top level xmlns for the resulting instance is the identity of the form [6:05:42 PM] Drew Roos: and the system breaks down if you have multiple forms generating the same type of instance, such as a standardized instance in this case [6:06:02 PM] Mitch Sundt: which would happen if you have versions, etc. [6:06:16 PM] Mitch Sundt: and the user started adding questions [6:06:29 PM] Drew Roos: isn't that why we have a version attribute? [6:06:51 PM] Mitch Sundt: version + form Id define the structure. [6:07:11 PM] Mitch Sundt: xmlns, if it were just formId would not be sufficent to characterize the submission [6:07:23 PM] Drew Roos: why not? [6:08:19 PM] Drew Roos: i guess i can concede the theoretical need to keep track of multiple xforms (note: not just multiple versions of the same form) that generate the same resulting instance [6:08:31 PM] Drew Roos: but i'm not sure this has ever come up in the whole of our xforms experience [6:08:37 PM] Mitch Sundt: If xmlns is dictating the structure of the xml, in the first form, without a "foo" question and with a "bar" question, it would have a different valid DTD than in the second. [6:09:00 PM] Drew Roos: and if you want to do that, the "id" attribute should probably not be on the instance, but in the <model> tag [6:09:31 PM] Mitch Sundt: the model tag doesn't get submitted up with the submission xml [6:11:19 PM] Mitch Sundt: I would argue that the human-graspable concept is ( formId + version ) the mapping to xmlns is more for computer verification of the structure of the xml and should be kept separate. [6:19:08 PM] Anton de Winter: It feels like we're diverging pretty hard here [6:19:30 PM] Anton de Winter: instead of converging on a solution [6:22:34 PM] Mitch Sundt: I hold firm to using id and version. If id is not present, fall back to using xmlns of the tag. If version is not present, it is handled as NULL [6:22:53 PM] Yaw Anokwa: +1 [6:23:17 PM] Matt Adams: +1 that's what we're doing with those values (id/xmlns). Seemed to make sense to me. [6:23:57 PM] Anton de Winter: and hey that's what the spec says.. [6:23:59 PM] Anton de Winter: (mostly) [6:29:52 PM] Anton de Winter: So, assuming that's settled, and versioning is settled [6:29:59 PM] Anton de Winter: are we good for throwing this to an official vote? [6:30:16 PM] Jonathan Jackson: just to be clear. [6:30:22 PM] Jonathan Jackson: we are basically saying its not worth really solving this. [6:30:28 PM] Jonathan Jackson: if tha'ts what we are saying, i'm fine with that. [6:30:41 PM] Matt Adams: What happened with the one bit that I was concerned about - length of IDs? [6:31:31 PM] Anton de Winter: What do you think would be a reasonable length limit? [6:32:19 PM] Matt Adams: If we need a hard limit I think that 256 is going to cover the vast majority of uses. [6:32:34 PM] Anton de Winter: cool, done. [6:34:29 PM] Mitch Sundt: 80 characters is a reasonable length. Nobody has URLs that are 256 characters. Why would we allow anything that long? [6:34:58 PM] Clayton Sims: 0, 1, or infinity [6:36:01 PM] Anton de Winter: 80 seems a little short [6:36:54 PM] Mitch Sundt: really??? This is not the Form Name, or the Description. [6:37:05 PM] Drew Roos: omg who cares [6:37:21 PM] Clayton Sims: We don't want people who are automatically dumping ID's from a foreign system to have to compress them to an arbitrary number [6:37:28 PM] Drew Roos: does any spec have limits on how long attributes can be? [6:37:38 PM] Clayton Sims: (this is a real, meaningful constraint for plenty of people we integrate with) [6:37:44 PM] Mitch Sundt: guys. Databases care. [6:38:01 PM] Anton de Winter: make it the limit of the db then. [6:38:03 PM] Clayton Sims: If you're confident you won't ever need more than 80 characters you should be capable of hashing incoming strings into 80 chars [6:38:04 PM] Anton de Winter: mysql limit [6:38:59 PM] Mitch Sundt: mysql limit applies to the entire row, so it depends upon what else you're storing in the row. [6:39:59 PM] Drew Roos: that sounds like mysql's problem [6:40:08 PM] Anton de Winter: yep. [6:40:10 PM] Mitch Sundt: Interoperability problem. [6:40:58 PM] Anton de Winter: It might be that you guys need to make allowances for when your tech can't deal with a longer ID [6:41:00 PM] Mitch Sundt: Again, do you have integration partners that currently have REEALLY long form ids? [6:41:01 PM] Clayton Sims: Yeah, I don't think we should be arbitrarily restricting our spec to interpoerate with mysql [6:41:03 PM] Anton de Winter: instead of making it part of the spec [6:41:05 PM] Clayton Sims: mitch: Yes [6:41:13 PM] Clayton Sims: They dump their table/internal names to stuff [6:41:15 PM] Clayton Sims: which are long [6:41:35 PM] Mitch Sundt: If you have a current requirement that 80 is too small for, we can increase it. v2 is always easier to grow rather than saddle ourselves with an overly-generous value in v1 [6:41:35 PM] Clayton Sims: Just hash strings [6:41:53 PM] Clayton Sims: an 80 character limit on an id is, like, something you would write in 1993 [6:41:58 PM] Mitch Sundt: hashes work for me. [6:42:01 PM] Clayton Sims: "who would need a file name over 75 characters?" [6:42:31 PM] Mitch Sundt: these days, I'd just do a sha-256 hash of the long string and call it good [6:42:56 PM] Anton de Winter: I honestly think if you guys set an internal limit of 80 chars on aggregate, it will work fine in most cases. A length doesn't need to be part of the spec (esp versin 1.0 of the spec) [6:43:03 PM] Clayton Sims: Cool, either way, the id on our side should be unbounded [6:43:10 PM] Anton de Winter: if you run into problems, actively, it's something we can tackle for version 1.1 [6:43:14 PM] Anton de Winter: since we'll have hard data [6:43:33 PM | Edited 6:43:36 PM] Anton de Winter: ("we drop 10% of submitted forms because their id's are too long, that's not ok") [6:43:55 PM] Anton de Winter: but as of right now, it seems like a lot of speculation on something that doesn't need to be specified right now [6:44:22 PM] Mitch Sundt: depends whether you want this spec to define the core interoperability or a loose, could-bite-you interoperability like the SQL spec. [6:44:37 PM] Mitch Sundt: I would prefer more explicit interoperability [6:44:46 PM] Anton de Winter: I think we should aim for looser and-at-least-get-it-done for 1.0 [6:44:49 PM] Anton de Winter: then tighten it up as needed [6:45:00 PM] Mitch Sundt: tightening never happens [6:45:35 PM] Anton de Winter: There are also ways for you to get around length limitations imposed by mysql [6:47:08 PM] Mitch Sundt: Yes and no. [6:47:42 PM] Drew Roos: including not accepting forms with super-long ids [6:47:52 PM] Mitch Sundt: My argument is set a common lower limit (e.g., 80) if your server handles larger IDs, it is more capable, and the market will drive others to increase their sizes to meet the need. [6:48:11 PM] Mitch Sundt: If you start large, there is no market pressure to change. [6:48:30 PM] Mitch Sundt: The issue is if you create a form, can it be uploaded to any server, and used. [6:49:21 PM] Drew Roos: are we going to limit how many nodes you can have in your instance? [6:49:27 PM] Mitch Sundt: This is why I think we should have the version spec'd as an integer and the form id value limited to 80 characters. Start small, if you have a form designer that does something different, and you walk away from compatibility downstream, so be it. [6:49:43 PM] Mitch Sundt: @Drew: no. [6:50:17 PM] Mitch Sundt: The form designer, though, should be able to specify an integer if you run it in compatibility mode. [6:50:59 PM] Mitch Sundt: Again, it isn't a limiting of the capability of the servers. it is about establishing a common agreed upon minimal functionality. [6:51:41 PM] Mitch Sundt: Add features and go beyond the standard, but still provide at least the common standard. [6:52:01 PM] Anton de Winter: setting it low at 80 is basically asking us to immediately break from the standard [6:52:13 PM] Anton de Winter: which makes the exercise somewhat pointless [6:52:14 PM] Drew Roos: i don't understand how the mysql row constraint doesn't affect nearly every aspect of your xform handling [6:52:21 PM] Drew Roos: and why similar limits aren't needed there [6:52:21 PM] Mitch Sundt: what value do you need right now? [6:53:26 PM] Anton de Winter: I don't think setting a limit makes much sense [6:53:32 PM] Anton de Winter: Drew's question is pretty valid [6:53:39 PM] Anton de Winter: are you going to limit the number of nodes in an instance? [6:53:48 PM] Anton de Winter: because if we limit here, we should limit there [6:53:54 PM] Anton de Winter: and while we're at it, limit the length of each nodeid [6:54:02 PM] Anton de Winter: so that it all fits nicely in mysql [6:54:36 PM] Mitch Sundt: If you need a longer string length for form ids, how about 249? That is the maximum length for a searchable string in Google BigTable. Anything beyond that would be onerous to support. [6:54:58 PM] Mitch Sundt: As for the representation of the form, I have a loop that looks like this: [6:55:03 PM] Mitch Sundt: for (;;) { [6:55:20 PM] Mitch Sundt: try { create table with columns [6:55:26 PM] Mitch Sundt: } catch ( failure ) { [6:55:49 PM] Mitch Sundt: divide table into tables with linked primary keys. [6:55:51 PM] Mitch Sundt: } [6:55:53 PM] Mitch Sundt: } [6:56:20 PM] Mitch Sundt: i.e., I keep trying to create the tables, splitting them until the database stops complaining about not being able to create them. [6:56:22 PM] Cory Zue: if you're already doing that why are you so concerned about these arbitrary limits? [6:56:38 PM] Drew Roos: i'd be happy with a practical limit in the neighbrhood of 256 but still don't think a limit needs to be explicitly specified [6:57:03 PM] Mitch Sundt: The tables have multiple columns. Spliltting them moves the storage requirement to a different table. [6:57:29 PM] Mitch Sundt: If you have one large field, the databases consume space storing that field. [6:58:05 PM] Mitch Sundt: If you say 1000 characters, and my system only works for 249, I've got a problem supporting the standard. [6:58:12 PM] Cory Zue: i think if we separate MUST and SHOULD then we can sandbox all the space/mysql constraints into SHOULDs and stop fighting ? [6:59:05 PM] Mitch Sundt: that would mean the systems implementing the MUSTs are not necessarily interoperable. I thought that was the goal. [6:59:21 PM] Mitch Sundt: MUST support form ids of 249 characters or less. [6:59:37 PM] Mitch Sundt: MUST support integer version values. [6:59:48 PM] Drew Roos: is there a limit on how long the User-Agent header in HTTP can be? [6:59:48 PM] Mitch Sundt: SHOULD support non-integer version values. [7:00:33 PM] Cory Zue: drew, i don't think so [7:00:47 PM] Cory Zue: mitch, i think that's fine? [7:00:47 PM] Drew Roos: right [7:01:22 PM] Drew Roos: if you get data that is larger than what you determine to be "reasonable", return an error [7:02:12 PM] Mitch Sundt: as long as the MUST is what everyone knows will interoperate. [7:02:20 PM] Mitch Sundt: i agree [7:03:12 PM] Anton de Winter: ok [7:03:29 PM] Anton de Winter: so MUST be 249 chars, SHOULD be any length [7:04:00 PM] Anton de Winter: does that work? [7:04:09 PM] Mitch Sundt: yes [7:04:11 PM] Anton de Winter: sweet [7:04:31 PM] Anton de Winter: The language you stated for version is something I'm also happy with including [7:04:38 PM] Mitch Sundt: great [7:04:53 PM] Anton de Winter: I'll likely kick off voting tomorrow but have it run into monday [7:04:59 PM] Anton de Winter: or early tuesday [7:05:08 PM] Mitch Sundt: ok. I'll not be online over the weekend. [7:05:17 PM] Anton de Winter: yeah, I'm not thinking that many people will be ;) [7:05:38 PM] Mitch Sundt: You should search for uiversion in the spec -- I think there's some reference to it down at the bottom of the page that will need to be updated. [7:05:40 PM | Edited 7:06:00 PM] Anton de Winter: but after all this discusison I think everyone has a good feel for how they want to vote (so it shouldn't take long) [7:05:49 PM] Anton de Winter: cool, will do [7:06:15 PM] Anton de Winter: Logs have also been archived [7:06:21 PM] Anton de Winter: Chat logs*
Updated