I was thinking that a key-value datastore might scale better for larger datasets. If not Cassandra, maybe RIAK (RESTful interface, fairly straightforward to deploy).

@Ross – maybe it’s my background in protocol engineering – one of the primary tenets of early protocol design was to make everything BOTH human and machine readable – makes for good error messages and diagnostics (e.g., you can telnet to a web site and, at least to an unencrypted port, and exchange commands and read the protocol responses) – the same principle is implicit in RESTful APIs as opposed to SOA-style web service interfaces – generally simplifies things all around

as to searching CouchDB – I was thinking that if the database is pre-processed, once, then writing map-reduce functions (e.g., for SRU/CQL queries) becomes pretty trivial

though, in thinking it all through, I’d think that something like Cassandra would be a better way to handle metadata

Since (I think) the intention is to still be able to share ISIS records amongst organizations, this would be lost by mapping to some other metadata format.

You can’t “search” CouchDB (without a map/reduce function), anyway, so you’d need some abstraction layer to translate SRU/CQL to something usable in the first place. At that point you can translate human readable terms into whatever your local representation uses without introducing any lossiness into your data model.

Bill Dueber came up with marc-hash: http://robotlibrarian.billdueber.com/new-interest-in-marc-hash-json/ which is extremely similar to ISIS-JSON Type 2. The rationale was that serialization/parsing would be extremely fast and it was 100% roundtrippable (which is a big deal in MARC circles).

Andrew Houghton created MARC-JSON: http://www.oclc.org/developer/content/marc-json-draft-2010-03-11 which is very heavily inspired by marcxml. It is also very well documented and 100% roundtrippable (ok, all of the current proposals are).

Both of these proposals have their advantages, but neither lend themselves terribly well to JSONPath type queries (which is useful for the JSON-based document stores), which is why I drafted MARC-in-JSON: http://dilettantes.code4lib.org/blog/2010/09/a-proposal-to-serialize-marc-in-json/. It’s considerably more verbose than Bill’s a little less (maybe) than Andrew’s.

I’m not sure Andrew’s is supported in any openly available MARC parsers; marc-hash and MARC-in-JSON are both available in File_MARC (PHP) and ruby-marc.

I definitely second Jonathan’s statement that, since the formats are so similar, it would be really nice to find some common ground between the two formats in JSON.

That said, there didn’t seem to be much interest in formalizing a MARC-JSON serialization in this (extremely long) CODE4LIB mailing list thread: http://www.mail-archive.com/code4lib@listserv.nd.edu/msg08077.html

http://dilettantes.code4lib.org/blog/2010/09/a-proposal-to-serialize-marc-in-json/

I have personally not yet compared them to see how they differ.

Since the ISIS format you are working with is based on ISO-2709, like Marc, it is a similar domain, although different in some ways. It might be nice to try and arrive at a standard that meets both needs.