diff options
Diffstat (limited to 'FILEFORMAT_1')
| -rw-r--r-- | FILEFORMAT_1 | 88 |
1 files changed, 88 insertions, 0 deletions
diff --git a/FILEFORMAT_1 b/FILEFORMAT_1 new file mode 100644 index 0000000..b16d85a --- /dev/null +++ b/FILEFORMAT_1 @@ -0,0 +1,88 @@ +Version 1 +--------- + +Following sections explain internals of metastore file (.metadata), version 1 + + +### Data types + + SIGNATURE = Magic signature, 10 bytes long = "MeTaSt00r3" + VERSION = Format version string, 8 bytes long. This version = "00000001" + URLSTRING = URL-encoded string. Chars 0x00 to 0x20 (inclusive), 0x7F, and + 0x25 (%) *must* be encoded. Terminated by any character which + must be encoded, which is not (e.g. "\t", "\n"). + INTSTRING = ASCII-encoded integer, in a pre-specified base from 2 (binary) + to 16 (hexadecimal). May be preceded by "-" for negative values + +### File layout + + SIGNATURE VERSION "\n" + n * (ENTRY "\n") + + +### ENTRY format + + URLSTRING - Path (absolute or relative) + "\t" URLSTRING - Owner (owner name, not uid) + "\t" URLSTRING - Group (group name, not gid) + "\t" INTSTRING - Mode (base 8, of struct stat.st_mode & 0177777. + i.e. File type and mode, as per inode(7) + "\t" URLSTRING - Mtime (including nanoseconds) in ISO-8601 format, UTC. + "YYYY-mm-ddTHH:MM:SS.nnnnnnnnnZ" + + m * ("\t" URLSTRING "\t" URLSTRING) + - xattr name/value pairs. `m` may be 0. + + "\n" - Entry-terminating newline. + + +### Discussion + +This format is designed to work with version control systems, specifically +`git(1)`. + +To fit in with `git` and its related tooling, this format is a line-based text +file. Each record is a bunch of text fields separated by tabs, terminated by a +newline. This means records should be identifiable and somewhat understandable +to readers, and should work with `diff(1)` and `patch(1)` (and their `git`ified +descendants). Merge conflicts should produce files that are resolvable with any +ordinary text editor. (Even `ed(1)`, if you insist!) + +This format is generally slightly larger than Format 0, but shouldn't be +significantly so for most use cases, and this is a reasonable trade-off for +readability and diff/patch/merge-ability. + +The format could be significantly larger if files have large amounts of binary +data in xattrs, as 35 out of the 256 possible bytes require URL-encoding as a +3-byte sequence, giving a 27.3% increase (by my calculations). This clearly +isn't ideal, but this author suspects that the proportion of files with large +binary xattrs is fairly small, and this should not cause an issue in practice. + +If a user does have large amounts of binary xattr data but can't handle the 27% +size increase this format incurs, they can still use Format 0 to store it +instead. If *you* have large amounts of binary xattr data that you have to store +in git in a way that's diff/patch/merge-able - well, feel free to submit patches +for Format 2 yourself ;-) + +If you do update this format, remember to change the man page as well as this +document! I've tried to keep the info in the man page as short as possible, and +to only include what a user should need to work with the resulting files. +Extended musings and notes for implementors go here (or in the `git commit` +log :-) + + +### UTF-8 cleanliness + +Note that because bytes >= 0x80 are not required to be URL-encoded, binary xattr +data is very unlikely to be UTF-8 clean. If this is a problem for the editor +you use to resolve conflicts... I dunno. Get a better editor maybe? We could +URL-encode all high bytes, but that would triple the size of half the bytes in +binary data, and of all non-ASCII byte sequences in UTF-8 text. I suppose it +might be possible to URL-encode all sequences of high bytes that are *not* UTF-8 +clean (and that would be backwards-compatible with the existing format) but I +don't want to add that much complexity at the moment. Also, it might not be +"enough" as you'd probably want to encode non-printable UTF-8 control codes +such as RTL/LTR marks (U+200E/U+200F) to prevent the possibility of "Trojan +Source" type attacks. + +(See <https://lwn.net/Articles/874951/> for more info on "Trojan Source") |