Home » Linux » Debian » Sylvain Beucler: dot-zed archive file format

Sylvain Beucler: dot-zed archive file format

TL,DR: I reverse-engineered the .zed encrypted archive format.
Following a clean-room design, I’m providing a description that can be implemented by a third-party.
Interested? :)

(reference version at: https://www.beuc.net/zed/)

.zed archive file format

Introduction

Archives with the .zed extension are conceptually similar to an
encrypted .zip file.

In addition to a specific format, .zed files support multiple users:
files are encrypted using the archive master key, which itself is
encrypted for each user and/or authentication method (password, RSA
key through certificate or PKCS#11 token). Metadata such as filenames
is partially encrypted.

.zed archives are used as stand-alone or attached to e-mails with the
help of a MS Outlook plugin. A variant, which is not covered here,
can encrypt/decrypt MS Windows folders on the fly like
ecryptfs.

In the spirit of academic and independent research this document
provides a description of the file format and encryption algorithms
for this encrypted file archive.

See the conventions section for
conventions and acronyms used in this document.

Structure overview

The .zed file format is composed of several layers.

  • The main container is using the (MS-CFB), which is notably used by MS
    Office 97-2003 .doc files. It contains several streams:

    • Metadata stream: in OLE Property Set format
      (MS-OLEPS), contains 2 blobs in a specific Type-Length-Value
      (TLV) format:

      • _ctlfile: global archive properties and access list
        It is obfuscated by means of static-key AES encryption.
        The properties include archive initial filename and a global IV.
        A global encryption key is itself encrypted in each user entry.

      • _catalog: file list
        Contains each file metadata indexed with a 15-bytes identifier.
        Directories are supported.
        Full filename is encrypted using AES.
        File extension is (redundantly) stored in clear,
        and so are file metadata such as modification time.

    • Each file in the archive compressed with zlib and encrypted with
      the standard AES algorithm, in a separate stream.
      Several encryption schemes and key sizes
      are supported.
      The file stream is split in chunks of 512
      bytes, individually encrypted.

    • Optional streams, contain additional metadata as well as pictures to
      display in the application background (“watermarks”). They are not
      discussed here.

Or as a diagram:

+----------------------------------------------------------------------------------------------------+
| .zed archive (MS-CBF)                                                                              |
|                                                                                                    |
|  stream #1                         stream #2                       stream #3...                    |
| +------------------------------+  +---------------------------+  +---------------------------+     |
| | metadata (MS-OLEPS)          |  | encryption (AES)          |  | encryption (AES)          |     |
| |                              |  | 512-bytes chunks          |  | 512-bytes chunks          |     |
| | +--------------------------+ |  |                           |  |                           |     |
| | | obfuscation (static key) | |  | +-----------------------+ |  | +-----------------------+ |     |
| | | +----------------------+ | |  |-| compression (zlib)    |-|  |-| compression (zlib)    |-|     |
| | | |_ctlfile (TLV)        | | |  | |                       | |  | |                       | | ... |
| | | +----------------------+ | |  | | +---------------+     | |  | | +---------------+     | |     | 
| | +--------------------------+ |  | | | file contents |     | |  | | | file contents |     | |     |
| |                              |  | | |               |     | |  | | |               |     | |     |
| | +--------------------------+ |  |-| +---------------+     |-|  |-| +---------------+     |-|     |
| | | _catalog (TLV)           | |  | |                       | |  | |                       | |     |
| | +--------------------------+ |  | +-----------------------+ |  | +-----------------------+ |     |
| +------------------------------+  +---------------------------+  +---------------------------+     |
+----------------------------------------------------------------------------------------------------+

Encryption schemes

Several AES key sizes are supported, such as 128 and 256 bits.

The Cipher Block Chaining (CBC) block cipher mode of operation is used
to decrypt multiple AES 16-byte blocks, which means an initialisation
vector (IV) is stored in clear along with the ciphertext.

All filenames and file contents are encrypted using the same
encryption mode, key and IV (e.g. if you remove and re-add a file in
the archive, the resulting stream will be identical).

No cleartext padding is used during encryption; instead, several
end-of-stream handlers are available, so the ciphertext has exactly
the size of the cleartext (e.g. the size of the compressed file).

The following variants were identified in the ‘encryption_mode’
field.

STREAM

This is the end-of-stream handler for:

  • obfuscated metadata encrypted with static AES key
  • filenames and files in archives with ‘encryption_mode’ set to “AES-CBC-STREAM”
  • any AES ciphertext of size < 16 bytes, regardless of encryption mode

This end-of-stream handler is apparently specific to the .zed format,
and applied when the cleartext’s does not end on a 16-byte boundary ;
in this case special processing is performed on the last partial
16-byte block.

The encryption and decryption phases are identical: let’s assume the
last partial block of cleartext (for encryption) or ciphertext (for
decryption) was appended after all the complete 16-byte blocks of
ciphertext:

  • the second-to-last block of the ciphertext is encrypted in AES-ECB
    mode (i.e. block cipher encryption only, without XORing with the IV)

  • then XOR-ed with the last partial block (hence truncated to the
    length of the partial block)

In either case, if the full ciphertext is less then one AES block (<
16 bytes), then the IV is used instead of the second-to-last block.

CTS

CTS or CipherText Stealing is the end-of-stream handler for:

  • filenames and files in archives with ‘encryption_mode’ set to “AES-CBC-CTS”.
    • exception: if the size of the ciphertext is < 16 bytes, then
      “STREAM” is used instead.

It matches the CBC-CS3 variant as described in Recommendation for
Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing
for CBC Mode
.

Empty cleartext

Since empty filenames or metadata are invalid, and since all files are
compressed (resulting in a minimum 8-byte zlib cleartext), no empty
cleartext was encrypted in the archive.

metadata stream

It is named 05356861616161716149656b7a6565636e576a33317a7868304e63
(hexadecimal), i.e. the character with code 5 followed by
‘5haaaaqaIekzeecnWj31zxh0Nc’ (ASCII).

The format used is OLE Property Set (MS-OLEPS).

It introduces 2 property names “_ctlfile” (index 3) and “_catalog”
(index 4), and 2 instances of said properties each containing an
application-specific VT_BLOB (type 0x0041).

_ctlfile: obfuscated global properties and access list

This subpart is stored under index 3 (“_ctlfile”) of the MS-OLEPS
metadata.

It consists of:

  • static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by 0100 (hexadecimal) (18 bytes total)
  • 16-byte IV
  • ciphertext
  • 1 uint32be representing the length of all the above
  • static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by “ZoneCentral (R)” (ASCII) and a NUL byte (32 bytes total)

The ciphertext is encrypted with AES-CBC “STREAM” mode using 128-bit
static key 37F13CF81C780AF26B6A52654F794AEF (hexadecimal) and the
prepended IV so as to obfuscate the access list. The ciphertext is
continuous and not split in chunks (unlike files), even when it is
larger than 512 bytes.

The decrypted text contain properties in a TLV format
as described in _ctlfile TLV:

  • global archive properties as a ‘fileprops’ structure,

  • extra archive properties as a ‘archive_extraprops’ structure

  • users access list as a series of ‘passworduser’ and ‘rsauser
    entries.

Archives may include “mandatory” users that cannot be removed. They
are typically used to add an enterprise wide recovery RSA key to all
archives. Extreme care must be taken to protect these key, as it can
decrypt all past archives generated from within that company.

_catalog: file list

This subpart is stored under index 4 (“_catalog”) of the MS-OLEPS
metadata.

It contains a series of ‘fileprops’ TLV structures,
one for each file or directory.

The file hierarchy can be reconstructed by checking the ‘parent_id’
field of each file entry. If ‘parent_id’ is 0 then the file is
located at the top-level of the hierarchy, otherwise it’s located
under the directory with the matching ‘file_id’.

TLV format

This format is a series of fields :

  • 4 bytes for Type (specified as a 4-bytes hexadecimal below)
  • 4 bytes for value Length (uint32be)
  • Value

Value semantics depend on its Type. It may contain an uint32be
integer, a UTF-16LE string, a character sequence, or an inner TLV
structure.

Unless otherwise noted, TLV structures appear once.

Some fields are optional and may not be present at all
(e.g. ‘archive_createdwith’).

Some fields are unique within a structure (e.g. ‘files_iv’), other
may be repeated within a structure to form a list (e.g. ‘fileprops’
and ‘passworduser’).

The following top-level types that have been identified, and detailed
in the next sections:

  • 80110600: fileprops, used for the file list as well as for the
    global archive properties
  • 001b0600: archive_extraprops
  • 80140600: accesslist

Some additional unidentified types may be present.

_ctlfile TLV

  • 80110600: fileprops (TLV structure): global archive properties
    • 00230400: archive_pathname (UTF-16LE string): initial archive filename (past versions also leaked the full pathname of the initial archive)
    • 80270200: encryption_mode (utf32be): 103 for “AES-CBC-STREAM”, 104 for “AES-CBC-CTS”
    • 80260200: encryption_strength (utf32be): AES key size, in bytes (e.g. 32 means AES with a 256-bit key)
    • 80280500: files_iv (sequence of bytes): global IV for all filenames and file contents
  • 001b0600: archive_extraprops (TLV structure): additionnal archive properties (optional)
    • 00c40500: archive_creationtime (FILETIME): date and time when archive was initially created (optional)
    • 00c00400: archive_createdwith (UTF-16LE string): uuid-like structure describing the application that initialized the archive (optional)
      {00000188-1000-3CA8-8868-36F59DEFD14D} is Zed! Free 1.0.188.
  • 80140600: accesslist (TLV structure): describe the users, their key encryption and their permissions
    • 80610600: passworduser (TLV structure): user identified by password (0 or more)
    • 80620600: rsauser (TLV structure): user identified by RSA key (via file or PKCS#11 token) (0 or more)
    • Fields common to passworduser and rsauser:
      • 80710400: login (UTF-16LE string): user name
      • 80720300: login_md5 (sequence of bytes): used by the application to search for a user name
      • 807e0100: priv1 (uchar): user privileges; present and set to 1 when user is admin (optional)
      • 00830200: priv2 (uint32be): user privileges; present and set to 2 when user is admin, present and set to 5 when user is a marked as mandatory, e.g. for recovery keys (optional)
      • 80740500: files_key_ciphertext (sequence of bytes): the archive encryption key, itself encrypted
      • 00840500: user_creationtime (FILETIME): date and time when the user was added to the archive
    • passworduser-specific fields:
      • 80760500: pbe_salt (sequence of bytes): salt for PBE
      • 80770200: pbe_iter (uint32be): number of iterations for PBE
      • 80780200: pkcs12_hashfunc (uint32be): hash function used for PBE and PBA key derivation
      • 80790500: pba_checksum (sequence of bytes): password derived with PBA to check for password validity
      • 807a0500: pba_salt (sequence of bytes): salt for PBA
      • 807b0200: pba_iter (uint32be): number of iterations for PBA
    • rsauser-specific fields:
      • 807d0500: certificate (sequence of bytes): user X509 certificate in DER format

_catalog TLV

  • 80110600: fileprops (TLV structure): describe the archive files (0 or more)
    • 80300500: file_id (sequence of bytes): a 16-byte unique identifier
    • 80310400: filename_halfanon (UTF-16LE string): half-anonymized filename, e.g. File1.txt (leaking filename extension)
    • 00380500: filename_ciphertext (sequence of bytes): encrypted filename; may have a trailing NUL byte once decrypted
    • 80330500: file_size (uint64le): decompressed file size in bytes
    • 80340500: file_creationtime (FILETIME): file creation date and time
    • 80350500: file_lastwritetime (FILETIME): file last modification date and time
    • 80360500: file_lastaccesstime (FILETIME): file last access date and time
    • 00370500: parent_directory_id (sequence of bytes): file_id of the parent directory, 0 is top-level
    • 80320100: is_dir (uint32be): 1 if entry is directory (optional)

Decrypting the archive AES key

rsauser

The user accessing the archive will be authenticated by comparing
his/her X509 certificate with the one stored in the ‘certificate’
field using DER format.

The ‘files_key_ciphertext’ field is then decrypted using the PKCS#1
v1.5 encryption mechanism, with the private key that matches the user
certificate.

passworduser

An intermediary user key, a user IV and an integrity checksum will be
derived from the user password, using the deprecated PKCS#12 method as
described at rfc7292 appendix
B
.

Note: this is not PKCS#5 (nor PBKDF1/PBKDF2), this is an incompatible
method from PKCS#12 that notably does not use HMAC.

The ‘pkcs12_hashfunc’ field defines the underlying hash function. The
following values have been identified:

  • 21: SHA-1
  • 22: SHA-256

PBA – Password-based authentication

The user accessing the archive will be authenticated by deriving an
8-byte sequence from his/her password.

The parameters for the derivation function are:

  • ID: 3
  • ‘pba_salt’: the salt, typically an 8-byte random sequence
  • ‘pba_iter’: the iteration count, typically 200000

The derivation is checked against ‘pba_checksum’.

PBE – Password-based encryption

Once the user is identified, 2 new values are derived from the
password with different parameters to produce the IV and the key
decryption key, with the same hash function:

  • ‘pbe_salt’: the salt, typically an 8-bytes random sequence
  • ‘pbe_iter’: the iteration count, typically 100000

The parameters specific to user key are:

  • ID: 1
  • size: 32

The user key needs to be truncated to a length of
‘encryption_strength’, as specified in bytes in the archive
properties.

The parameters specific to user IV are:

  • ID: 2
  • size: 16

Once the key decryption key and the IV are derived,
‘files_key_ciphertext’ is decrypted using AES CBC, with
PKCS#7 padding.

Identifying file streams

The name of the MS-CFB stream is derived by shuffling the bytes from
the ‘file_id’ field and then encoding the result as hexadecimal.

The reordering is:

Initial  offset: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Shuffled offset: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15

The 16th byte is usually a NUL byte, hence the stream identifier is a
30-character-long string.

Decrypting files

The compressed stream is split in chunks of 512 bytes, each of them
encrypted separately using AES CBS and the global archive encryption
scheme. Decryption uses the global AES key (retrieved using the user
credentials), and the global IV (retrieved from the deobfuscated
archive metadata).

The IV for each chunk is computed by:

  • expressing the current chunk number as little endian on 16 bytes
  • XORing it with the global IV
  • encrypting with the global AES key in ECB mode (without IV).

Each chunk is an independent stream and the decryption process
involves end-of-stream handling even if this is not the end of the
actual file. This is particularly important for the CTS handler.

Note: this is not to be confused with CTR block cipher mode of
operation with operates differently and requires a nonce.

Decompressing files

Compressed streams are zlib stream with default compression options
and can be decompressed following the
zlib format.

Test cases

Excluded for brevity, cf. https://www.beuc.net/zed/#test-cases.

Conventions and references

Feedback

Feel free to send comments at beuc@beuc.net.
If you have .zed files that you think are not covered by this
document, please send them as well (replace sensitive files with other
ones). The author’s GPG key can be found at
8FF1CB6E8D89059F.

Copyright (C) 2017 Sylvain Beucler

Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved. This file is offered as-is,
without any warranty.

Source: Debian Planet

Facebook Comments