When indexing message attachment metadata using numeric indexes such as:
```javascript
{
conversationId: '+12223334455',
received_at: 123,
attachments: […],
numAttachments: 2,
},
{
conversationId: '+12223334455',
received_at: 456,
attachments: [],
numAttachments: 0,
}
{
conversationId: '+12223334455',
received_at: 789,
attachments: [],
numAttachments: 1,
}
```
It creates an index as follows:
```
[conversationId, received_at, numAttachments]
['+12223334455', 123, 2]
['+12223334455', 456, 0]
['+12223334455', 789, 1]
```
This means a query such as…
```
lowerBound: ['+12223334455', 0, 1 ]
upperBound: ['+12223334455', Number.MAX_VALUE, Number.MAX_VALUE]
```
…will return all three original entries because they span the `received_at`
from `0` through `Number.MAX_VALUE`. One workaround is to index booleans using
`1 | undefined` where `1` is included in the index and `undefined` is not, but
that way we lose the ability to query for the `false` value. Instead, we flip
adjust the index to `[conversationId, hasAttachments, received_at]` and can
then query messages with attachments using
```
[conversationId, 1 /* hasAttachments */, 0 /* received_at */]
[conversationId, 1 /* hasAttachments */, Number.MAX_VALUE /* received_at */]
```
1. MessageReceiver always pulls down thumbnails included in quotes
2. Message.upgradeSchema has a new schema that puts all thumbnails on
disk just like happens with full attachments.
3. handleDataMessage pipes quote from dataMessage into the final message
destined for the database
Allows errors to be formatted and sanitized for logging. Removes sensitive paths
such as the app root directory.
Ideally, this module would be called singular `Error` but that is already a
global name. Using `Errors` plural is similar to Java convention for utilities
such as `Arrays`, `Collections`, `Files`, etc. See:
https://stackoverflow.com/a/11673838
As a user, when I receive a file attachment, I want to have confidence that the
filename I see in the Signal Desktop app is the same as it will be on disk.
To prevent user confusion when receiving files with Unicode order override
characters, e.g. `test<LTRO>fig.exe` appearing as `testexe.gif`, we replace all
occurrences of order overrides (`U+202D` and `U+202E`) with `U+FFFD`.
**Changes**
- [x] Bump `Attachment` `schemaVersion` to 2.
- [x] Replace all Unicode order overrides in `attachment.filename`:
`Attachment.replaceUnicodeOrderOverrides`.
- [x] Add tests for existing `Attachment.upgradeSchema`
- [x] Add tests for existing `Attachment.withSchemaVersion`
- [x] Add tests for `Attachment.replaceUnicodeOrderOverrides` positives.
- [x] Add `testcheck` generative property-based testing library
(based on QuickCheck) to ensure valid filenames are preserved.
---
commit 855bdbc7e647e44f73b9e1f5e6d64f734c61169a
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Thu Feb 22 13:02:01 2018 -0500
Log error stack in case of error
commit 6e053ed66aee136f186568fa88aacd4814b2ab07
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Thu Feb 22 12:30:28 2018 -0500
Improve `upgradeStep` error handling
commit 8c226a2523b701cb578b2137832c3eaf3475bb2b
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Thu Feb 22 12:30:08 2018 -0500
Check for expected version before upgrade
Prevents out of order upgrade steps.
commit 28b0675591e782169128f75429b7bab2a22307fa
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Thu Feb 22 12:29:52 2018 -0500
Reject invalid attachments
commit 41f4f457dae9416dae66dc2fa2079483d1f127a9
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Thu Feb 22 12:29:36 2018 -0500
Fix upgrade pipeline order
commit 3935629e91c49b8d96c1e02bd37b1b31d1180720
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Thu Feb 22 12:28:25 2018 -0500
Avoid `_.isPlainObject`
Attachments are deserialized from a protocol buffer and can have a
non-plain-object constructor.
commit 39f6e7f622ff4885e2ccafa354e0edb5864c55d8
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Thu Feb 22 12:19:07 2018 -0500
Define basic attachment validity
commit adcf7e3243cd90866cc35990c558ff7829019037
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Thu Feb 22 12:18:54 2018 -0500
Add tests for attachment upgrade pipeline
commit 82fc4644d7e654eea9f348518b086497be2b0cb4
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Wed Feb 21 12:20:24 2018 -0500
Favor `async` / `await` over `then`
commit 8fe49e3c40e78ced0b8f2eb0b678f4bae842855d
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Wed Feb 21 12:19:59 2018 -0500
Add `eslint-more` plugin
This will enable us to disallow `then` in favor of `async` / `await`.
commit 020beefb25f508ae96cf3fc099599fbbca98802b
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Wed Feb 21 11:31:49 2018 -0500
Remove unnecessary `async` modifiers
commit 177090c5f5ad9836f0ca0a5c2f298779519e3692
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Wed Feb 21 11:30:55 2018 -0500
Document `operator-linebreak` ESLint rule
commit 25622b7c59291cb672ae057c47e7327a564cca40
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Wed Feb 21 11:14:15 2018 -0500
Prefix internal function with `_`
commit 6aa3cf5098df71e9b710064739ec49d74f81b7bf
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 19:00:07 2018 -0500
Replace all Unicode order override occurrences
commit fd6e23b0a519bce3c12c5b9ac676bcd198034fed
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 17:48:41 2018 -0500
Whitelist `testcheck` `check` and `gen` globals
commit 400bae9fac5078821813bc0ca17a5d7a72900161
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 17:46:57 2018 -0500
🎨 Fix lint errors
commit da53d3960aa7aa36b7cc1fcff414c9e929c0d9fc
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 17:42:42 2018 -0500
Add tests for `Attachment.withSchemaVersion`
commit ec203444239d9e3c443ba88cab7ef4672151072d
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 17:42:17 2018 -0500
Add test for `Attachment.upgradeSchema`
commit 4540d5bdf7a4279f49d2e4c6ee03f47b93df46bf
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 17:05:29 2018 -0500
Rename `setSchemaVersion` --> `withSchemaVersion`
Put the schema version first for better readability.
commit e379cf919feda31d1fa96d406c30fd38e159a11d
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 17:03:22 2018 -0500
Add filename sanitization to upgrade pipeline
commit 1e344a0d15926fc3e17be20cd90bfa882b65f337
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 17:01:55 2018 -0500
Test that we preserve non-suspicious filenames
commit a2452bfc98f93f82bed48b438757af2e66a6af82
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 17:00:56 2018 -0500
Add `testcheck` dependency
Allows for generative property-based testing similar to Haskell’s QuickCheck.
See: https://medium.com/javascript-inside/f91432247c27
commit ceb5bfd2484a77689fdb8e9edd18d4a7b093a486
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 16:15:33 2018 -0500
Replace Unicode order override characters
Prevents users from being tricked into clicking a file named `testexe.fig`
that appears as `testexe.gif` due to a Unicode order override character.
See:
- http://unicode.org/reports/tr36/#Bidirectional_Text_Spoofing
- https://krebsonsecurity.com/2011/09/right-to-left-override-aids-email-attacks/
commit bc605afb1c6af3a5ebc31a4c1523ff170eb96ffe
Author: Daniel Gasienica <daniel@gasienica.ch>
Date: Fri Feb 16 16:12:29 2018 -0500
Remove `CURRENT_PROCESS_VERSION`
Reintroduce this whenever we need it. We currently only deal with schema version
numbers within this module.
