mirrors/signal-desktop
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
4458 commits 1 branch 101 tags 16 GiB
Commit graph

255 commits

Author SHA1 Message Date
Scott Nonnenberg
054d3887a1
Quotes: The full pipeline into the database 
1. MessageReceiver always pulls down thumbnails included in quotes
2. Message.upgradeSchema has a new schema that puts all thumbnails on
   disk just like happens with full attachments.
3. handleDataMessage pipes quote from dataMessage into the final message
   destined for the database
 2018-04-13 18:05:45 -07:00
Scott Nonnenberg
21bf02c94d
Fixed examples in Quote.md, rough Android visuals 2018-04-13 18:05:44 -07:00
Daniel Gasienica
d7b845326d ESLint auto-fix link_text 2018-04-11 16:36:42 -04:00
Daniel Gasienica
9d41b86162 Remove escaping from linkText 
We leverage jQuery’s HTML escaping in `$.html(…)`.
 2018-04-11 16:36:42 -04:00
Daniel Gasienica
f04c65088b Fork link-text module 2018-04-11 16:36:42 -04:00
Daniel Gasienica
24f4ad53bc Use single quotes for identifiers 2018-04-11 15:54:32 -04:00
Daniel Gasienica
432a6ebd7f Redact file paths with escaped slashes 2018-04-11 15:30:59 -04:00
Daniel Gasienica
a8a7525609 Redact stack traces with forward and backslashes 2018-04-11 15:30:59 -04:00
Daniel Gasienica
b0da7d965e Redact URL encoded file paths in stack traces 2018-04-11 15:30:59 -04:00
Daniel Gasienica
f2c9ccae90 Prefer is.* over Lodash is* functions 2018-04-11 15:30:59 -04:00
Daniel Gasienica
2ee78ec556 Prefer path over Path 2018-04-11 15:30:59 -04:00
Scott Nonnenberg
c6c3b65bbc
Introduce React, TypeScript, TSLint and React-StyleGuidist (#2219) 
Quite a bit of change here.

First, the basics:

- New dependencies were added: react, typescript, tslint, and react-styleguidist
- A new npm script: transpile. It uses typescript to process .tsx files in js/react, putting .js files next to the original file. It's part of the watch functionality of grunt dev as well as the default task run with just grunt (used to build the app prior to release). A lighter-weight to get watch behavior when just working on React components is to run yarn transpile --watch.
- yarn run clean-transpile will remove generated .js files


Style guide via react-styleguidist. Example site: https://react-styleguidist.js.org/examples/basic/

- Start with yarn styleguide
- Component.md files right next to the .tsx file
- jsdoc-style comments are picked up and added to the generated part of the styleguide - the overall summary and a table listing methods and properties of the component
- It has hot-reloading!
- It uses webpack, which means that our app now pulls in webpack though we don't use it to generate anything for the production app.
- I did a bunch of work to enable the use of Backbone views in this context, which will allow us to move smoothly from the old world to the new. First, add all the permutations in the old way, and then slowly start to re-render those same views with React.

A bit of dependency cleanup to enable use in React components:

- moment was moved from our Bower dependencies to our npm dependencies, so it can be used in React components not running in a browser window.
- i18n was moved into the new commonjs format, so it can be used in React components even if window is not available.

Lastly, a bit of Gruntfile cleanup:

- Removal of Chrome App-era modifications of background.js
- Make jshint/jscs watch more targeted, since more and more we'll be using other tools
 2018-04-06 08:13:00 -07:00
Scott Nonnenberg
50d4dbaae7
Remove 'use strict' from i18n.js 2018-04-05 13:42:41 -07:00
Scott Nonnenberg
76be13552f
js/i18n.js -> js/modules/i18n.js 2018-04-05 12:51:29 -07:00
Daniel Gasienica
195e9b777c Rename willHaveAttachmentsSavedOnFileSystemDuringUpgrade 
New name: `willAttachmentsGoToFileSystemOnUpgrade`
 2018-04-04 19:11:46 -04:00
Daniel Gasienica
4c4443390a Rename createImporter to createAttachmentDataWriter 2018-04-04 19:11:46 -04:00
Daniel Gasienica
5ea1a305a6 Clarify error origin 2018-04-04 18:41:50 -04:00
Daniel Gasienica
9dcb64b53c Initialize schema version upon import 2018-04-04 18:41:50 -04:00
Daniel Gasienica
cfad983d08 Consistently use ciphertext instead of encrypted 2018-04-04 18:41:50 -04:00
Daniel Gasienica
a61c905146 Impliclity decrypt data if key is present 2018-04-04 18:41:50 -04:00
Daniel Gasienica
94c75762b5 Add Message.createAttachmentLoader 2018-04-04 18:41:50 -04:00
Daniel Gasienica
43ab320d31 Load attachments before writing them upon export 2018-04-04 18:41:50 -04:00
Daniel Gasienica
7eeabb5b1a Add precondition for writeAttachment 2018-04-04 18:41:50 -04:00
Daniel Gasienica
0faaf52b71 Import and upgrade messages upon save 2018-04-04 18:41:50 -04:00
Daniel Gasienica
cf7355bd2f Rename backupToDirectory to exportToDirectory 
For symmetry reasons.
 2018-04-04 18:41:50 -04:00
Daniel Gasienica
bf67254cc5 Add Message.createImporter 2018-04-04 18:41:50 -04:00
Daniel Gasienica
efd673083d Rename writeAttachmentData to writeNewAttachmentData 2018-04-04 18:41:50 -04:00
Daniel Gasienica
ce8fe0d345 Add Signal.Startup module 2018-04-04 18:10:52 -04:00
Daniel Gasienica
8474f3cf7f Use create* prefix to clarify curried functions 2018-04-03 15:25:24 -04:00
Daniel Gasienica
12cdeee7ec Rename count to numMessagesPerBatch 2018-04-03 14:43:17 -04:00
Daniel Gasienica
38ac9972e8 Improve termination condition 2018-04-03 13:16:05 -04:00
Daniel Gasienica
5b1a0c45df Replace getAll with cursor 
This reduces the likelihood of IPC message size error.
 2018-04-03 13:16:05 -04:00
Daniel Gasienica
ed77378293 Fix bug in termination condition 2018-04-03 12:30:13 -04:00
Daniel Gasienica
052fb841f6 Allow database name override in migrations 2018-04-03 11:45:10 -04:00
Daniel Gasienica
1e04083813 Support database upgrades in Database.open 2018-04-03 11:44:50 -04:00
Daniel Gasienica
f50ac1c99e Reduce excessive logging about database status 2018-04-03 11:06:56 -04:00
Daniel Gasienica
77b1f3c505 Replace startup with attachment background migration 
Iterates through `Message::id` index.
 2018-04-02 20:07:07 -04:00
Daniel Gasienica
057762806e Add support for attachment background migration without index 2018-04-02 19:41:28 -04:00
Daniel Gasienica
2e4893f4c1 Clarify implications of introducing new migrations 2018-04-02 18:54:51 -04:00
Daniel Gasienica
7413b787b6 Make migrations private 2018-04-02 18:54:28 -04:00
Daniel Gasienica
b07c66eaa7 Generate attachments from local files 
This ensures our benchmarking is closer to real-world usage, e.g. images, video,
etc. that are not compressible.
 2018-04-02 17:58:37 -04:00
Daniel Gasienica
064a40544c Fix error message formatting 2018-04-02 17:42:04 -04:00
Daniel Gasienica
442f5d7295 Use done to match JavaScript iterators 
See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators
 2018-04-02 17:42:04 -04:00
Daniel Gasienica
a4f0c5242f Document why batch size is one 2018-04-02 17:42:04 -04:00
Daniel Gasienica
805031ade8 Conditionally run post-attachment migrations 
Introduce placeholder migrations for Backbone models so they never implicitly
run migrations whenever they are `fetch`ed. We prefer to run our migrations
explicitly upon app startup and then let Backbone models be (slightly) dumb(er)
models, without inadvertently triggering migrations.
 2018-04-02 17:42:04 -04:00
Daniel Gasienica
bfbeedab5c Temporarily disable post-attachment migration migrations 2018-04-02 15:26:24 -04:00
Daniel Gasienica
d9be6a0f94 Destructure Lodash requires 2018-04-02 15:12:04 -04:00
Daniel Gasienica
b7b6195cfc Extract IIFE into separate function 2018-04-02 12:17:36 -04:00
Daniel Gasienica
11f98474ba Capture how many messages we have to process 2018-04-02 12:17:36 -04:00
Daniel Gasienica
0fdc1140dd Add Database.getCount function 2018-04-02 12:17:36 -04:00
Daniel Gasienica
1f8556b049 Remove unused createCollection 2018-04-02 12:17:36 -04:00
Daniel Gasienica
c67c2a858a Remove Backbone references for attachment migration 
It has to run without any other migrations interfering.
 2018-04-02 12:17:36 -04:00
Daniel Gasienica
0c40f35623 Document disadvantage of fetching messages without index 2018-04-02 12:17:36 -04:00
Daniel Gasienica
d3c9de4712 Add debug module 
Lets us generate large conversations with large attachments.
 2018-04-02 12:17:36 -04:00
Daniel Gasienica
30037e5308 Reduce attachment migration batch size to 1 
This prevents ‘Maximum IPC message size exceeded’ due to IDB `getAll` operation.

- https://github.com/zincbase/zincdb/issues/17
- https://cs.chromium.org/chromium/src/content/browser/indexed_db/indexed_db_database.cc?q=%22Maximum+IPC+message+size+exceeded%22&sq=package:chromium&l=1160
 2018-04-02 12:17:36 -04:00
Daniel Gasienica
21147a20a0 Add sleep module 2018-04-02 12:17:36 -04:00
Daniel Gasienica
08f73b8420 Remove last processed index after attachment migration 2018-04-02 12:17:36 -04:00
Daniel Gasienica
696a144ab7 Add settings.deleteItem 2018-04-02 12:17:36 -04:00
Daniel Gasienica
5910f84af4 Remove outdated documentation 2018-04-02 12:17:36 -04:00
Daniel Gasienica
efe3cd67fc Allow attachment migration run on higher database version 2018-04-02 12:17:36 -04:00
Daniel Gasienica
921c3dba7c Skip migrations that have already been applied 2018-04-02 12:17:36 -04:00
Daniel Gasienica
417511ffd2 Add database.getVersion 2018-04-02 12:17:36 -04:00
Daniel Gasienica
c5c94bc3ab Extract getMigrationVersions 2018-04-02 12:17:36 -04:00
Daniel Gasienica
6aea36240d Rename closeDatabase to closeDatabaseConnection 2018-04-02 12:17:36 -04:00
Daniel Gasienica
016432826b Extract database and settings modules 2018-04-02 12:17:36 -04:00
Daniel Gasienica
5bea894abd Close database connection via Backbone IDB adapter 2018-04-02 12:17:36 -04:00
Daniel Gasienica
f50e9ae364 Log closing connection of database 2018-04-02 12:17:36 -04:00
Daniel Gasienica
4ff8bc3357 Use camelCase for non-constructors 2018-04-02 12:17:36 -04:00
Daniel Gasienica
3720c3f3bb Improve log message 2018-04-02 12:17:36 -04:00
Daniel Gasienica
ce5b450fdb Log targetSchemaVersion 2018-04-02 12:17:36 -04:00
Daniel Gasienica
f7f24b5822 Log total number of processed messages 2018-04-02 12:17:36 -04:00
Daniel Gasienica
eca930770c Remove hard-coded database connection settings 2018-04-02 12:17:36 -04:00
Daniel Gasienica
1df6dc8378 Abort processing if there are no more messages 2018-04-02 12:17:36 -04:00
Daniel Gasienica
d5d0eabdfd Remove usage of storage module 2018-04-02 12:17:36 -04:00
Daniel Gasienica
8966e80284 Improve identifier names 2018-04-02 12:17:36 -04:00
Daniel Gasienica
3e2d575506 Document MessageDataMigrator module design 2018-04-02 12:17:36 -04:00
Daniel Gasienica
3c57dbfb56 Extract NUM_MESSAGES_PER_BATCH 2018-04-02 12:17:36 -04:00
Daniel Gasienica
85788d3c4a Match items storage format to Backbone adapter 2018-04-02 12:17:36 -04:00
Daniel Gasienica
7de7fcf561 Avoid no-param-reassign violation 2018-04-02 12:17:36 -04:00
Daniel Gasienica
070235b59b Implement MessageDataMigrator.processAll 
Upgrades schema of all messags upon startup.
 2018-04-02 12:17:36 -04:00
Daniel Gasienica
178a3cc262 Reduce work for verifying transaction completion 2018-04-02 12:17:36 -04:00
Daniel Gasienica
b6e978f74c Implement MessagesDataMigrator.processAll 2018-04-02 12:17:36 -04:00
Daniel Gasienica
457bf7ab9d Add createCollection function 2018-04-02 12:17:36 -04:00
Daniel Gasienica
8ea257ad4d Use double quotes for identifiers in error messages 2018-04-02 12:17:36 -04:00
Daniel Gasienica
172616ca4f Add log message for dummy migration 18 2018-04-02 12:17:36 -04:00
Daniel Gasienica
579b01283e Replace wrapDeferred with deferredToPromise 2018-04-02 12:17:36 -04:00
Daniel Gasienica
40c40c800a Prefer exports 2018-04-02 12:17:36 -04:00
Daniel Gasienica
da144edc56 Manually close database connection after migration 2018-04-02 12:17:36 -04:00
Daniel Gasienica
106ce21c49 Remove redundant log message 2018-04-02 12:17:36 -04:00
Daniel Gasienica
fcd30cd919 Close database after migration 
This is not 100% reliable as database connections are closed in a separate
thread according to the documentation:
- https://developer.mozilla.org/en-US/docs/Web/API/IDBDatabase/close
- https://stackoverflow.com/a/18639298
- 80c7a06d5c/backbone-indexeddb.js (L558-L565)
 2018-04-02 12:17:36 -04:00
Daniel Gasienica
c765422fa1 Extract deferredToPromise 2018-04-02 12:17:36 -04:00
Daniel Gasienica
e2f1339ab9 Explicitly run post-attachment migrations 2018-04-02 12:17:36 -04:00
Daniel Gasienica
d7c8d33edb Extract runMigrations 2018-04-02 12:17:36 -04:00
Daniel Gasienica
d16178638e Split database migrations into pre- and post-attachment migration 
- Run light-weight migrations before attachment migration.
- Run regular migrations after attachments have been moved to disk.
 2018-04-02 12:17:36 -04:00
Daniel Gasienica
90de9d85a3 Remove type annotation 2018-03-26 16:56:37 -04:00
Daniel Gasienica
4c10fcfa17 Reduce threshold until optimization appears to 1s 2018-03-26 16:56:37 -04:00
Daniel Gasienica
add19aa732 Add initialization view 2018-03-26 16:56:37 -04:00
Daniel Gasienica
9b112489cd Upgrade 2 messages every 30 seconds 
This results in 4 messages per minute or 240 messages per hour.
 2018-03-26 16:49:21 -04:00
Daniel Gasienica
7c2fde4433 Add Signal.Workflow namespace 2018-03-26 16:49:21 -04:00
Daniel Gasienica
4a7cc22de3 🎨 Use window.requestIdleCallback 2018-03-26 16:49:21 -04:00
Daniel Gasienica
4651cbc2eb Improve `Message.upgradeSchema preconditions 2018-03-26 16:49:21 -04:00
Daniel Gasienica
ca28f3030b Add MessageMigration module 2018-03-26 16:49:21 -04:00
Daniel Gasienica
8e762797c0 Remove attachment validation 
Turns out `fileName` is optional and maybe even `contentType`. We should revisit
this at a later point in time.
 2018-03-26 16:49:21 -04:00
Daniel Gasienica
09ee2d4ea2 Switch to using requestIdleCallback 
Example:
https://developer.mozilla.org/en-US/docs/Web/API/Background_Tasks_API#Example
 2018-03-26 16:49:21 -04:00
Daniel Gasienica
d13668544d Rename IdleListener to IdleDetector 2018-03-26 16:49:21 -04:00
Daniel Gasienica
3140e4d66d Implement basic IdleListener 2018-03-26 16:49:21 -04:00
Daniel Gasienica
867f73b80a Use double quotes for identifiers in error messages 2018-03-26 16:38:34 -04:00
Daniel Gasienica
70cdd2b350 Add stringToArrayBuffer utility 
This way we can avoid an extra dependency.
 2018-03-26 16:38:34 -04:00
Daniel Gasienica
2ad0fd7db6 Use Lodash omit instead of delete 2018-03-26 16:38:34 -04:00
Daniel Gasienica
279b3f81c7 Add Attachment.deleteData 2018-03-26 16:38:34 -04:00
Daniel Gasienica
9a540d6d18 Load attachment data before sending message 2018-03-26 16:38:34 -04:00
Daniel Gasienica
6355c54114 Return relative path from writeAttachmentData 
This will make our app more robust to changes in location on the file system.
 2018-03-26 16:38:34 -04:00
Daniel Gasienica
2cd3d5ac16 Add Message schema version 3 2018-03-26 16:38:34 -04:00
Daniel Gasienica
aa82a2a5fb Add context to Message.upgradeSchema 
Allows us to pass in Electron/Node.js specific functions. This allows better
unit testing in Mocha where we don’t have access to Electron APIs.
 2018-03-26 16:38:34 -04:00
Daniel Gasienica
ebe2a769c9 Add Attachment.migrateDataToFileSystem 2018-03-26 16:38:34 -04:00
Scott Nonnenberg
cea42bde7d
Encryption support for backup and restore 
Also moved to the _ prefix in backup.js for all private methods exported
for testing.
 2018-03-20 11:53:54 -07:00
Scott Nonnenberg
6d8f4b7b6e
Backup: zipped messages.json, flat attachments dir 
Backup creates, in a target directory:
  - An attachments folder, with all attachments, each named for their
    parent message's id - a GUID. If there is more than one attachment
    in a given message,  each attachment beyond the first will end with
    '-N', zero-indexed.
  - A file named messages.zip. It contains exactly what went to disk in
    the original export code, but zipped up.

Export is now only 'light,' and in this new messages.zip format.

Import supports both the new format and the old format. If the target
directory has a messages.zip file, we'll treat it as the new format.

Next up: Encrypting attachments and the messages.zip!
 2018-03-20 11:53:22 -07:00
Scott Nonnenberg
153a22f46b
Exclude unnecessary data in export: profiles, replayable errors 2018-03-20 11:53:22 -07:00
Scott Nonnenberg
c0c758d459
Backup: All exports are now 'light' exports 2018-03-20 11:53:21 -07:00
Scott Nonnenberg
e583434366
Refactor: Move Backup under window.Signal 2018-03-20 11:53:19 -07:00
Daniel Gasienica
c88381efe3 Use async / await to improve readability 2018-03-19 19:27:59 -04:00
Daniel Gasienica
2642844c27 Rewrite migration 17 without idb 
We ran into issues when doing async operations inside of an IndexedDB
`onupgradeneeded` handler. The errors were ‘The transaction is not active’ or
‘Transaction has finished’. The following documentation confirmed that
transactions are committed/terminated when control returns to the event loop:

Spec
- https://www.w3.org/TR/IndexedDB/#transaction-lifetime-concept
- https://www.w3.org/TR/IndexedDB/#upgrade-transaction-construct

Stack Overflow
- https://stackoverflow.com/a/11059085
- https://stackoverflow.com/a/27338944

Since the initial database migration is so critical, I decided to avoid `idb`
with promise support for IndexedDB for now, but will reconsider using it for
other tasks in the future to improve readability of IndexedDB code.
 2018-03-19 19:27:59 -04:00
Daniel Gasienica
a619d48fac Update schema version history 2018-03-19 19:27:59 -04:00
Daniel Gasienica
0e20e8e2ea Use Errors.toLogFormat 2018-03-19 19:27:59 -04:00
Daniel Gasienica
a5edbf8328 Initialize schema as first step in upgradeSchema 2018-03-19 19:27:59 -04:00
Daniel Gasienica
5d927b73e6 Use while loop for IDB cursor iteration 
Previously, I messily combined promises and callbacks because I thought we
were affected by the microtask issue:
https://github.com/gasi/idb#iteratecursor--iteratekeycursor

ESLint’s `more/no-then` encouraged me to revisit this and it works as expected.
 2018-03-19 19:27:59 -04:00
Daniel Gasienica
182e6ffe10 Add version 17 migration 2018-03-19 19:27:59 -04:00
Daniel Gasienica
752c8f97e6 🎨 Format ternaries 2018-03-19 19:27:59 -04:00
Daniel Gasienica
8dfaa5619f Prefix private functions with underscore 2018-03-19 19:27:59 -04:00
Daniel Gasienica
ed336d31d7 Move schema versioning from Attachment to Message 2018-03-19 19:27:59 -04:00
Daniel Gasienica
b9e9f5e19a 🎨 withInheritedSchemaVersion to inheritSchemaVersion 2018-03-19 19:27:59 -04:00
Daniel Gasienica
df693ade7c Allow Attachment.fileName to be null 2018-03-19 19:27:59 -04:00
Daniel Gasienica
c27746b79e Add Message.withInheritedSchemaVersion 2018-03-19 19:27:59 -04:00
Daniel Gasienica
e9e46464c2 Add SchemaVersion type 2018-03-19 19:27:59 -04:00
Daniel Gasienica
add4b11df3 Skip invalid attachments and make function sync 2018-03-19 19:27:59 -04:00
Daniel Gasienica
3dfc823716 Add Attachment.removeSchemaVersion 2018-03-19 19:27:59 -04:00
Scott Nonnenberg
26c273618a
Refactor: db tasks to database.js, log delete to modules/logs.js 2018-03-14 14:42:15 -07:00
Daniel Gasienica
43b47fd349 Avoid ensure-error for privacy reasons 
Example:

```
> node
> const nonError = {foo: 'i-am-private'};
undefined

// before
> util.inspect(nonError);
'{ foo: \'i-am-private\' }'

// after
> nonError.toString()
'[object Object]'
>
```
 2018-03-08 16:23:51 -05:00
Daniel Gasienica
ea07915e6b Escape special characters in file path 2018-03-08 16:23:51 -05:00
Daniel Gasienica
44b81f68dd Remove privacy redaction from Errors.toLogFormat 2018-03-08 16:23:51 -05:00
Daniel Gasienica
49e0850fb2 Extract Privacy module 
Centralizes how we redact sensitive information.
 2018-03-08 16:23:51 -05:00
Daniel Gasienica
0e2f8a8a06 Extract Errors.redactSensitivePaths 2018-03-08 16:22:38 -05:00
Daniel Gasienica
3dffdc3f0b Rename Errors to GlobalErrors for clarity 2018-03-08 16:22:38 -05:00
Daniel Gasienica
2575196617 Extract Errors.addGlobalHandler procedure 2018-03-08 16:22:38 -05:00
Daniel Gasienica
be3e4d86c2 Add Errors.toLogFormat 
Allows errors to be formatted and sanitized for logging. Removes sensitive paths
such as the app root directory.

Ideally, this module would be called singular `Error` but that is already a
global name. Using `Errors` plural is  similar to Java convention for utilities
such as `Arrays`, `Collections`, `Files`, etc. See:
https://stackoverflow.com/a/11673838
 2018-03-08 16:22:38 -05:00
Daniel Gasienica
b049412bfd Document workaround for got FormData bug 
See: https://github.com/sindresorhus/got/pull/466
 2018-03-08 16:21:27 -05:00
Daniel Gasienica
710701d0fc Document why key comes first 2018-03-08 16:21:27 -05:00
Daniel Gasienica
52f7de6a10 Add debuglogs module 2018-03-08 16:21:27 -05:00
Daniel Gasienica
85b121aca4 Rename shouldShowAudioNotificationSetting 
Use `isAudioNotificationSupported` to make it less presentation layer specific.
 2018-03-05 10:29:11 -05:00
Daniel Gasienica
21339c6866 Add Settings type 2018-03-05 10:29:11 -05:00