Previously I attempted to save the cost of reading/decompressing files
by keeping a copy in memory. That resulted in too much memory pressure
so I later swapped in temporary files. This defeated the initial goal
of keeping the streams alive to some extent since the files would
be flushed to disk.
In practice since all the nupkgs we are packing also have raw files so
we weren't even saving much CPU by avoiding a second decompression.
Switch the archiver to just first index all files and save the source
information to read at a later point. After this when building the
archive we'll reopen the files/zips and copy from there.
My measurements show that this actually improve the archiving perf and
the lack of temp files means we don't hit the ulimit restriction on OSX.
This adds LZMA to the THIRD-PARTY-NOTICES list for CLI. Since we've
modified the SDK in porting it to .NET Core and minor bugfixes we've
also been advised to add the .NET Foundation copyright header.
* Updating README.md files as well as man pages
This aligns with the latest changes in the official documentation. It also aligns dotnet test readme.md to official docs and introduces a new doc in the "Documentation" directory that deals with dotnet test protocols.
Previously we'd keep track of any file that we extracted once and try
to reuse that file (by copying it) if we needed the same file later at a
different destination. The reason was that it's theoretically faster to
a file copy than a createfile and write, since the copy can happen
entirely in the kernel. In practice we were foiled by AV scanners.
This happens to be the only time during extraction where we let a file
close after writing it and then try and use it again. Sure enough on
fast machines we were seeing that as soon as we closed it MsMpEng would
map the file for exclusive access causing our copy to fail with a
sharing violation.
To fix this, I've removed the copy optimization and will just copy the
file from the in-memory archive every time.
