[Bug-wget] [bug #51666] Please hash the hostname in ~/.wget-hsts files

[NoëlKöthe]

URL:
<http://savannah.gnu.org/bugs/?51666>

Summary: Please hash the hostname in ~/.wget-hsts files
Project: GNU Wget
Submitted by: nok
Submitted on: Sat 05 Aug 2017 04:14:03 PM CEST
Category: Feature Request
Severity: 3 - Normal
Priority: 5 - Normal
Status: None
Privacy: Public
Assigned to: None
Originator Name:
Originator Email:
Open/Closed: Open
Discussion Lock: Any
Release: trunk
Operating System: GNU/Linux
Reproducibility: Every Time
Fixed Release: None
Planned Release: None
Regression: No
Work Required: None
Patch Included: No

_______________________________________________________

Details:

Hello,

a feature request from a Debian user:

--8<--
Hi,

I recently discovered the .wget-hsts file in my home directory which is used
to persist state for HSTS:

https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security

It contains lines such as:

# HSTS 1.0 Known Hosts database for GNU Wget.
# Edit at your own risk.
# <hostname> <port> <incl. subdomains> <created> <max-age>
github.com 0 1 1450887745 31536000
ftp-master.debian.org 0 0 1472482586 15552000
diffoscope.org 0 0 1449765396 15768000
reproducible-builds.org 0 0 1471542629 15552000
www.dropbox.com 0 1 1458394011 15552000
reproducible.debian.net 0 0 1448074844 15552000
[…]

I can't help but think of ~/.ssh/known_hosts which moved to hashing the
hostname for various security/privacy concerns. Shouldn't wget make the
parallel change?
--8<--
https://bugs.debian.org/870813

Thank you.




_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?51666>

_______________________________________________
Message sent via/by Savannah
http://savannah.gnu.org/

[Tim Ruehsen]

Follow-up Comment #1, bug #51666 (project wget):

The files are read/write only to the user (in case you didn't protect your
home directory appropriately).

If you are still concerned, what about
~/.mozilla/firefox/*.default/SiteSecurityServiceState.txt and likely hundreds
or thousands of other files containing potential private information ?

Not thinking of your private keys lying around in .ssh/.

And even file dates and sizes are potential private data leaks.

While we could hash anything, it would be way safer for you to protect your
complete home directory.

We could think about an option, just in case you want to keep your HSTS cache
in a public space, e.g. /var. But we already think of using SQLite for storing
and maybe sharing with Firefox.


_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?51666>

_______________________________________________
Message sent via/by Savannah
http://savannah.gnu.org/

[Ander Juaristi]

Follow-up Comment #2, bug #51666 (project wget):

I'm not generally against these kind of small tweaks that don't harm and
slightly improve user's privacy.

If Firefox doesn't do it, we don't care: it's their business and they will end
up doing it if users request that feature (maybe because they saw it in
wget).

Private SSH keys can be protected with a password if you want to.

Post by Tim Ruehsen
While we could hash anything, it would be way safer for you to protect your

complete home directory

We can do both, hash and still keep the readable to the user only. If the
overhead is not much I would go for it. That is the basis of every security
framework out there: if the benefits of having 2 security mechanisms instead
of only 1 outweigh the drawbacks, then implement 2 instead of 1.



_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?51666>

_______________________________________________
Message sent via/by Savannah
http://savannah.gnu.org/

[Ander Juaristi]

Post by Ander Juaristi
We can do both, hash and still keep the readable to the user only

... hash and still keep the _files_ readable ...



_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?51666>

_______________________________________________
Message sent via/by Savannah
http://savannah.gnu.org/

[Ethan Welker]

Follow-up Comment #4, bug #51666 (project wget):

Hello!

I've written a series of patches (attached) that adds this feature. Please let
me know if there's some other way that I should be submitting these instead.

Some notes:

* I have little security experience, so these should be checked over somewhat
carefully.
* The OpenSSL implementation of ssl_hash_salt uses functions only in OpenSSL
0.9.7 and later. I'm not familiar enough with OpenSSL to get it to support
anything prior to that, but the rest of openssl.c seems to imply that some
versions before that are supported. Sorry about this!
* This patch seamlessly and automatically converts old hsts databases to
hashed databases. Unpatched versions of Wget parse those databases without
issue, though they functionally ignore any converted entry.
** If a patched Wget creates a hashed entry for a HSTS-enabled website, then
an unpatched Wget creates a unhashed entry for the same website, then a
patched Wget converting that entry will create a duplicate entry in the store.
I'm not sure how much of a concern this is.
* This switches the HSTS store from a hash table to an array, and each search
for a host creates a hash for each element in that array until either one
matches or the end of the array is reached. This isn't a problem for
reasonably sized stores; in my test on a 50-entry .wget-hsts, an unpatched
wget was 100 ms _slower_ (presumably due to network jitter). However I can
imagine that if you have several thousand entries this could become a
noticeable slowdown.

(file #44842, file #44843, file #44844)
_______________________________________________________

Additional Item Attachment:

File name: 0001-Add-utility-functions-for-hashing-and-salting.patch Size:4 KB
File name: 0002-src-hsts.c-Hash-HSTS-enabled-domains-to-improve-priv.patch
Size:23 KB
File name: 0003-src-hsts.c-Add-test-for-hashed-domains-stored-in-dat.patch
Size:2 KB


_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?51666>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[Tim Ruehsen]

Follow-up Comment #5, bug #51666 (project wget):

Thanks for addressing the issue.

Saving the salt together with the (salted) hash isn't of big help when we talk
about a limited set of input strings. You can get complete lists of existing
domains and brute force through them in a few seconds. Can even be optimized
by starting with the top 1m domains. I just mention this to make clear that
this way of obscuring is far from being safe. It is just slightly more effort
to reverse the domain names in comparison to unsalted hashes.

Anyways, it helps from being fly-by looked at, e.g. on the console.

I would like to ask you to not use OpenSSL for hashing. We have/use the SHA256
digest functions from gnulib anyways. So it should be straight forward.


_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?51666>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[Ethan Welker]

Follow-up Comment #6, bug #51666 (project wget):

Thank you for your feedback.

Regarding the salt: I used one because OpenSSH uses one and that was both what
the original bug report references, and written by somebody likely more
experienced than me Re: security. If you want me to remove the salting or
something I can do so.

This version of the patch (only one this time, because there's little point in
only separating adding the tests) has the following changes from the previous
patchset:

* Uses gnulib hashing functions instead of gnutls/openssl ones (Sorry, I
didn't know gnulib had those!).
* Fixes the interleaved versions duplicate entry thing.
* Adds dependency on gnulib's crypto/gc and crypto/gc-random (not any
crypto/gc-<hash algorithm>) to generate decent quality randomness for the
salt. Seeing as the salt apparently isn't that important, I guess it could be
switched to random_number if this is a problem.
* In case someone does want to switch to crypto/gc-<hash algorithm> in the
future (to easily add another hashing algorithm or something), the prefix was
switched from 1 to 5 to match the value of GC_SHA256 in gc.h.

(file #44848)
_______________________________________________________

Additional Item Attachment:

File name: 0001-Hash-domains-in-HSTS-database-to-improve-privacy.patch Size:25
KB


_______________________________________________________

Reply to this item at:

<http://savannah.gnu.org/bugs/?51666>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[Tim Rühsen]

Post by Ander Juaristi
I'm not generally against these kind of small tweaks that don't harm and
slightly improve user's privacy.
If Firefox doesn't do it, we don't care: it's their business and they will
end up doing it if users request that feature (maybe because they saw it in
wget).
Private SSH keys can be protected with a password if you want to.

As long as it is optional...

It would be nice being file compatible with Firefox (at least reading Firefox
HSTS db).
Maybe the sqlite backend that has been mentioned earlier should then work with
the same settings (hashed/not hashed).

Post by Ander Juaristi
We can do both, hash and still keep the readable to the user only. If the
overhead is not much I would go for it. That is the basis of every security
framework out there: if the benefits of having 2 security mechanisms instead
of only 1 outweigh the drawbacks, then implement 2 instead of 1.

Absolutely, but in this special case you open up a can of worms. From a
security standpoint, the average home directory is a nightmare. Once someone
gets access to it (read or write)...

Regards, Tim