Processing passwords etc.

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Processing passwords etc.

Mark Morgan Lloyd-5
Is my understanding correct that when a string or a dynamic array is
extended it might result in its existing content being released to the heap?

If so, is it possible to ensure that this is zeroed or randomised first,
without having to do it manually?

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Michael Van Canneyt


On Fri, 11 Apr 2014, Mark Morgan Lloyd wrote:

> Is my understanding correct that when a string or a dynamic array is extended
> it might result in its existing content being released to the heap?
>
> If so, is it possible to ensure that this is zeroed or randomised first,
> without having to do it manually?

Currently not, although such behaviour could easile be introduced as an option.

Current HeartBleed frenzy getting you (or your bosses) scared ? :)

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Jonas Maebe-2
In reply to this post by Mark Morgan Lloyd-5

On 11 Apr 2014, at 09:36, Mark Morgan Lloyd wrote:

> Is my understanding correct that when a string or a dynamic array is extended it might result in its existing content being released to the heap?
>
> If so, is it possible to ensure that this is zeroed or randomised first, without having to do it manually?

You can install a memory manager that does this for all (de)allocations and then calls through to the original memory manager. There is no way to only do this for strings and dynamic arrays, and I don't think it would be a good idea to do so. Not all passwords are strings, so that would probably mostly give a false sense of security.


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Mark Morgan Lloyd-5
Jonas Maebe wrote:
> On 11 Apr 2014, at 09:36, Mark Morgan Lloyd wrote:
>
>> Is my understanding correct that when a string or a dynamic array is extended it might result in its existing content being released to the heap?
>>
>> If so, is it possible to ensure that this is zeroed or randomised first, without having to do it manually?
>
> You can install a memory manager that does this for all (de)allocations and then calls through to the original memory manager. There is no way to only do this for strings and dynamic arrays, and I don't think it would be a good idea to do so. Not all passwords are strings, so that would probably mostly give a false sense of security.

Using a memory manager would reliably wipe strings etc. when reallocated
(i.e rather than when an assignment didn't trigger reallocation). On the
other hand it would have the overhead of also overwriting blocks that
the user knew were being freed and could audit first, as well as stuff
that was being freed as part of the RTL operation.

It's that latter case- where the RTL is copying something without the
user being aware- that I think is significant.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Mark Morgan Lloyd-5
In reply to this post by Michael Van Canneyt
Michael Van Canneyt wrote:

> On Fri, 11 Apr 2014, Mark Morgan Lloyd wrote:
>
>> Is my understanding correct that when a string or a dynamic array is
>> extended it might result in its existing content being released to the
>> heap?
>>
>> If so, is it possible to ensure that this is zeroed or randomised
>> first, without having to do it manually?
>
> Currently not, although such behaviour could easile be introduced as an
> option.
>
> Current HeartBleed frenzy getting you (or your bosses) scared ? :)

:-) No, but I don't think enough people are focussing on the real
problem which is that the OpenSSL developers were letting sensitive data
leak to the freelist.

If, when they wrote the code some years ago, they'd been rigorous in
their handling of passwords and private keys then the current bug-
introduced in 2012- would have been far less serious.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Jonas Maebe-2
In reply to this post by Mark Morgan Lloyd-5

On 11 Apr 2014, at 10:10, Mark Morgan Lloyd wrote:

> Jonas Maebe wrote:
>> On 11 Apr 2014, at 09:36, Mark Morgan Lloyd wrote:
>>> Is my understanding correct that when a string or a dynamic array is extended it might result in its existing content being released to the heap?
>>>
>>> If so, is it possible to ensure that this is zeroed or randomised first, without having to do it manually?
>> You can install a memory manager that does this for all (de)allocations and then calls through to the original memory manager. There is no way to only do this for strings and dynamic arrays, and I don't think it would be a good idea to do so. Not all passwords are strings, so that would probably mostly give a false sense of security.
>
> Using a memory manager would reliably wipe strings etc. when reallocated (i.e rather than when an assignment didn't trigger reallocation). On the other hand it would have the overhead of also overwriting blocks that the user knew were being freed and could audit first, as well as stuff that was being freed as part of the RTL operation.

Such stuff from RTL operations may include getmem'd buffers used to read private key data from disk.

> It's that latter case- where the RTL is copying something without the user being aware- that I think is significant.

Either the memory manager guarantees trashing, or it does not. Having half-solutions are worse than nothing at all, because it gives a false sense of security. Even the best audits are guaranteed to miss things (as in case of the person who wrote and the person who reviewed/audited the patch that introduced the heartbleed bug), and as you mentioned there is no control over the internals of the RTL (which has not been written at all from the point of view that buffers that might sometimes contain sensitive data should always be cleansed).


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Michael Van Canneyt
In reply to this post by Mark Morgan Lloyd-5


On Fri, 11 Apr 2014, Mark Morgan Lloyd wrote:

> Michael Van Canneyt wrote:
>> On Fri, 11 Apr 2014, Mark Morgan Lloyd wrote:
>>
>>> Is my understanding correct that when a string or a dynamic array is
>>> extended it might result in its existing content being released to the
>>> heap?
>>>
>>> If so, is it possible to ensure that this is zeroed or randomised first,
>>> without having to do it manually?
>>
>> Currently not, although such behaviour could easile be introduced as an
>> option.
>>
>> Current HeartBleed frenzy getting you (or your bosses) scared ? :)
>
> :-) No, but I don't think enough people are focussing on the real problem
> which is that the OpenSSL developers were letting sensitive data leak to the
> freelist.
>
> If, when they wrote the code some years ago, they'd been rigorous in their
> handling of passwords and private keys then the current bug- introduced in
> 2012- would have been far less serious.

Correct.

Having looked at the openssl library, I can say it's a miracle it works at all.
Rarely seen such a mess of macros and whatnot, a good showcase of why I think
C is not a good language choice. So, not surprised that it contains leaks :(

OTOH, I think people are hugely exaggerating the problem, considering it was
introduced relatively recently and that I got my security update before it
hit the newspapers.

That is of course not to say that it shouldn't be fixed and people shouldn't bother.
But the way it is presented is more about scaring people than anything else.
Hysterics...


Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Jonas Maebe-2

On 11 Apr 2014, at 10:26, Michael Van Canneyt wrote:

> OTOH, I think people are hugely exaggerating the problem, considering it was introduced relatively recently and that I got my security update before it hit the newspapers.

The exploit code was also on github before news about the bug hit the newspapers. There is even some evidence it may have been exploited for at least 3 months already, maybe longer (because unless you used some special intrusion detection system rules, it left no traces at all in the log files, so there's only very little data to go on).

Also, the fact that you updated your server so quickly, doesn't mean that everyone did. Our university's mail servers were only patched yesterday morning (more than 24 hours after the story broke), because they needed time to prepare the patching (don't ask, I don't know the details). I bet tons of credentials and private data has been accessed over the past days all over the world.

> That is of course not to say that it shouldn't be fixed and people shouldn't bother.
> But the way it is presented is more about scaring people than anything else. Hysterics...

I very strongly disagree. All certificates and login data used with vulnerable services over the past year or so should be considered compromised. It will probably take months before all affected certificates are replaced (if that ever happens for most of them), and many of the replaced and hence potentially compromised certificates will probably never be revoked. The result is a huge increase in chances for man-in-the-middle attacks, not to mention all the compromised login data and private information (emails, bank statements, ...).


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Michael Van Canneyt


On Fri, 11 Apr 2014, Jonas Maebe wrote:

>
> On 11 Apr 2014, at 10:26, Michael Van Canneyt wrote:
>
>> OTOH, I think people are hugely exaggerating the problem, considering it was introduced relatively recently and that I got my security update before it hit the newspapers.
>
>> That is of course not to say that it shouldn't be fixed and people shouldn't bother.
>> But the way it is presented is more about scaring people than anything else. Hysterics...
>
> I very strongly disagree. All certificates and login data used with
> vulnerable services over the past year or so should be considered
> compromised.  It will probably take months before all affected
> certificates are replaced (if that ever happens for most of them), and
> many of the replaced and hence potentially compromised certificates will
> probably never be revoked.  The result is a huge increase in chances for
> man-in-the-middle attacks, not to mention all the compromised login data
> and private information (emails, bank statements, ...).

Like I said, this is not to say that no action should be taken.
I expect that all sensitive sites (banks, google, etc) have taken immediate action.

That the login of my local tennis/pool/golf club was compromised is not really so scary, sorry.

Anyway, getting off topic.

The main point is that in FPC you can install a memory manager that wipes
out any memory when getting or releasing it, if you want to make your software more secure that way.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

wkitty42
On 4/11/2014 5:03 AM, Michael Van Canneyt wrote:
> The main point is that in FPC you can install a memory manager that wipes out
> any memory when getting or releasing it, if you want to make your software more
> secure that way.

how would one go about doing this? i learned in my TP3/6 days to use fillchar on
everything to ensure that it was filled with 0x00... especially my data files...
when looking at them with a hex editor, my OCD would hit strongly because the
data file was "not clean and holding only my data"...

i don't know how one would go about cleaning released memory as someone else
asked about (eg: extending an array or string or etc)... once the memory is
released, it is no longer accessible, right?

--
NOTE: No off-list assistance is given without prior approval.
       Please keep mailing list traffic on the list unless
       private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

wkitty42
In reply to this post by Mark Morgan Lloyd-5
On 4/11/2014 4:10 AM, Mark Morgan Lloyd wrote:
> Using a memory manager would reliably wipe strings etc. when reallocated (i.e
> rather than when an assignment didn't trigger reallocation). On the other hand
> it would have the overhead of also overwriting blocks that the user knew were
> being freed and could audit first, as well as stuff that was being freed as part
> of the RTL operation.
>
> It's that latter case- where the RTL is copying something without the user being
> aware- that I think is significant.

agreed...

--
NOTE: No off-list assistance is given without prior approval.
       Please keep mailing list traffic on the list unless
       private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Mark Morgan Lloyd-5
In reply to this post by wkitty42
waldo kitty wrote:

> On 4/11/2014 5:03 AM, Michael Van Canneyt wrote:
>> The main point is that in FPC you can install a memory manager that
>> wipes out
>> any memory when getting or releasing it, if you want to make your
>> software more
>> secure that way.
>
> how would one go about doing this? i learned in my TP3/6 days to use
> fillchar on everything to ensure that it was filled with 0x00...
> especially my data files... when looking at them with a hex editor, my
> OCD would hit strongly because the data file was "not clean and holding
> only my data"...

The ideal is to overwrite sensitive data with random bytes, since even
the length of a zero block can be useful to an attacker.

> i don't know how one would go about cleaning released memory as someone
> else asked about (eg: extending an array or string or etc)... once the
> memory is released, it is no longer accessible, right?

But since the deallocated memory is going to a local heap, sooner or
later you're likely to get that back as a new block. That, as I
understand it, is what happened in OpenSSL.

The worst case would be if a cautious programmer zeroed everything that
he was freeing explicitly, without realising that any strings he
extended were going back into the heap intact so now stood out like a
sore thumb. Anybody who was able to inspect the heap would see only
strings that had subsequently been expanded:

     password := getFromUser();          // Probably about 7 chars
     password += #$00 + systemName();    // Leaves password on heap
     saveToDB(Tiger2(password));
     zeroString(password)                // Length doesn't change
   end;                                  // Zeroed block freed to heap

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Sven Barth-2
In reply to this post by wkitty42

Am 11.04.2014 20:50 schrieb "waldo kitty" <[hidden email]>:
>
> On 4/11/2014 5:03 AM, Michael Van Canneyt wrote:
>>
>> The main point is that in FPC you can install a memory manager that wipes out
>> any memory when getting or releasing it, if you want to make your software more
>> secure that way.
>
[snip]
>
> i don't know how one would go about cleaning released memory as someone else asked about (eg: extending an array or string or etc)... once the memory is released, it is no longer accessible, right?

That's where the heap manager comes in. In FPC every allocation that is not done manually through OS functions (namely string/array allocation, class allocation, getmem() and new()) go through the heap manager of which you can register a custom one. This heap manager hooks the allocation and deallocation calls and thus you could provide a manager that clears deallocated memory (or fills with random data) before it's passed on to the default heap manager.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Michael Van Canneyt


On Sat, 12 Apr 2014, Sven Barth wrote:

>
> Am 11.04.2014 20:50 schrieb "waldo kitty" <[hidden email]>:
> >
> > On 4/11/2014 5:03 AM, Michael Van Canneyt wrote:
> >>
> >> The main point is that in FPC you can install a memory manager that wipes out
> >> any memory when getting or releasing it, if you want to make your software more
> >> secure that way.
> >
> [snip]
> >
> > i don't know how one would go about cleaning released memory as someone else asked about (eg: extending an array or string
> or etc)... once the memory is released, it is no longer accessible, right?
>
> That's where the heap manager comes in. In FPC every allocation that is not done manually through OS functions (namely
> string/array allocation, class allocation, getmem() and new()) go through the heap manager of which you can register a
> custom one. This heap manager hooks the allocation and deallocation calls and thus you could provide a manager that clears
> deallocated memory (or fills with random data) before it's passed on to the default heap manager.
Attached is an implementation that allows you to specify:

What to do when allocating memory (zero out, randomize, nothing)
What to do when freeing memory (zero out, randomize, nothing).

Careful, it does not play well with the heaptrc unit.

If there is an interest in such thing, we can add it to the RTL.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

safemmgr.pp (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

wkitty42
On 4/12/2014 8:24 AM, Michael Van Canneyt wrote:
> Attached is an implementation that allows you to specify:
>
> What to do when allocating memory (zero out, randomize, nothing)
> What to do when freeing memory (zero out, randomize, nothing).

very nice! thank you sir! i'm still learning FPC and all its fancy (to me
compared to TP/BP) features :)

> Careful, it does not play well with the heaptrc unit.

why is that? is there a way to fix it? i do use the heaptrc unit to try to
ensure that i'm not leaving stuff behind...

> If there is an interest in such thing, we can add it to the RTL.

as miniscule as it is compared with others', here's my vote in the positive ;)

--
NOTE: No off-list assistance is given without prior approval.
       Please keep mailing list traffic on the list unless
       private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Michael Van Canneyt


On Sat, 12 Apr 2014, waldo kitty wrote:

> On 4/12/2014 8:24 AM, Michael Van Canneyt wrote:
>> Attached is an implementation that allows you to specify:
>>
>> What to do when allocating memory (zero out, randomize, nothing)
>> What to do when freeing memory (zero out, randomize, nothing).
>
> very nice! thank you sir! i'm still learning FPC and all its fancy (to me
> compared to TP/BP) features :)
>
>> Careful, it does not play well with the heaptrc unit.
>
> why is that? is there a way to fix it? i do use the heaptrc unit to try to
> ensure that i'm not leaving stuff behind...

It works together, no worries there, but heaptrc does it's own
'trashing' a bit: it writes some housekeeping data, so released
zeroed-out memory is not 100% zeroed-out.

The regular heap manager itself does this too, but heaptrc just a bit more...

>
>> If there is an interest in such thing, we can add it to the RTL.
>
> as miniscule as it is compared with others', here's my vote in the positive
> ;)

Duly noted :)

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Ludo Brands
In reply to this post by Michael Van Canneyt
On 04/12/2014 02:24 PM, Michael Van Canneyt wrote:

>
> Attached is an implementation that allows you to specify:
>

A few comments:
- allocmem already zeros the memory. No need to do it a second time
- Getmem and Allocmem can return nil (dependent on mem manager,
sometimes on ReturnNilIfGrowHeapfails). A test in ZeroMem and RandomMem
on nil would be "Safe".
- SafeReAllocMem doesn't clear/scramble the memory in case a realloc
moves the memory block to a different place.

Cheers

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Michael Van Canneyt


On Sun, 13 Apr 2014, Ludo Brands wrote:

> On 04/12/2014 02:24 PM, Michael Van Canneyt wrote:
>
>>
>> Attached is an implementation that allows you to specify:
>>
>
> A few comments:
> - allocmem already zeros the memory. No need to do it a second time

Indeed, copy&paste from getmem. Removed the zeroing.

> - Getmem and Allocmem can return nil (dependent on mem manager,
> sometimes on ReturnNilIfGrowHeapfails). A test in ZeroMem and RandomMem
> on nil would be "Safe".

Indeed :) Good point, I have added this check :)

> - SafeReAllocMem doesn't clear/scramble the memory in case a realloc
> moves the memory block to a different place.

You are right. A logic error. ReallocMem is the more tricky one.
Revised.

That's why I post such quick code; for peer review.

Thanks for pointing these out, revised version attached.
Again, comments/reviews welcome.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

safemmgr.pp (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Ludo Brands
On 04/13/2014 09:02 AM, Michael Van Canneyt wrote:
>

>
> You are right. A logic error. ReallocMem is the more tricky one.
> Revised.

In case of a block move, the memory (OP) is already freed when you
clear/scramble it. I'm afraid you can't simply re-use the underlying
ReAllocMem (think cmem for example).

>
> That's why I post such quick code; for peer review.
>
> Thanks for pointing these out, revised version attached.
> Again, comments/reviews welcome.
>
You're welcome :)

Ludo

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Processing passwords etc.

Mark Morgan Lloyd-5
In reply to this post by Michael Van Canneyt
Michael Van Canneyt wrote:

>>> If there is an interest in such thing, we can add it to the RTL.
>>
>> as miniscule as it is compared with others', here's my vote in the
>> positive ;)
>
> Duly noted :)

Can I add my positive vote vote to that please, for the specific case
where a string etc. is expanded. As others have said, if a programmer
/explicitly/ releases unwiped sensitive data that's his problem.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
12