Coroutines and VirtualAlloc

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Coroutines and VirtualAlloc

Ryan Joseph
I was curious about possible ways coroutines could work in FPC and found this example that claims to implement just that. It appears to be designed for Windows though and makes use of a function called VirtualAlloc which I don’t understand. Is there a non platform specific version of this function? I’d like to test the code but I have no idea what that function does or how to replace it.

http://www.festra.com/wwwboard/messages/12899.html

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Daniel Gaspary
On Sat, Apr 15, 2017 at 6:20 AM, Ryan Joseph <[hidden email]> wrote:
> I was curious about possible ways coroutines could work in FPC and found this example that claims to implement just that. It appears to be designed for Windows though and makes use of a function called VirtualAlloc which I don’t understand. Is there a non platform specific version of this function? I’d like to test the code but I have no idea what that function does or how to replace it.


Using  SetJmp and LongJmp?

I believe some months ago it was a discussion on the list on why this
was not really the way to implement coroutines.

Searching for longjmp/setjmp you can find the thread, I guess.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Ryan Joseph

> On Apr 19, 2017, at 2:34 AM, Daniel Gaspary <[hidden email]> wrote:
>
> Using  SetJmp and LongJmp?
>
> I believe some months ago it was a discussion on the list on why this
> was not really the way to implement coroutines.
>
> Searching for longjmp/setjmp you can find the thread, I guess.

I never heard of those functions and I did find a thread about them but it seemed inconslusive.

Here’s a little test I made not knowing exactly how those functions work. I expected the stack to be restored and “i” incremented but that’s not what happens. The program jumps in and out of the loop but “i” remains the same each time. Is this not how to use those functions?

type
        TCoroutine = class (TObject)
                public
                        procedure Start;
                        procedure Yield;
                        procedure Resume;
                private
                        entry: jmp_buf;
                        env: jmp_buf;
                        passes: longint;
                        yieldNext: boolean;
        end;
       
procedure TCoroutine.Start;
var
        i: integer = 0;
begin
        Setjmp(entry);
        if passes > 0 then
                exit;
        writeln('start');
        while i < 10 do
                begin
                        writeln('loop ', i);
                        i += 1;
                        Setjmp(env);
                        if not yieldNext then
                                Yield;
                        yieldNext := false;
                end;
end;

procedure TCoroutine.Yield;
begin
        writeln('yield');
        passes += 1;
        yieldNext := false;
        Longjmp(entry, 1);
end;

procedure TCoroutine.Resume;
begin
        writeln('resume');
        yieldNext := true;
        Longjmp(env, 1);
end;
       
procedure CoroutineTest;
var
        co: TCoroutine;
        i: integer;
begin
        co := TCoroutine.Create;
        co.Start;
        writeln('stepped out');
        for i := 0 to 4 do
                co.Resume;
        writeln('finished');
end;


========

start
loop 0
yield
stepped out
resume
loop 1
yield
resume
loop 1
yield
resume
loop 1
yield
resume
loop 1
yield
resume
loop 1
yield
finished


Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Mark Morgan Lloyd-5
On 19/04/17 05:00, Ryan Joseph wrote:
>> On Apr 19, 2017, at 2:34 AM, Daniel Gaspary <[hidden email]> wrote:> > Using  SetJmp and LongJmp?> > I believe some months ago it was a discussion on the list on why this> was not really the way to implement coroutines.> > Searching for longjmp/setjmp you can find the thread, I guess.
> I never heard of those functions and I did find a thread about them but it seemed inconslusive.
> Here’s a little test I made not knowing exactly how those functions work. I expected the stack to be restored and “i” incremented but that’s not what happens. The program jumps in and out of the loop but “i” remains the same each time. Is this not how to use those functions?

I thought that Windows implemented "fibers" for this sort of thing?

It is possible to partially-simulate coroutines with setjmp/longjmp, but
you need to store state outside the function. The key thing about
coroutines, at least as implemented by Wirth in Modula-2, is that you
can transfer arbitrarily between them i.e. it's not just a case of a
scheduler selecting one specific task which runs and then returns.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Ryan Joseph

> On Apr 19, 2017, at 3:28 PM, Mark Morgan Lloyd <[hidden email]> wrote:
>
> It is possible to partially-simulate coroutines with setjmp/longjmp, but you need to store state outside the function. The key thing about coroutines, at least as implemented by Wirth in Modula-2, is that you can transfer arbitrarily between them i.e. it's not just a case of a scheduler selecting one specific task which runs and then returns.

I only used coroutines in Lua and they were basically a way for a control structure like a loop to return control the main program without using threading. From the descriptions it sounds like longjmp moves the current point of execution of the program (if that concept exists like I imagine it does) so you could step in and out of loops or functions. If not that then what do those functions actually do?

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Free Pascal - General mailing list
In reply to this post by Ryan Joseph

Am 19.04.2017 06:35 schrieb "Ryan Joseph" <[hidden email]>:
>
>
> > On Apr 19, 2017, at 2:34 AM, Daniel Gaspary <[hidden email]> wrote:
> >
> > Using  SetJmp and LongJmp?
> >
> > I believe some months ago it was a discussion on the list on why this
> > was not really the way to implement coroutines.
> >
> > Searching for longjmp/setjmp you can find the thread, I guess.
>
> I never heard of those functions and I did find a thread about them but it seemed inconslusive.

Those functions simply store (setjmp) and restore (longjmp) register values (and setjmp also returns the value passed to longjmp if it had been reached by a longjmp). Nothing more, nothing less. So while the stack register while be changed, the contents on the stack will not.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Ryan Joseph

> On Apr 19, 2017, at 4:14 PM, Sven Barth via fpc-pascal <[hidden email]> wrote:
>
> Those functions simply store (setjmp) and restore (longjmp) register values (and setjmp also returns the value passed to longjmp if it had been reached by a longjmp). Nothing more, nothing less. So while the stack register while be changed, the contents on the stack will not.

Why doesn’t “i” in my example increment? The value keeps going back to 1 even after I used += 1 so its like the old copy of the stack before the jump got pushed back on top and it started over. I don’t understand how assembly works but I thought it would just start over and the state of the stack in that function would still be the same as before so I could keep adding 1 every pass.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Michael Van Canneyt


On Wed, 19 Apr 2017, Ryan Joseph wrote:

>
>> On Apr 19, 2017, at 4:14 PM, Sven Barth via fpc-pascal <[hidden email]> wrote:
>>
>> Those functions simply store (setjmp) and restore (longjmp) register values (and setjmp also returns the value passed to longjmp if it had been reached by a longjmp). Nothing more, nothing less. So while the stack register while be changed, the contents on the stack will not.
>
> Why doesn’t “i” in my example increment?  The value keeps going back to 1
> even after I used += 1 so its like the old copy of the stack before the
> jump got pushed back on top and it started over.  I don’t understand how
> assembly works but I thought it would just start over and the state of the
> stack in that function would still be the same as before so I could keep
> adding 1 every pass.
The stack remains untouched by setjmp/longjmp.

Your reasoning contains a wrong assumption, namely that I is on the stack.

If I is a register variable, then it is not on the stack, and will be reset
with each longjmp.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Free Pascal - General mailing list
In reply to this post by Ryan Joseph

Am 19.04.2017 11:26 schrieb "Ryan Joseph" <[hidden email]>:
>
>
> > On Apr 19, 2017, at 4:14 PM, Sven Barth via fpc-pascal <[hidden email]> wrote:
> >
> > Those functions simply store (setjmp) and restore (longjmp) register values (and setjmp also returns the value passed to longjmp if it had been reached by a longjmp). Nothing more, nothing less. So while the stack register while be changed, the contents on the stack will not.
>
> Why doesn’t “i” in my example increment? The value keeps going back to 1 even after I used += 1 so its like the old copy of the stack before the jump got pushed back on top and it started over. I don’t understand how assembly works but I thought it would just start over and the state of the stack in that function would still be the same as before so I could keep adding 1 every pass.

Ah, I didn't catch that earlier. Probably "i" is kept in a register due to optimization. You should check the generated assembler code to be sure.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Ryan Joseph
In reply to this post by Michael Van Canneyt

> On Apr 19, 2017, at 4:33 PM, Michael Van Canneyt <[hidden email]> wrote:
>
> Your reasoning contains a wrong assumption, namely that I is on the stack.
>
> If I is a register variable, then it is not on the stack, and will be reset
> with each longjmp.

I thought all variables declared inside a function (like below) were “on the stack” and I don’t even know what a register variable is honestly. Is there a way to make my example work and not have “i” reset each jump by declaring it differently or is that the choice of the compiler?

procedure TCoroutine.Start;
var
        i: integer = 0;
begin
end;

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Michael Van Canneyt
In reply to this post by Free Pascal - General mailing list


On Wed, 19 Apr 2017, Sven Barth via fpc-pascal wrote:

> Am 19.04.2017 11:26 schrieb "Ryan Joseph" <[hidden email]>:
>>
>>
>>> On Apr 19, 2017, at 4:14 PM, Sven Barth via fpc-pascal <
> [hidden email]> wrote:
>>>
>>> Those functions simply store (setjmp) and restore (longjmp) register
> values (and setjmp also returns the value passed to longjmp if it had been
> reached by a longjmp). Nothing more, nothing less. So while the stack
> register while be changed, the contents on the stack will not.
>>
>> Why doesn’t “i” in my example increment? The value keeps going back to 1
> even after I used += 1 so its like the old copy of the stack before the
> jump got pushed back on top and it started over. I don’t understand how
> assembly works but I thought it would just start over and the state of the
> stack in that function would still be the same as before so I could keep
> adding 1 every pass.
>
> Ah, I didn't catch that earlier. Probably "i" is kept in a register due to
> optimization. You should check the generated assembler code to be sure.
Glad we responded the same thing :)

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Mark Morgan Lloyd-5
In reply to this post by Ryan Joseph
On 19/04/17 09:30, Ryan Joseph wrote:
>> On Apr 19, 2017, at 3:28 PM, Mark Morgan Lloyd <[hidden email]> wrote:> > It is possible to partially-simulate coroutines with setjmp/longjmp, but you need to store state outside the function. The key thing about coroutines, at least as implemented by Wirth in Modula-2, is that you can transfer arbitrarily between them i.e. it's not just a case of a scheduler selecting one specific task which runs and then returns.
> I only used coroutines in Lua and they were basically a way for a control structure like a loop to return control the main program without using threading. From the descriptions it sounds like longjmp moves the current point of execution of the program (if that concept exists like I imagine it does) so you could step in and out of loops or functions. If not that then what do those functions actually do?

SetJmp records the current state, LongJmp reverts to it. There's some
common-sense limitations.

I've got an example program of about 100 lines that would demonstrate
what I've hacked together, I could tack it onto a message if the ML
managers consider that acceptable.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Ryan Joseph

> On Apr 19, 2017, at 4:57 PM, Mark Morgan Lloyd <[hidden email]> wrote:
>
> SetJmp records the current state, LongJmp reverts to it. There's some common-sense limitations.
>
> I've got an example program of about 100 lines that would demonstrate what I've hacked together, I could tack it onto a message if the ML managers consider that acceptable.

yes, I’d like to see that so I know why my example doesn’t work as I expected. Everything I’m hearing makes me think “i” should keep incrementing after I call SetJmp and then return with JongJmp but there’s something I’m missing obviously.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Michael Van Canneyt
In reply to this post by Ryan Joseph


On Wed, 19 Apr 2017, Ryan Joseph wrote:

>
>> On Apr 19, 2017, at 4:33 PM, Michael Van Canneyt <[hidden email]> wrote:
>>
>> Your reasoning contains a wrong assumption, namely that I is on the stack.
>>
>> If I is a register variable, then it is not on the stack, and will be reset
>> with each longjmp.
>
> I thought all variables declared inside a function (like below) were “on
> the stack” and I don’t even know what a register variable is honestly.
It's a variable which the compiler does not put on the stack, it exists just
in a register.

>  Is
> there a way to make my example work and not have “i” reset each jump by
> declaring it differently or is that the choice of the compiler?

The compiler chooses this. But you can disable it, I think, by disabling
optimizations.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Ryan Joseph

> On Apr 19, 2017, at 5:17 PM, Michael Van Canneyt <[hidden email]> wrote:
>
> It's a variable which the compiler does not put on the stack, it exists just
> in a register.

That kind of defeats the purpose then if you can’t rely on function scoped variables to be restored. Maybe that’s what the code in the original link did? I was going to try it but I never figured out how to replace VirtualAlloc without Windows (I’m on a Mac).

Thanks.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Karoly Balogh (Charlie/SGR)
In reply to this post by Ryan Joseph
Hi,

On Wed, 19 Apr 2017, Ryan Joseph wrote:

> yes, I?d like to see that so I know why my example doesn?t work as I
> expected. Everything I?m hearing makes me think ?i? should keep
> incrementing after I call SetJmp and then return with JongJmp but
> there?s something I?m missing obviously.

Your example is simply broken. A few points:

First, When you're LongJmp'ing to entry state in Resume, you're expecting
the Start stack state (and with that, your "i" variable's state) to be
intact. This is not the case. After the Start method returns (stepped out)
it's stack contents are destroyed, and even get overwritten by the next
'writeln' method call, so the next time Resume is called, you get
unpredictable behavior by jumping into the stack state of Start, which no
longer exist and was overwritten since. For example, it simply crashes
with Runtime Error 216 on Mac OS X 32 bit. For you, it just uses some
"random" value on the stack (or register), which happens to be 1. I get
this even without register variables, so I doubt it has anything to do
with that... (Sorry Michael/Sven... ;)

Basically, you can only jump to a "higher" level on the stack, not equal
(from a different function) and not to a "deeper" level.

Second, if you really want to use SetJmp/LongJmp states you can use SetJmp
return value for that. If SetJmp returns 0, it wasn't returning from a
LongJmp. If it returns 1, it was returning from a LongJmp. You don't need
external state to track that.

Charlie
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Ryan Joseph

> On Apr 19, 2017, at 5:37 PM, Karoly Balogh (Charlie/SGR) <[hidden email]> wrote:
>
> Your example is simply broken. A few points:

Thanks for the description. Yeah, I assumed the stack would be restored but that isn’t the case apparently.

I think the coroutine implementation in the link below tries to manage the stack frames and restore them but it requires Windows for the VirtualAlloc function and I’m on a Mac. No idea if that code works or not and it looks kind of risky honestly.

http://www.festra.com/wwwboard/messages/12899.html

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Marco van de Voort
In our previous episode, Ryan Joseph said:
> > Your example is simply broken. A few points:
>
> Thanks for the description. Yeah, I assumed the stack would be restored but that isn?t the case apparently.
>
> I think the coroutine implementation in the link below tries to manage the stack frames and restore them but it requires Windows for the VirtualAlloc function and I?m on a Mac. No idea if that code works or not and it looks kind of risky honestly.
>
> http://www.festra.com/wwwboard/messages/12899.html

Not just virtual alloc, nearly the whole of the asm is accessing SEH via fs
and thus Windows dependent.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Daniel Gaspary
So..

Any chance of an Official implementation ?

On Wed, Apr 19, 2017 at 10:50 AM, Marco van de Voort <[hidden email]> wrote:

> In our previous episode, Ryan Joseph said:
>> > Your example is simply broken. A few points:
>>
>> Thanks for the description. Yeah, I assumed the stack would be restored but that isn?t the case apparently.
>>
>> I think the coroutine implementation in the link below tries to manage the stack frames and restore them but it requires Windows for the VirtualAlloc function and I?m on a Mac. No idea if that code works or not and it looks kind of risky honestly.
>>
>> http://www.festra.com/wwwboard/messages/12899.html
>
> Not just virtual alloc, nearly the whole of the asm is accessing SEH via fs
> and thus Windows dependent.
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Coroutines and VirtualAlloc

Marco van de Voort
In our previous episode, Daniel Gaspary said:
> So..
>
> Any chance of an Official implementation ?

I don't know. Such effort should chiefly come from the people interested in
it I guess.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
12