Need help to fix a bug

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Need help to fix a bug

José Mejuto
Hello FPC-Pascal,

I wish to fix this bug http://bugs.freepascal.org/view.php?id=15460
but I had found serious problems to understand how data is structured
in the TBufIndex and descendant classes, specially the
TDoubleLinkedBufIndex.

Can somebody help me ? From TBufDataset.BuildIndex I think that:

PBufRecLinkItem.Next points to next record.
PBufRecLinkItem.Prior points to previous record.

Maybe PBufRecLinkItem points to the dataset item data ?

Which is supossed to be pointed by:

(AIndex as TDoubleLinkedBufIndex).FFirstRecBuf
(AIndex as TDoubleLinkedBufIndex).FLastRecBuf

The first and last item in the dataset ?

Also which is the meaning of this line:

(AIndex as TDoubleLinkedBufIndex).FLastRecBuf[(AIndex as TDoubleLinkedBufIndex).IndNr].next:=(AIndex as TDoubleLinkedBufIndex).FFirstRecBuf;

I'm completly lost in this piece of code, and I really need it as my
SQL fetches could take around 1 minute and reorder data is a must. I
know I can reissue the query with a different order but this will take
around 20 seconds again while doing it in local memory is a breeze.

PS: To check changes in FPC I'm launching a complete build (without
clean) but this process will take 2 minutes anyway. Is there a faster
way to correctly build only the affected package (in this case fcl-db)
?

--
Best regards,
 JoshyFun

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Need help to fix a bug

Joost van der Sluis
On Wed, 2010-01-13 at 16:13 +0100, JoshyFun wrote:
> I wish to fix this bug http://bugs.freepascal.org/view.php?id=15460
> but I had found serious problems to understand how data is structured
> in the TBufIndex and descendant classes, specially the
> TDoubleLinkedBufIndex.

Oohh... nice... Someone to help me. ;)

Too bad you used such a general topic-name, I almost overlooked it.

> Can somebody help me ? From TBufDataset.BuildIndex I think that:

Note that TBufDataset supports more then one index-system. (index-based
and double-linked-list based) Only the second form supports indexing
atm, so that TBufDataset.BuildIndex assumes that the double-linked-list
us used. I'll also assume this further on in this mail.

> PBufRecLinkItem.Next points to next record.
> PBufRecLinkItem.Prior points to previous record.
>
> Maybe PBufRecLinkItem points to the dataset item data ?

Somewhat. The item-data and the linked list are stored within one memory
block. The PBufRecLinkItem points at the start of that memory block. The
block starts with the several indexes, then the real data. Now assume
that RecPtr is a PBufRecLinkItem which points at address 0000.

Assume a pointer is 32 bit/4 bytes, one record is stored into memory as
follows:

Address  What is stored inside it   Pointer to this address

0000     PBufRecLinkItem.prior (1)  RecPtr[0].prior
0004     PBufRecLinkItem.next  (1)  RecPtr[0].next  
0008     PBufRecLinkItem.prior (2)  RecPtr[1].prior
000c     PBufRecLinkItem.next  (2)  RecPtr[1].next  
0010     PBufRecLinkItem.prior (3)  RecPtr[2].prior
0014     PBufRecLinkItem.next  (3)  RecPtr[2].next  
0018     Start of real data

So in one record-memory block, the indexes and the data are stored. As
you can see the block above can contain three indexes. Not more, because
in that case the memory for all records should be re-allocated. This is
no problem when you define your indexes before opening the dataset.
(This is forced by TBufDataset when using this double-linked index) By
default there are always two indexes defined: the primary index which
contains the records in the order they are read. And a second index
which is initially blank. This index is used when you use the
IndexFieldNames property. Now you also know why it's not necessary to
allocate memory when building this second index. It's already allocated
during the initial allocation for the data for the records. It's also
why this merge-sort algorithm is used. It doesn't need any _additional_
space.

> Which is supossed to be pointed by:
>
> (AIndex as TDoubleLinkedBufIndex).FFirstRecBuf
> (AIndex as TDoubleLinkedBufIndex).FLastRecBuf
> The first and last item in the dataset ?

Well, the TDoubleLinkedBufIndex contains the additional
index-information and functions to work with it. FFirstRecBuf points to
the first record. (It's address 0000 as in the memory-layout above)
FLastRecBuf points to the record _after_ the last record. This record is
used as the 'spare' record. It is used for all kinds of tasks, like the
record which you use when you set a filter. Or in which data is read

> Also which is the meaning of this line:
>
> (AIndex as TDoubleLinkedBufIndex).FLastRecBuf[(AIndex as TDoubleLinkedBufIndex).IndNr].next:=(AIndex as TDoubleLinkedBufIndex).FFirstRecBuf;

Well, AIndex is the used index. FLastRecBuf is it's spare record.
AIndex.IndNr is the number of the index. Since FLastRecBuf is defined as
a record containing a prior and a next pointer, FLastRecBuf[0].next
points to the .prior element of the first (zeroth) index.
FLastRecBuf[1].next points to the second index. Which is hard-coded the
index used for the IndexFieldNames-index.

So the line above let the next record of the spare-record point to the
first record of the index. (Creating a closed loop. Seems to be a bad
idea, but there is probably some logic behind it)

> I'm completly lost in this piece of code, and I really need it as my
> SQL fetches could take around 1 minute and reorder data is a must. I
> know I can reissue the query with a different order but this will take
> around 20 seconds again while doing it in local memory is a breeze.

It should work. Maybe you can try if you can get the tests in
packages/fcl-db/tests running. There is also a test for IndexFieldNames,
iirc. I can't help you with the bug report as long as I don't have an
example.

It would be completely perfect if you can create a test for the
db-testsuite that fails, so that I can fix that.

> PS: To check changes in FPC I'm launching a complete build (without
> clean) but this process will take 2 minutes anyway. Is there a faster
> way to correctly build only the affected package (in this case fcl-db)

Yes, go to the packages/fcl-db directory and do a 'make all install'.
Then compile your program. You only have to recompile the LCL (nothing
more) if you changed something in the interface-section of one of the
db-units.

You can use the 'before compilation' settings of Lazarus to do the 'make
all install' You can even let Lazarus parse it's output, so that in case
of an error, Lazarus will show you the offending line.

This works in 90% of the cases. So you almost never have to clean and
rebuild everything.

You can also simply add packages/fcl-db/src, packages/fcl-db/src/base,
packages/fcl-db/src/sqldb, packages/fcl-db/src/sqldb/interbase and all
other database-paths to the compiler-search-path in Lazarus. But cleanup
the created .ppu's and .o's afterwards, or you'll run into trouble when
you forget them.

Joost.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re[2]: Need help to fix a bug

José Mejuto
Hello FPC-Pascal,

Sunday, January 24, 2010, 6:18:22 PM, you wrote:

JvdS> On Wed, 2010-01-13 at 16:13 +0100, JoshyFun wrote:
>> I wish to fix this bug http://bugs.freepascal.org/view.php?id=15460
>> but I had found serious problems to understand how data is structured
>> in the TBufIndex and descendant classes, specially the
>> TDoubleLinkedBufIndex.

JvdS> Oohh... nice... Someone to help me. ;)

I'll try, but I can not promise anything :(

JvdS> Too bad you used such a general topic-name, I almost overlooked it.

For sure the "need help to fix a bug" is not a common topic :) but I
understand that a reference to fcl-db should be included.

I'll answer to this email partially before a more in deep read.

JvdS> during the initial allocation for the data for the records. It's also
JvdS> why this merge-sort algorithm is used. It doesn't need any _additional_
JvdS> space.

Yes I understand the fact that the space is preallocated, but the data
flow in the BuildIndex is wrong in my opinion as in the first place it
copies index 0 over index 1, a simple copy, and them start to
mergesort everything in index 1, it it uses index 1 as a temporal
space to itself. I could had overlooked something but Merge Sort needs
a temporal space to drop data to be copied back or sometimes you will
overwrite data that will be moved again in next step. Something like:

2
1
3

Merging this will render in move "index 1" to position 0, and move
"index 0" to position 1, using itself as a temporal space will render
in:

2 (1) 1  1
1  1 (1) 1
3  3  3  3

Index element 0 has dissapear () shows the moved element in each step.

Anyway I'll take an additional look to the code after reading your in
detail explanation. Thank you for the help.

--
Best regards,
 JoshyFun

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal