mysqldump – GSoC Weekly Report – Week 9

PROJECT: Adding parallelism to mysqldump
WEEK 09

KEY ACCOMPLISHMENTS LAST WEEK

* The number of threads created is limited, not one for each table anymore. This implied changes in the thread_data structure, in the way threads are created, also a mutex was used to synchronize data between threads.

KEY TASKS THAT STALLED LAST WEEK

* Lost some time due to problems with the environment where I run tests and had to recreate this in another machine.

KEY CONCERNS

Need to speed up a little bit to get in track again.

TASKS IN THE UPCOMING WEEK

* Add command-line option to allow user to choose the number of threads (right now it is hardcoded, just for test purposes).
* Add possibility to execute code in old behavior (serial) rather than new one (parallel).

mysqldump – GSoC Weekly Report – Week 8

KEY ACCOMPLISHMENTS LAST WEEK

* Solved out-of-memory error, finally!!! (credits go to my mentor, Stewart Smith)
* Refactored code to adapt to coding style/guidelines.
* I’m finally getting the way of bzr.

KEY TASKS THAT STALLED LAST WEEK

None

KEY CONCERNS

It took a lot of time to solve this error, so now I gotta speed up a little bit to get in track again.

TASKS IN THE UPCOMING WEEK

* Add command-line option to allow the user to specify how many threads will be created. This will imply changes in the way each thread receives the tables (currently just one) it will dump, probably using a list shared among threads.
* If time allows, include my code in the source tree

Ajude a sustentar a Wikipedia e outros projetos, sem colocar a mão no bolso, e concorra a um Eee PC!

Uma iniciativa para incentivar a contribuição com projetos open-source/livres.
A brazilian iniciative to increase contribution to open-source/free projects.

Ajude a sustentar a Wikipédia e outros projetos, sem colocar a mão no bolso, e concorra a um Eee PC!
…e também a pen drives, card drives, camisetas geeks, livros e mais! O BR-Linux e o Efetividade lançaram uma campanha para ajudar a Wikimedia Foundation e outros mantenedores de projetos que usamos no dia-a-dia on-line. Se você puder doar diretamente, ou contribuir de outra forma, são sempre melhores opções. Mas se não puder, veja as regras da promoção e participe – quanto mais divulgação, maior será a doação do BR-Linux e do Efetividade, e você ainda concorre a diversos brindes!

Uma ótima iniciativa, muito criativa… pelo que vi, já ultrapassa de 850 (+1 com este, hehe) o número de blogs participantes

mysqldump – GSoC Weekly Report – Week 7

PROJECT: Adding parallelism to mysqldump
WEEK 07

KEY ACCOMPLISHMENTS LAST WEEK
* Completed midterm evaluations
* Finally was able to create a branch with bazaar, commit my changes and tried submitting a patch to my mentor. There were some problems sending the patch, but it should be done by tonight)

KEY TASKS THAT STALLED LAST WEEK
* Still couldn’t solve the out-of-memory error. Now with my code in a branch will be easier to get help from the community.

KEY CONCERNS
* Need to fix this error ASAP and move on with my project.

TASKS IN THE UPCOMING WEEK
* Solve the out-of-memory error
* Submit a patch with bzr
* If time allows, include my code in the source tree after solving the error and doing more testing

mysqldump – GSoC Weekly Report – Week 6

PROJECT: Adding parallelism to mysqldump
WEEK 06

KEY ACCOMPLISHMENTS LAST WEEK

None.

KEY TASKS THAT STALLED LAST WEEK

Out-of-memory error when dumping big databases

KEY CONCERNS

I couldn’t fix the out-of-memory error. It only happens when I have DBs with big tables. The error happens because memory to the dynamic string could not be allocated. Hope I can fix this soon and move on with the project.

TASKS IN THE UPCOMING WEEK

* Solve the out-of-memory error
* Start using BZR to submit patches, and also submit to internals mailing list, to get feedback from the community
* Start refining the prototype to add a command-line parameter to specify the number of threads

mysqldump – GSoC Week Report #5

Hi all,

Classes are finally over… now I can dedicate more time to this project

Report for last week is a little ‘empty’…

–

PROJECT: Adding parallelism to mysqldump
WEEK 04

KEY ACCOMPLISHMENTS LAST WEEK
None.
Last week of exams at university took me a lot of time. I should have take that into account when doing my last report.

KEY TASKS THAT STALLED LAST WEEK
N/A

KEY CONCERNS
None. In fact, I’m glad I’ll be able to dedicate more time to the program now that classes are over

TASKS IN THE UPCOMING WEEK
(same as last week)

* Solve the out-of-memory error
* Refine the prototype to add a command-line parameter to specify the number of threads
* Fix some minor coding standard issues
* Start using BZR to submit patches, and also submit to internals mailing list, to get feedback from the community

That’s it…
Expect more to come this week

mysqldump – GSoC Weekly Report – Week 4

So, report for week #4 is here… a little late, though
Classes at MScs course are almost over … last exam is next tuesday, and then I’ll be able to dedicate more time to the program.
But this week I thought I got a lot of progress… the threaded mysqldump is working (with some problems when large tables are selected), and a speedup of 1,9x was measured with a DB of 600MB, in a quad-core machine. COOL!!!

–

PROJECT: Adding parallelism to mysqldump
WEEK 04

First of all, sorry this report is a little late…

KEY ACCOMPLISHMENTS LAST WEEK

* Reworked connect_to_db so that I can create one connection for each thread now.
* Reworked some functions to turn them into thread-safe. This implied eliminating global variables mysql, extended_row and insert_pat, that are now passed as parameters when needed.
* Finished a prototype where one thread is generated by table to be dumped. With a database of 600MB, a speedup of 1,9x was obtained in a quad-core machine. I’m having problems with larger databases. This prototype was also delivered as an assignment of a master course I’m doing.

KEY TASKS THAT STALLED LAST WEEK

* With big databases, the program is crashing, out of memory error:
mysqldump: Out of memory (Needed 1024 bytes)
mysqldump: Couldn’t perform DYNAMIC_STRING operation
I’m working to solve this now.

KEY CONCERNS

* Semester is almost ending… next week is the last one

TASKS IN THE UPCOMING WEEK

* Solve the out-of-memory error
* Refine the prototype to add a command-line parameter to specify the number of threads
* Fix some minor coding standard issues
* Start using BZR to submit patches, and also submit to internals mailing list, to get feedback from the community

GSoC Weekly Report – Week 3

PROJECT: Adding parallelism to mysqldump
WEEK 03

KEY ACCOMPLISHMENTS LAST WEEK

* Started implementing a new prototype, adding support to more threads. The inicial idea is to have one thread per table to be dumped, for now. This can be refined later.

KEY TASKS THAT STALLED LAST WEEK

* Couldn’t finish this new prototype. Actually, I’m having problems reworking the connect_to_db method, so that I can create one connection for each thread.

KEY CONCERNS

* I have to deliver this prototype working and with some speedup for a grad. course by the end of the week.
* Semester at grad ends by the end of June.. Exams, assignments

TASKS IN THE UPCOMING WEEK

* Finish this prototype
* Refine the prototype to add a command-line parameter to specify the number of threads, and manage the way each thread gets the tables they will be in charge of dumping
* Still need to patch the prototype and submit to internals@ to receive feedback

GSoC Weekly Report – Week 2

So, here it goes my new report…
Until the end of June, when my semester at grad. ends, things are going to progress slowly

PROJECT: Adding parallelism to mysqldump

KEY ACCOMPLISHMENTS LAST WEEK

* Implemented and tested the second prototype, that starts a single thread and runs the dump code inside it.
* Set up bazaar repository

KEY TASKS THAT STALLED LAST WEEK

* Couldn’t blog more and work as I wish…
* Didn’t send the patch to internals mailing list

KEY CONCERNS

* I have to deliver a prototype supporting lots of threads running simultaneously and with some speedup for a grad. course by June, 18;
* Semester at grad ends by the end of June.. Exams, assignments

TASKS IN THE UPCOMING WEEK

* Refine the prototype to add support for more threads – synchronization, passing the correct parameters to each thread
* Patch prototype and submit to internals@ to receive feedback

Daniel.

GSOC Weekly Report – Week 1

New week… new report

KEY ACCOMPLISHMENTS LAST WEEK

* Identified in the source code where parallelism could be exploited
* Implemented first prototype – some global variables that are going to be passed to the threads were identified and passed as parameters to the dump methods.
* Tested first prototype.
* Communicated via IRC with my mentor (just via e-mail previously) – timezone doesn’t help
* Discovered some related script-based tools. Thanks to Giuseppe Maxia for the tip!

KEY TASKS THAT STALLED LAST WEEK

* Some problems working with a mysql development build for the very first time, but already ok.

KEY CONCERNS

* Semester at grad ends at the end of June.. Exams, assignments

TASKS IN THE UPCOMING WEEK

* Set up bazaar repository
* Patch the first prototype and submit to internals@ to receive feedback
* Implement next step: use pthreads to create a single thread that executes just the dump related code. And refine this as much as I can to support more threads – synchronization, etc..)
* Blog more… not just the week reports

I hope I can post more this week!!!
After all, reports aren’t the most exciting things in the world