Planet OSDB

Planet MySQL: PDO: Learn how the PDO SQL parser causes bug reports

Thu, 03 Jul 2008 20:41:05 +0000

The PDO SQL parser causes many bugs. All drivers suffer from it. By default the parser replaces placeholder with '<bound_value>'.'<bound_value>' is a string. This can clash with the SQL syntax. PDO can make an INSERT fail. Please learn why and stop reporting bogus bugs.

Why does PDO parse my SQL commands?

PDO has chosen Prepared Statements as its preferred way of running database queries. The entire API is build around prepared statements. Wez has explained that primarily security considerations have caused this decision. Read PDO_MYSQLND: Prepared Statements, again for a discussion of prepared statements in the context of PDO.

Not every database system supports prepared statements. And those who do use a different syntax for placeholder. Some systems support named parameters (SELECT * FROM test WHERE id = :name), some support positional/ordinal parameter (SELECT * FROM test WHERE id = ?) and some support both. The cause of the differences is the lack of a standard. PDO tries to hide the differences from the user. This is done by a prepared statement emulation. The emulation includes a SQL parser which allows you to use whatever format you prefer for placeholder. If the underlying database system does not support the placeholder format you have chosen, PDO emulates it for you.

In many cases the emulation works flawless. But the bug databases on bugs.php.net and pecl.php.net show plenty of reports that boil down to the PDO SQL parser. Please learn how the SQL parser works. This should help avoiding duplicates in the bug database.

(...)
Read the rest of PDO: Learn how the PDO SQL parser causes bug reports (1,030 words)

Planet MySQL: Notable no-shows at SourceForge.net Community Choice Award finals

Thu, 03 Jul 2008 20:06:58 +0000

Maybe I shouldn't be surprised, but not a single project from Microsoft's Codeplex made it to the finals of the SourceForge.net Community Choice Awards (CCAs). When Microsoft was announced as a sponsor, the press release highlighted that the CCAs would welcome any project nomination, including projects hosted at Codeplex. Most of you know that Codeplex is the home for open source projects predominately targeted at Microsoft environments. Maybe the news of the CCA nomination process didn't reach the Codeplex user community as this was the first year that the CCAs were opened to projects regardless of their home? Also, I'm... READ MORE

Planet MySQL: MySQL support fun, multiplication

Thu, 03 Jul 2008 19:45:08 +0000

There was a question how to do an aggregate multiplication in MySQL. MySQL does not provide such functionality, so we were looking at various workarounds.

We discussed UDF interface that allows to construct custom aggregates, also did look at @a:=@a*field hack, and how different initializers have results wrapped differently.

Then Scott killed our discussion with this simple query:

select exp(sum(log(c)))

Of course, thats nice mathematical approach to solve the multiplication issue with just SUM() at hands, but while we were still in awe, Scott explained it with this wit:

The Great Flood is over, and as the animals are departing 3×3, Noah is blessing each, saying “Go forth, be fruitful and multiply.” Two snakes come down the ramp and say to Noah, “We can’t. We’re adders.”

Noah groans and says, “That’s the worst pun I’ve heard in 40 days and nights. Go sit in that pile of sticks until I can deal with you!”

After mucking out the ark, Noah returns to the sticks and lifts them up, to find baby snakes everywhere. “What happened?” he asks. “I thought you were having problems!”

The snakes reply, “Even adders can multiply with logs.”

:-)

Planet MySQL: How to load large files safely into InnoDB with LOAD DATA INFILE

Thu, 03 Jul 2008 19:23:27 +0000

Recently I had a customer ask me about loading two huge files into InnoDB with LOAD DATA INFILE. The goal was to load this data on many servers without putting it into the binary log. While this is generally a fast way to load data (especially if you disable unique key checks and foreign key checks), I recommended against this. There are several problems with the very large transaction caused by the single statement. We didn't want to split the file into pieces for the load for various reasons. However, I found a way to load the single file in chunks as though it were many small files, which avoided splitting the file and let us load with many transactions instead of one huge transaction.

The smaller file is 4.1GB and has 260M lines in it; each row is just two bigints. The bigger file was about 20GB and had wider rows with textual data and about 60M lines (as I recall).

When InnoDB loads the file, it creates one big transaction with a lot of undo log entries. This has a lot of costs. To name a few:

the big LOAD DATA INFILE clogs the binary log and slows replication down. If the load takes 4 hours on the master, it will cause the slave to fall 4 hours behind.
lots of undo log entries collect in the tablespace. Not only from the load -- but from other transactions' changes too; the purge thread cannot purge them, so everything gets bloated and slow. Even simple SELECT queries might have to scan through lots of obsolete, but not-yet-purged, row versions. Later, the purge thread will have to clean these up. This is how you make InnoDB behave like PostgreSQL
If the undo log space grows really big, it won't fit in the buffer pool and InnoDB essentially starts swapping between its buffer pool and the tablespace on disk.

Most seriously, if something should happen and the load needs to roll back, it will take a Very Long Time to do -- I hate to think how long. I'm sure it would be faster to just shut everything down and re-clone the machine from another, which takes about 10 or 12 hours. InnoDB is not optimized for rollbacks, it's optimized for transactions that succeed and commit. Rollback can take an order of magnitude longer to do.

For that reason, we decided to load the file in chunks of a million rows each. (InnoDB internally does operations such as ALTER TABLE in 10k row chunks, by the way; I chose 1M because the rows were small). But how to do this without splitting the file? The answer lies in the Unix fifo. I created a script that reads lines out of the huge file and prints them to a fifo. Then we could use LOAD DATA INFILE on the fifo. Every million lines, the script prints an EOF character to the fifo, closes it and removes it, then re-creates it and keeps printing more lines. If you 'cat' the fifo file, you get a million lines at a time from it. The code is pretty simple and I've included it in Maatkit just for fun. (It's unreleased as of yet, but you can get it with the following command: "wget http://www.maatkit.org/trunk/fifo").

So how did it work? Did it speed up the load?

Not appreciably. There actually was a tiny speedup, but it's statistically insignificant IMO. I tested this first on an otherwise idle machine with the same hardware as the production machines. First, I did it in one big 4.1GB transaction, then I did it 1 million rows at a time. Here's the CREATE TABLE:

PLAIN TEXT

SQL:

CREATE TABLE load_test (
col1 bigint(20) NOT NULL,
col2 bigint(20) DEFAULT NULL,
KEY(col1),
KEY(col2)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

Here's the result of loading the entire 4GB file in one chunk:

PLAIN TEXT

CODE:

time mysql -e "set foreign_key_checks=0; set sql_log_bin=0; set unique_checks=0; load data local infile 'infile.txt' into table load_test fields terminated by '\t' lines terminated by '\n' (col1, col2);"
real 234m53.228s
user 0m1.098s
sys 0m5.959s

While this ran, I captured vmstat output every 5 seconds and logged it to a file; I also captured the output of "mysqladmin ext -ri5 | grep Handler_write" and logged that to a file.

To load the file in chunks, I split my screen session in two and then ran (approximately -- edited for clarity) the following in one terminal:

PLAIN TEXT

CODE:

perl mk-fifo-split infile.txt --fifo /tmp/my-fifo --lines 1000000

And this in the other terminal:

PLAIN TEXT

CODE:

while [ -e /tmp/my-fifo ]; do
mysql -e "..... same as above.... ";
sleep 1;
done

After I was done, I ran a quick Perl script on the vmstat and mysqladmin log files to grab out the disk activity and rows-per-second to see what the progress was. Here are some graphs. This one is the rows per second from mysqladmin, and the blocks written out per second from vmstat.

And this one is the bytes/sec from Cacti running against this machine. This is only the bytes out per second; for some reason Cacti didn't seem to be capturing the bytes in per second.

You can see how the curves are roughly logarithmic, which is what you should expect for B-Tree indexes. The two curves on the Cacti graph actually show both files being loaded. It might seem counter-intuitive, but the second (smaller) curve is actually the larger file. It has fewer rows and that's why it causes less I/O overall.

I also used 'time' to run the Perl fifo script, and it used a few minutes of CPU time during the loads. So not very much at all.

Some interesting things to note: the load was probably mostly CPU-bound. vmstat showed from 1% to 3% I/O wait during this time. (I didn't think to use iostat to see how much the device was actually used, so this isn't a scientific measurement of how much the load was really waiting for I/O). The single-file load showed about 1 or 2 percent higher I/O wait, and you can see the single-file load uses more blocks per row; I can only speculate that this is the undo log entries being written to disk. (Peter arrived at the same guess independently.)

Unfortunately I didn't think to log the "cool-down period" after the load ended. It would be fun to see that. Cacti seemed to show no cool-down period -- as soon as the load was done it looked like things went back to normal. I suspect that's not completely true, since the buffer pool must have been overly full with this table's data.

Next time I do something like this I'll try smaller chunks, such as 10k rows; and I'll try to collect more stats. It would also be interesting to try this on an I/O-bound server and see what the performance impact is, especially on other transactions running at the same time.

Entry posted by Baron Schwartz | One comment

Add to: | | | |

Planet MySQL: JBoss plays the choice card

Thu, 03 Jul 2008 18:45:53 +0000

News from Sacha (and covered by InfoWorld) that JBoss Application Server 5.0 is close to GA kicked off a debate at TSS. Some commented that they were "Suspicious of anything that takes three years to develop," while others questioned if there was anything new in JBoss AS5 that SpringSource and GlassFish (or for that matter Apache Geronimo) hadn't already delivered. Others congratulated JBoss on closing in on JEE5 certification and refactoring their runtime to be more flexible. What caught my attention is the way that Sacha (JBoss CTO) responded to two comments from Douglas Dooley. When Douglas suggested that JBoss... READ MORE

Planet MySQL: Google, YouTube, Data

Thu, 03 Jul 2008 13:53:36 +0000

While reading my RSS feeds this morning I picked up this:
YouTube Must Give All User Histories To Viacom

After Scientology's DMCA request on Slashdot we made an active choice to squash data on users to limit the possibility of this sort of request. We randomized incoming trackable data on users and tossed everything but aggregate data for long term storage.

Why?

For one, we simply did not need to keep terabytes of log data sitting around collecting dust. Secondly, while the data might be useful for determining trends we risked our user's privacy. We believed this was unacceptable. One court request and we could be handing over who knows what to any company that could find an uneducated judge to sign away the privacy of millions. The data was just not that valuable.

What would I like to see?

Sites handing control of data retention over to users.

It would be good to see more sites give users the opportunity to have their tracking information removed after a period of time from companies databases.

We could start a trend by having websites publish data retention policies .

So what would it take to make this happen? Would peer pressure work? I am not in favor of creating more laws.

What if we petition sites to make steps in this direction. A few at a time, with a goal of long term of putting peer pressure on sites that do not follow the lead of user privacy oriented sites.

Is this too much too ask for?

Planet MySQL: FORCE INDEX

Thu, 03 Jul 2008 11:36:22 +0000

SELECT ... FROM
tblProduct as prd FORCE INDEX(vProductName,iCategoryId,eProductType,vProductImage,fPrice,eProductStatus,dProductDateAdded)
LEFT OUTER JOIN (SELECT bidjoin.* FROM tblProduct as prdjoin, tblBid as bidjoin WHERE bidjoin.iProductId = prdjoin.iProductId) bid ON prd.iProductId = bid.iProductId
, tblCategory as cat FORCE INDEX(vCategoryName,eCategoryStatus,dCategoryDateAdded) 
, tblAuctionProduct as aucprd FORCE INDEX(iRequiredBids,iProductId,eAuctionType,iFreeBidLevel,fAdminFee,dDateClosed)
, tblPurchaseProduct as purprd FORCE INDEX(dProductCloseDate,iProductId,iTotalInventory,dLastUpdated)

I know, unreadable. But the main issue today is the FORCE INDEX. This is just one query (real, with permission - I just adjusted some identifiers) from an app that is litterally full of queries using FORCE INDEX. So what does FORCE INDEX do? It forces the MySQL optimiser to use the specified index (or choose from one of the specified, if multiple), even if it reckons it's not the best choice. Likewise there's an IGNORE INDEX modifier that denies the optimiser the choice for using a specified index in a particular query.

I generally hold that these modifiers should only be used for testing things and tracking down optimiser issues (not that common these days, but 3.23 had plenty that 4.0 fixed). This because when you hardcode them into an app, perhaps fixing a real problem you see *now*, your data will still change over time and so the optimiser needs the freedom to change its choices too. It might be ok for a week, a month, or even a year, but at some point it's going to cost.

The above query takes about 5 times as long compared to the same query without the FORCE modifiers. Considering the app is chockers with it, I'm guessing that it might have been coded by someone who didn't know that MySQL actually has an optimiser at all. Who can say... anyway it serves here as an extreme example.

The take-away message is: use these modifiers with care, and be very hesitant ever adding it to production code. There's generally always a better way to resolve whatever problem you might be observing.

Planet MySQL: Memcached for MySQL Webinar: Advanced Use Cases

Thu, 03 Jul 2008 10:39:00 +0000

Today at 1PM EST I am presenting the second part of memcached for MySQL webinar. I was told that the registration numbers look as good as the previous one. This one will be a bit more technical than the previous webinar. Sorry for the late notice but hope you can join!

SimpleDBM: Update on SimpleDBM's Data Dictionary

noreply@blogger.com (Dibyendu) — Thu, 03 Jul 2008 09:05:11 +0000

I have chosen to implement option 3 as described in a previous post. For each table, a special container is created, which is used to store not only the table structure but also structures of all associated indexes. Special containers are created for indexes as well but these contain pointers to the table structures. (The reason we need the index definition containers is that during recovery an index update may be performed prior to a table update.)

A memory cache is maintained of the entire data dictionary. When a new table is created, its structure is immediately saved into the memory cache. The structure is also persisted using a PostCommitAction - which is a type of redo only log that is executed once the transaction commits.

At system restart, the storage system is scanned for containers that have a specific naming convention and are in a well known location. This way, SimpleDBM is able to initialize the memory cache at startup. Note that this initialization occurs post recovery. The list of open containers is consulted to ensure that only those definitions are loaded that are relevant.

There is a problem that during recovery, any redo/undo operation on a table or index will require access to the table/index definition. Since the data dictionary memory cache is not available at this stage - it is populated after recovery, the solution is to retrieve the definitions on demand.

The table/index creation process uses a standalone transaction. During this time the container IDs are exclusively locked. This ensures that by the time any subsequent transaction performs a data manipulation operation against the table, the data dictionary memory cache has been updated to contain the table definition.

At present tables cannot be dropped, as this has not yet been implemented.

Along with the Data Dictionary implementation, there is also a high level Database API that is available. This hasn't been released yet as it is still being tested. I am hoping that a fully tested version will be available towards the later part of this year.

Planet MySQL: PDO_MYSQLND: Calling Stored Procedures works fine with mysqlnd

Thu, 03 Jul 2008 08:21:57 +0000

Mike is asking I`m hoping that MySQL stored procedures will work properly. Especially multiple calls in one request. in reply to PDO_MYSQLND: R[a|u]mbling and a breeze of progress. Mike, it depends what you mean by "properly". If you use the API properly it works fine with mysqlnd. Read on for code examples.

	mysqlnd		libmysql
	Emulated PS	Native PS	Emulated PS	Native PS
buffered fetch	PASS	PASS	PASS*	PASS*
unbuffered fetch	PASS	PASS	PASS*	PASS*
			* See below!

Executing a basic stored procedure twice

Code take from the internal test pdo_mysql_attr_oracle_nulls.phpt. Status: PASS. NOTE the use of $stmt->nextRowset(). If you do not call $stmt->nextRowset() in a loop to fetch all result sets you will get the usual 2014 error. MySQLs Stored Procedures always return n (here: 1) result sets for the actual data followed by another result set with the status code. You must use PDOStatement::nextRowset() to fetch them.

(...)
Read the rest of PDO_MYSQLND: Calling Stored Procedures works fine with mysqlnd (673 words)

Planet MySQL: Two Tech Jobs: Technology Evangelist and Network Operations

Thu, 03 Jul 2008 05:16:05 +0000

I've been getting lots of inquiries from recruiters recently looking to find good people for various tech companies. Two in particular might be interesting if you're in the market or know someone who is.

Technology Evangelist at New York City Based Startup

If you saw Fred Wilson's post Are You A Connector?, you know a bit about this job already. It's a NYC based startup developing a new platform in an area that's likely to see serious growth in the next few years.

They're looking for someone with coding experience who loves showing other developers and users how stuff works: on stage, via blogs, in screencasts, and so on. It's important that this person have a technical (programming) background and also be very comfortable getting in front of people to demo and speak.

The company is New York based and this job is too. However, I'm told that a Bay Area candidate may work as well, since a presence here will be important. Otherwise, I'm sure that frequent travel to the Bay Area will be part of the gig.

The company is still a bit stealthy but more information will be forthcoming soon.

Ping me if you're interested and I'll put you in touch.

Network/Systems Administrator at Rapleaf in San Francisco

The folks at Rapleaf ping me from time to time looking for talented engineers too. Here's the description of the position they're currently hiring for.

Rapleaf is a well-funded San Francisco startup (we’re currently at 15 people). We gather publicly available information from the social web on hundreds of millions of people to enable developers and companies to give their consumers a better user experience. Rapleaf has built the largest portable social graph in the world. We provide rich insight on customers for clients such as retailers, airlines, hotel chains, social networks, lead gen firms, telcos, political campaigns, financial services, and more (these companies learn about their consumers in order to give them a better user experience). The company has processed over 175 million unique searches for businesses and consumers.

You will maintain Rapleaf’s entire infrastructure and enhance the system to do great things as we’re on the trajectory to change the world. Helping grow one of the largest and most complex databases for a small start-up.

Role:

Manage all Rapleaf servers (Linux – CentOS, Redhat), backups, web servers (Apache clusters)

Manage relationship with hosting provider and hardware vendors

Scalability and expansion (Hadoop)

Systems administration (DNS,LDAP,NFS,TCP/IP,SELinux)

Some scripting (Shell, Ruby, Python, or Perl)

Administer MySQL databases (multi-master replication, snapshot backups)

Learn how to scale with Ruby on Rails

Manage complex Java systems

Manage billions of data items of pages being served

On-call duty

Note: this job is really hard. You’ll be working with some of the top search engineers in the world and they are going to expect that you kick ass. We’re doing things that no one has ever done before and solving problems that have been open for years.

Qualifications:

Master of all things Internet and Linux.

Incredibly smart, can learn fast, and takes no prisoners
Learn new platforms fast. We use Ruby on Rails and Java … you can pick this up quickly.

Intensely driven and proactive person.

Extremely hard working. This is a start-up - team members work long hours.

Quick learner and real doer. Err on execution over strategy.

Thrive on working with A-players. Too good to spend long hours with B-players.

Likeable person who garners respect on and off the job.

Thrive on chaos, risk, and uncertainty.

Should be easy to get along with, nice, fun, smart, ethical, and low-maintenance.

Strong desire to build a more ethical society.

Desire to be an early employee and want to be a real owner in Rapleaf’s future.

Want to work with extremely large datasets and indirectly build portable APIs that thousands of other companies can build applications on top of.

Ability to lift and install servers (50 lbs)

Should want to live in or near San Francisco (relocation available if necessary)

Perks:

Good salary/stock compensation.

Personal MacBook Pro or Linux based machine

Medical insurance, 401k.

Kitchen stocked with food

Work with some of the smartest engineers

Contribute to the Rapleaf Dev blog (http://blog.rapleaf.com/dev/)

Again, ping me if you're interested or know someone who might be. I'll make an introduction for you.

(comments)

Planet PostgreSQL: Kenny Gorman: Python script showing PostgreSQL objects in linux memory: pg_osmem.py

Thu, 03 Jul 2008 01:21:44 +0000

I got some email and comments about the code I used for my post; Mapping Linux memory for PostgreSQL using fincore so I thought I would post the code I am using. Nothing too fancy here, it needs a config file and some more bits. I highly recommend someone do this in perl vs python and incorporate fincore as a module instead. I used psycopg2, other than that, all the other modules are stock with python 2.5.2. If someone wants to show me how to query the data directory PostgreSQL is currently using and incorporate that into the script vs the static string: mydir = “/data/mydb/base”, that would be great. In order to use this script, you must change the variables for fincore, and mydir below.

 
#!/home/postgres/python/bin/python
#
# script to find show memory usage of PG buffers in OS cache
# 2008 kcg
#
import os
import psycopg2
import commands
import re
import sys
from optparse import OptionParser
 
parser = OptionParser()
parser.add_option("-u","--username",dest="username",help="username for PostgreSQL")
parser.add_option("-m","--machine",dest="machine",help="machine to connect to.. aka: hostname")
parser.add_option("-d","--dbname",dest="dbname",help="database name to connect to")
parser.add_option("-p","--password",dest="password",help="password for PostgreSQL")
(options, args) = parser.parse_args()
 
osmem   ={}
# change these to match actual locations
fincore ="/home/kgorman/fincore.pl"
mydir   = "/data/mydb/base"
 
# get list of dbs on host, and return dictionary of db=oid sets
def lookup_dbs():
 dbs={}
 connectstr="host="+options.machine+" dbname="+options.dbname+" user="+options.username+" port=5432 password="+options.password
 handle=psycopg2.connect(connectstr)
 curs=handle.cursor()
 sql="select datname,oid from pg_database where datname = '"+options.dbname+"' and datname not like '%template%'"
 curs.execute(sql)
 for d in curs.fetchall():
   dbs[d[0]]=d[1]
 return dbs
 
# get object
def lookup_oid(oid,dbname):
 connectstr="host="+options.machine+" dbname="+dbname+" user="+options.username+" port=5432 password="+options.password
 handle=psycopg2.connect(connectstr)
 curs=handle.cursor()
 sql="select relname from pg_class where oid = "+oid
 curs.execute(sql)
 for d in curs.fetchall():
  return d[0]
 
dbs=lookup_dbs()
for v, i in dbs.iteritems():
  for ii in os.listdir(mydir+"/"+str(i)):
     p = re.compile('\d')
     if p.match(ii):
     	#print ii
        rel=lookup_oid(ii,v)
        fullpath=mydir+"/"+str(i)+"/"+ii
	cmd=fincore+" "+fullpath
	#print cmd
	pages=commands.getstatusoutput(cmd)
	#print pages
	n=pages[1].split(' ')
        size=n[1]
	if p.match(size):
           if rel:
	     osmem[v+":"+rel]=(int(size)*1024)
 
# sort and output
sdata=sorted(osmem.iteritems(), key=lambda (k,v): (v,k), reverse=True)
a=0
print "OS Cache Usage:"
while a < len(sdata):
  print sdata[a][0]+":"+str(sdata[a][1])
  a=a+1

Planet MySQL: MySQL Sandbox 2.0 has been released

Wed, 02 Jul 2008 23:30:00 +0000

Taking advantage of Launchpad excellent development services, I completed the tasks necessary to release MySQL Sandbox 2.0.
So, you may ask, what's new? Quite a lot, actually:

more intuitive script names
master sandbox directory
faster
supports circular replication
more error checking
includes a test suite
uses a default options file

more intuitive script names

One of the problem in the old sandbox was that the names of the scripts suggested some installation of the sandbox itself. This has been addressed in version 2.0. You now have the very clear names:

 make_sandbox                  creates a single sandbox
make_replication_sandbox      creates a replication sandbox
make_multiple_sandbox         creates sandbox containing many servers
                                of the same version
make_multiple_custom_sandbox  creates a sandbox containing many servers
                                of different versions

master sandbox directory

By default, MySQL Sandbox will create a directory under your home, named "sandboxes", and all sandboxes will be created under that directory. In addition of making your sandboxes more organized, the master directory provides some handy scripts to deal with many sandboxes at once. You can start, stop, or send a SQL command to all the sandboxes with just one command.

faster

The Sandbox had a few "sleep" instructions, to wait for the server being installed to come online. This solution was slowing down fast machines, where the server can start much faster than the standard 3 seconds I added to the sandbox, but was sometimes not enough for slower machines, where the server may take more than 3 seconds to start up, especially with replication. The new sandbox uses a loop coupled with a timeout limit. Thus, quick machines are faster, and slow machines don't fail.

supports circular replication

Now you can create a replication ring.

./make_replication_sandbox --topology=circular --how_many_nodes=5 VERSION
# or
./make_replication_sandbox --master_master VERSION

includes a test suite

If you have resources and MySQL binary tarballs at your disposal, you can stress test MySQL Sandbox (and your box!) with this test suite. For example, assuming that your 5.0.51 and 5.1.25 binaries are in $HOME/opt/mysql

./test/test_sandbox.pl --versions=5.1.51,5.1.25

This will create 22 sandboxes

more error checking

This is a direct consequence of the above item. Having a test suite has made the sandbox much more robust.

uses a default options file

There is also a default option file, where you can keep the options that you need to be executed always.

Try it!

Go get the 2.0 tarball from launchpad. If you find any bugs, there is a very convenient bugs reporting system. Please use it!

Planet MySQL: db4free.net again offers latest MySQL 6.0 (compiled from source)

Wed, 02 Jul 2008 22:20:00 +0000

The db4free.net's MySQL 6.0 server is at the very latest state again. I have compiled the server from source, so you can test the very latest of MySQL development.

Unfortunately there were some issues with the former version (6.0.4) and I didn't manage to simply upgrade the server as I used to. After upgrading, the server didn't start up with the old data directory and dumping database by database to re-import it into the new version resulted in frequent crashes. After multiple failures to do a straight update I decided to install a fresh and empty new 6.0 server.

Now the most important question for you if you used the 6.0 server: where did my data go?

I left the old server up and running, but configured it to port 3308. This means, if you had data in your MySQL 6.0 database, you can still access it - here's how, if you use the default MySQL client:

mysql -h db4free.net -P 3308 -u [your_username] -p[your_password] [your_database]

I am also going to configure phpMyAdmin to allow you accessing the old MySQL server using it.

If you want to keep your data on the 6.0 server, please migrate it to the new one (running on port 3307 as usual) as soon as possible. I will keep the old server (running 6.0.4) running for about a month.

Have fun experimenting with the very latest MySQL 6.0!

Planet MySQL: Stay up to date with what's cooking@MySQL: RSS feeds galore

Wed, 02 Jul 2008 21:18:14 +0000

Did you know that many parts of the MySQL web sites provide news and updates via RSS Feeds? Markus Popp from our web team did a great job on making some of these more visible by adding RSS icons to the respective pages. If you want to keep up with what's happing at MySQL, consider adding the following feeds to your feed reader:

New downloads on dev.mysql.com
New DevZone articles
PlanetMySQL (You probably knew that one already)
MySQL Forge - All New Items
MySQLForge Wiki - Recent changes
MySQL Announcements on Launchpad.net
Branch activity on Launchpad.net
New MySQL Training Courses
MySQL Live Web Seminars
MySQL OnDemand Web Seminars
Latest Quickpolls from MySQL AB
Latest Updates from MySQL AB
Latest Press Articles about MySQL AB
The MySQL Bug database allows you to subscribe to the results of any search using the 'Feed' link that can be found on the search result page.
The MySQL Forums provide an RSS feed for each forum, e.g. the Announcements Forum

Did I miss any? Do you know any other feed? Please let me know.

RSS feed readers exist for a wide range of platforms. I personally use Akregator, which is part of the KDE Desktop environment.

Planet PostgreSQL: Jignesh K. Shah: wal_buffers and PostgreSQL 8.3.3

Wed, 02 Jul 2008 21:11:20 +0000

Recently Felix Malinkevich joined the ISV-E Open Source Team at Sun as an Intern working on PostgreSQL [Read More]

Planet MySQL: PDO_MYSQLND: R[a¦u]mbling and a breeze of progress

Wed, 02 Jul 2008 18:36:37 +0000

The modification of PDO_MYSQL to support the MySQL native driver for PHP (mysqlnd) is progressing. We are using the project title “PDO_MYSQLND” for the modification. The goal of PDO_MYSQLND is to provide a PDO driver for MySQL which can be compiled either against the MySQL Client Library or against the MySQL native driver for PHP. This is the same type of modification we did with ext/mysql and ext/mysqli already.

The use of any of the libraries is transparent for the PHP user. You may continue to use the MySQL Client Library, like you do today with PDO_MYSQL, or give mysqlnd a try. The MySQL native driver for PHP (mysqlnd) is easier to compile as its tightly integreated into the PHP internals and ships with PHP as of version PHP 5.3. In case of ext/mysql and ext/mysqli our benchmarks and first user feedback indicates that mysqlnd gives you at least the same performance of libmysql. And sometimes it is faster and more memory efficient. Read below and see my blog archive for more on mysqlnd.

PDO_MYSQLND test results

Finally, the PDO_MYSQLND test results are looking fine. On a 64bit Linux box PDO_MYSQLND compiled against mysqlnd passes 98.8% of the tests and the code coverage is above 90%. When compiled against libmysql, the results are a little worse: 97.7%. In case of mysqlnd, the remaining 1.2% test failures are PDO (not driver) bugs. We are making use of the new --XFAIL-- feature of the PHP test tool run-tests.php to mark them as expected failures as long as we do not have the time to fix PDO itself as we need to finish our driver itself.

(...)
Read the rest of PDO_MYSQLND: R[a|u]mbling and a breeze of progress (2,040 words)

Planet MySQL: Book link, wee!

Wed, 02 Jul 2008 17:59:14 +0000

Finally got my copy of High Performance MySQL 2nd ed two days ago.

Yesterday I vaguely remembered Baaron asking if DPM had a homepage. So I flipped through the book a bit, and sure enough, on page 599, is a short mention of my proxy!

Thanks, Baaron! With the upcoming release of DPM it'll be exciting enough to be worth having a link in your awesome book :)

Planet PostgreSQL: Robert Hodges: PostgreSQL Gets Religion About Replication

Wed, 02 Jul 2008 17:56:53 +0000

The PostgreSQL community is getting really serious about replication. On Thursday May 29th, Tom Lane issued a manifesto concerning databas replication on behalf of the PostgreSQL core team to the pgsql-hackers mailing list. Tom's post basically said that lack of easy-to-use, built-in replication is a significant obstacle to wider adoption of PostgreSQL and proposed a technical solution based on log shipping, which is already a well-developed and useful feature.

What was the reaction? The post generated close to 140 responses within the next two days, with a large percentage of the community weighing in. It's one of the most significant announcements on the list in recent history. There is pent up demand for this feature and within a few hours people were already deep into the details of the implementation.

The basic idea comes from an excellent presentation by Takahiro Itagaki and Masao Fujii of NTT at PGCon 2008 in Ottowa. They have developed a system that replicates database log records synchronously to a standby database. The standby can recover quickly and without data loss, which makes it a good availability solution. The core team manifesto proposes to integrate this into the PostgreSQL core and add the ability to open the standby for reads.

So, is this the end of the story on replication? I don't think so. There's no question that synchronous log shipping with reads would be a great feature. Basic availability is the first problem users run into when setting up production systems and this feature looks considerably better than alternatives for other databases like MySQL. It will help if NTT donates their code to the community, but still the whole effort will take considerable time. Adding the ability to open a standby for reads is at least a version out (read: up to 2 years).

More importantly, log shipping is most useful for availability. It does not help you replicate across database versions (nice for upgrades), between different databases, from a master to large numbers of slaves, or bi-directionally between databases. Finally, it's a less than ideal solution for clustering data between sites, something that is rapidly becoming one of the most important overall uses of replication. For these and other cases you need logical replication, which turns log records into SQL statements and applies them using a client.

I'm therefore starting an effort to get logical replication hooks included as a parallel effort. If you are interested in this let me know. Meanwhile, stay tuned. Tom's message represents a real change of heart for the PostgreSQL community. Accepting the important of replication opens up the doors for a new round of innovation in scale-out based on PostgreSQL. It could not come at a better time.

Planet MySQL: mysql proxy 0.6.1 performance tests

Wed, 02 Jul 2008 17:11:13 +0000

The mysql proxy project has tremendous potential to make mysql administration and usage easier. I decided to throw some load at it to get a feel for how stable and performant it is.

On EC2, I set up 6 “small” images in an example proxy setup:

- One client machine to run sysbench
- One machine to act as a mysql proxy machine, running 0.6.1 (FC4 binary)
- Four identical database servers, running mysql 5.0.45

The database configuration was largely default, with InnoDB configured for 64MB buffer pool (just enough to ensure the sysbench table was entirely in memory), 512MB log files, and 1024 max connections.

mysql-proxy was run with the following command:

mysql-proxy –proxy-backend-addresses=ip-10-251-66-63.ec2.internal:3306 –proxy-backend-addresses=ip-10-251-71-21.ec2.internal:3306 –proxy-backend-addresses=ip-10-251-43-70.ec2.internal:3306 –proxy-backend-addresses=ip-10-251-75-52.ec2.internal:3306

Using the sysbench database benchmark utility to generate some heavy load on the boxes individually (to verify they were all more or less equal), and on the proxy itself, I got some interesting results:

Read/write operations/second:

Transactions/second:

Somewhat as I expected, mysql-proxy presents a bit of overhead when load is light. With only 8 concurrent threads, running against the proxy pointing to 4 equal database servers is slower than hitting one server itself.

As load goes up, however, you can see that the proxy scales reasonably well, while the accesses to the direct systems begin to tank rapidly after 64 concurrent threads (I lost some of my 128 thread results inadvertently and the instances are destroyed I’m fairly comfortable in interoplating the result to be roughly in between 64 and 256 )

Somewhat unsettling, however, was the fact that sysbench or mysql proxy failed to work beyond about 500 concurrent threads. My debug output for sysbench just stops here:

DEBUG: mysql_real_connect(0×9adec70, “ip-10-251-75-160″, “sbtest”, “(null)”, “sbtest”, 4040, “(null)”, CLIENT_MULTI_STATEMENTS)
DEBUG: mysql_init(0×9ae51b0)
DEBUG: mysql_options(0×9ae51b0, MYSQL_READ_DEFAULT_

On the servers themselves, I can see the 125+ connections to each individual box, and all looks normal. I tried tweaking some of the thread settings in ulimit and sysbench itself to no avail, but didn’t have a ton of time to play with it. I decided the results were already interesting enough and will see if I can revisit this another time.

Another reason i am not so concerned is that more than one proxy can be utilised in many typical environments, so the ability of a single mysql-proxy instance to handle 512+ concurrent connections might not be show-stopper.

If anyone out there has done some similar tests with mysql proxy, i welcome your feedback!

Planet MySQL: dbWatch News

Wed, 02 Jul 2008 17:03:04 +0000

Seems I have turned into a bit of a news source. dbWatch Software sent me a news release on their dbWatch monitoring platform, which looks like it might be an interesting product for those who work in a heterogeneous database environment. Here’s the release:

FOR IMMEDIATE RELEASE

Contact:

Rob Shuster, VP US Sales
dbWatch USA
6122B S. Eagle Pass Rd.
Gold Canyon, AZ
85218

Phone: 800-270-9892

http://www.dbwatchusa.com

info@dbwatchusa.com

dbWatch Software Announces US based Sales & Support Oslo, Norway — June 23, 2008 ? dbWatch Software has launched a US sales and support organization. The flagship product, dbWatch? (v8.1), is a heterogeneous database monitoring/reporting tool currently supports Oracle, SQL Server, and MySQL.

Marek Jablonski, CEO states ?As a DBA you are often stuck with multiple platforms (both OS and database systems) and multiple unmanageable tools from database providers. dbWatch? is a single tool allowing the DBA to monitor all databases and generate professional custom reports.?

Clear Channel Norway IT Director, Jan Erik Rasmussen, reports ?dbWatch provides an effective overview of our databases, and allows me to sleep well at night.?

Three dbWatch? editions are available: Standard ($195), Professional ($1,995 to $9,995), and Enterprise ($14,995). A free 30-day evaluation is available for download.

For information: http://www.dbwatchusa.com or

Contact: info@dbwatchusa.com

Phone: 800-270-9892

It was a timely contact, coinciding with Sheeri’s review of MONyog. It sounds like dbWatch is the kind of tool that would be valuable here at Pythian, where we work with all of the DBMSs it supports. But — I haven’t tried it yet, so I put it to you: have any of you used dbaWatch? If so, what are you opinions?

Planet PostgreSQL: Robert Hodges: What's Your Favorite Database Replication Feature?

Wed, 02 Jul 2008 16:40:34 +0000

Replication is one of the most flexible technologies available for databases. We are implementing a new open-source, database-neutral replication product that works with MySQL, Oracle, and PostgreSQL. Naturally we've done a lot of thinking about the feature set. It's tough to pick any single feature as the most important, but one that really stands out is optional statement replication. Here's why.

Database replication products tend to replicate row changes and DDL. However, Mark Callaghan has a great example of why you want to replicate statements as well--it enables Maatkit distributed consistency checking to work. If you dissect the mk-table-checksum --replicate command you will see that it uses a nice trick. The SQL queries generate checksums into the master table and then replicate as statements rather than row updates out to slaves. That way the slaves recompute the checksum locally at the same point in the overall transaction history. Very elegant!

Replicated consistency checks are a wonderful feature for large systems that can't afford to stop in order to compare tables between servers. However, you cannot use it if your database cannot replicate statements. As Mark points out, not even all MySQL engines do this. The proposed replication additions for PostgreSQL won't support it either.

Optional statement replication is really the best kind of feature: it is useful on its own, but also enables features like consistency checking and other nice administrative tricks. We're going to put a "worm-hole" in our replication engine that allows applications to invoke statement replication at the SQL level. Can you guess how we are going to do it? If not, you'll have to wait until we release. :)

So what's your favorite database replication feature?

Planet MySQL: The Value of Software Architecture

Wed, 02 Jul 2008 16:35:53 +0000

Every company, no matter how big or how small, already has software to manage their business. So if they are looking at opentaps Open Source ERP + CRM, it's because their system has failed them. That system may still meet some needs well, but as a whole it can no longer support the users' long-term needs.

Why this is happening? Usually for the following reasons:

The users have no access to the source code, so they have no way to make changes they need.
The system can't support new functionality. For example, it may have been built solely with direct marketers in mind and cannot be adapted for manufacturing or more complex B2B sales.
The source code is no longer maintainable because its original authors have moved on, and there's a lack of documentation and unit tests to help new developers work with it.
The system cannot support new technologies. It may have been written with a DOS-based rapid application development tool, which will never support the Internet.
The hardware which runs the system is no longer made or supported. For an extreme example of this, see this story about Ecometry and the HP3000.

What We Can Do (Better)

These problems show how important software architecture become over a long period of time. The most important thing about well architected software is that it can withstand change. It should allow you to modify, extend, or even replace parts of the software while the rest of it keeps humming along. In this way, a good architecture could extend the lifespan of your software and allow it to adapt to new technologies, new people, or new requirements.

For us, as developers of opentaps, it means that we need to produce a system that is not only feature rich and user friendly, but also a well architected. In this respect, there are somethings we already do well, and there are somethings that we need to do a lot better.

I think we already do the following well:

Our source code is openly available to all of our users, and this is a huge advantage, because it allows them to modify opentaps to meet their needs. In theory, this should actually be all they need, but in practice that is not the case. Having access to the source code doesn't actually do you any good unless the source code is also easy to work with and can support changes easily.
We have a very robust data model, which can support the core business needs of most industries.
Because opentaps is a newer system, it is built with current industry standard technologies such as Java, SQL databases, and XML.

However, there are some things that we need to do a much better job of. Much of our code is still too static and closely tied to the underlying data structure and one particular framework. (See this example.) What we really need is to adopt a more object-oriented design, so that it's easier for other developers to understand and extend opentaps, and so that we can be up-to-date with the best open source technical frameworks available.

Therefore, we have begun implementing a Domain Driven Design for opentaps, which is an object oriented design pattern that builds applications from business domains such as customers, orders, invoices, and shipments and separates the infrastructure tier of dealing with servers and databases from the business knowledge. We believe that this Domain Driven Design will give us -- and most importantly you -- the ability to extend opentaps, replace some parts of it with other systems, or upgrade opentaps to newer technical infrastructure, while preserving your investment in the rest of it.

The Plan for opentaps

This Domain Driven Design will be an evolutionary change from the current architecture. We will be moving our business tier code to the new design over the next few months while continuing to piggyback off the existing ofbiz framework. While doing this, we will continue to advance the development of opentaps version 1.4.

At some point, we will begin creating a new framework for opentaps version 2.0. This will be fully compatible with the ofbiz framework, allowing you (and us) to incorporate current and future versions of ofbiz and all the opentaps and custom applications we have already built with it. The drivers of the new opentaps framework, however, will more likely be a yet to be decided combination of open-source projects such as Hibernate, Spring, guice, JBoss Seam, Struts, Wicket, etc. It will allow us to re-factor, at our leisure, legacy ofbiz-based code to this new framework and at the same time incorporate other Java open source projects into opentaps more easily.

Planet MySQL: Erlang User Group Berlin

Wed, 02 Jul 2008 14:35:55 +0000

Berlin has a long history of blooming user groups and the Erlang User Group Berlin will be no exception!

That’s right, Germany’s capital and one of the nicest places to be in Europe (among all the other nice places, hooray for plurality) finally has place for Erlang enthusiasts. That is in fact a pleonasm, I haven’t found a single soul yet who is not enthusiastic about working with Erlang, but what do I know?

Anyway.

Join the discussion group and help to pick a date for meetings.

If you are an Erlang developer on visit in Berlin, let us know, so we can have an out-of-order meeting with you.

Sorry if that shows up at a seemingly non-related planet feed, this is a spam measure and won’t happen again. Thanks for your patience.

Planet MySQL: At OSCON

Wed, 02 Jul 2008 14:33:29 +0000

Boy am I thrilled.

OSCON, back in the old days, was the Mekka for open source developers. Glorious but out of reach and too far away. I don’t know if that image holds true still, but judging the enthusiastic reports by previous attendees, it does. And this year I will be able to attend.

I will not only be able to attend, I will even give a presentation. You can guess the topic of course and I don’t want to stretch this post into advertising CouchDB yet again. (I’ll leave that to other posts :-)

This is a heads-up for all the friends and colleagues I’ve known and worked with for quite a few years now but haven’t had a chance to meet in a while. If you are coming to OSCON, please get in touch (see the sidebar). For all the folks I have not yet met but want to say hi, do the same.

See you in three weeks!

—Jan

Planet MySQL: OSCON 2008 - Getting in touch with the open source makers

Wed, 02 Jul 2008 14:23:00 +0000

For some reasons, I have never been able to attend OSCON so far. I have been going to the MySQL Users Conference almost every year in April, and that usually left me without a great desire of crossing the Atlantic again. This year is different. I have crossed the Atlantic already three times, and yet I am about to board a plane once more, heading for the Portland, Oregon.

And for some lucky circumstances, I am also a speaker.
I will join Ronald Bradford on the podium, to speak about (surprise surprise!) MySQL Proxy, from architecture to implementation.
I will have also my hands full in the evenings. There will be plenty of BoFs (including two that I will host) and parties, social gatherings, competitions, which I plan to enjoy for my personal enlightenment, not only to fulfill my duties.
I am looking forward to meet old friends and to make new ones.

Planet MySQL: Gallery updates and new camera

Wed, 02 Jul 2008 14:05:56 +0000

Last weekend I finally found some time to upload pictures that I had taken during various events that I attended in the past few months. So here are my impressions from the following events:

These are probably the last pictures that I have taken with my trusty old Pentax Optio S4 - I just received my new camera, a Canon PowerShot A720 IS. I have just started to toy around with it, but the first results look promising! The Pentax served me well for several years - I've taken 9745 pictures with it. But it had a few deficiencies, particularly the slow startup and flash recharge time and the bad quality of pictures indoors bothered me for quite a while. But it is very small and handy and the metal housing makes it quite sturdy.

The first picture taken with my new camera: a picture of the old one.

And probably the last picture taken using the old camera: my new Canon.

Planet PostgreSQL: Hubert Lubaczewski: Who/what is trashing db performance?

Wed, 02 Jul 2008 13:40:04 +0000

Every so often I need to find who exactly is damaging database performance. I mean - I have db connection which does something strange/wrong with the database, but what exactly is responsible? Switching to separate user for every program, script and developer would solve the problem, but it is not always an option. So, what should [...]

Planet MySQL: Dublin sales meeting and group photo

Wed, 02 Jul 2008 13:18:29 +0000

The MySQL Telecom sales team is a global team, but it happens to be organized under the EMEA region of the MySQL sales department. Since the EMEA region is having a staff meeting in Dublin this week, it means also we in the telecom team are gathered here. A good opportunity to meet your globally dispersed collagues.

Zack Urlocker was the special guest star from headquarters. He presented a freudian-slip-slide saying that MySQL 5.1 will be released GA in July 2009. After lots of laughter he concessed that maybe engineering can pull it of faster, like maybe July 2008.

Another hot topic Zack talked about was to review how the integration with Sun is going...

...we concluded at least some of us are adapting just fine :-)

Planet MySQL: MONyog 2.5 RC has been released

Wed, 02 Jul 2008 12:47:55 +0000

Bug fixes and Enhancements:

Optimized SFTP code. This means that the Log Analyzer is now several times faster than previous Betas
Fixed an issue with trending
Log file parsing issue with MySQL 5.1.x resolved

Downloads: http://webyog.com/en/downloads.php
Purchase: http://webyog.com/en/buy.php

Planet PostgreSQL: Robert Hodges: MySQL, Sun, and the Future of Open Source Databases

Wed, 02 Jul 2008 11:38:43 +0000

So what's it like now that Sun now owns MySQL? The executive summary: a little weird. I was at the MySQL User Conference a couple of weeks ago and had a chance to talk with a lot of people in the community as well as many MySQL folks. Marten Mickos is now the head of database products at Sun. It's not very hard to figure out what Sun will do with MySQL products for the near future--pretty much what MySQL was doing already.

The real question for a lot of people is what will happen with databases like PostgreSQL and Derby. Sun has invested heavily in both of them, and PostgreSQL in particular is now quite fast. With the MySQL acquisition, Sun has an opportunity to run the table with multiple offerings that cover both enterprise applications as well as web and embedded. However, that would mean cutting down the MySQL roadmap to concentrate, for example, on scale-out rather than scale-up. It would also require thinking big to combine with other vendors in order to disrupt the market leader Oracle. Done right, there's a chance to upend the industry in a way that has not occurred since Microsoft muscled into databases in the early 1990s using code bought from Sybase.

Based on talks from people like Rich Green and Marten Mickos, it's hard to see this happening. Sun is taking a hands-off approach to MySQL *and* giving MySQL management control of overall database strategy. A disruptive change therefore seems unlikely. In fact, the more likely result is stagnation, now that MySQL no longer has to fight for its existence. The MySQL roadmap is still pretty diffuse and there has been little product movement since the 2007 User Conference. MySQL 5.1 is still not out the door. Falcon is likely to show up ready for production use around the time the Boeing Dreamliner rolls out. MySQL is still working on multiple storage engines (2 new ones plus NDB and MyISAM, to name a couple.) There's not even a glimmer of a date for cool new replication features like a pluggable replication interface. In short, not much evidence for radical changes of any kind.

Also, there must be the awful temptation to focus on vertical scaling so that MySQL can work on Sun hardware with large numbers of cores. I asked Marten Mickos specifically about the choice between scaling up and scaling out but didn't get a very clear answer. Personally I think for MySQL to concentrate very hard on vertical scaling would be a strategic error. The community that made MySQL great is into commodity hardware and scale-out in a big way. First rate support for highly scaled SMP architectures is going to be a long slog that will compromise delivery of many other features.

Given all of this it's hard not to see innovation, particularly in problems like scale-out, shifting away from MySQL to other databases as well as middleware. This would be a great time for the PostgreSQL community to get really serious about data replication. MySQL won't fade--it's already a great database. But there's likely to be a crowd of people in the MySQL community eying other solutions. It's going to be interesting to see what they come up with.

Planet PostgreSQL: Robert Hodges: Tungsten Scale-Out Stack Presentation from MySQL Conference

Wed, 02 Jul 2008 11:37:12 +0000

There have been a number of requests for copies of the slides to the Tungsten Scale-Out Stack talk I gave at the MySQL Conference in April. Here they are courtesy of the nice folks at O'Reilly who organized the conference.

Tungsten is our codename for a set of technologies to raise database performance and availability using scale-out. In the database world scale-out is a term of art that means spreading data across servers on multiple systems. With data in multiple places you are less subject to failures--when one copy crashes you just use the others. Similarly, if your application runs a lot of queries, you can spread them over different machines, which makes for faster and more stable response times.

So database scale-out sounds great (and is too), but getting it to work properly is harder than you would think. Along with practical issues like management, there are theoretical barriers. Let's say you are creating a product catalog service using database replicas on different hosts. Applications connect to any replica to get information. Your manager, a guy with pointed hair, tells you to make sure of the following:

1. The catalog service is always available.
2. The service keeps working even if you get a network partition between hosts.
3. The copies are always consistent (e.g., you can go to any copy and get the same data).

Here's an ugly surprise. It turns out your data service can only have two of the three properties at any given time, a result that was proven only recently and is now called the CAP Principle. If you want to be available and handle network partitions, you must accept that data will sometimes be inconsistent. Your manager is going to be very disappointed.

That's where we get back to Tungsten and the Scale-Out Stack. We realized a while back that you can't think in terms of a single product or even family of products to solve scale-out in a general way. It's better to design a flexible set of technologies with different strengths and weaknesses that users choose based on what's important to them. If you need to cluster over a WAN, use master/slave replication. If you don't want master failures, use synchronous replication in middleware.

Read the slides to learn more about the thinking. Database scale-out is a fascinating problem and we are looking forward to making it much easier to handle. Please stay tuned! I'll be writing more about this in the weeks and months to come.

Planet PostgreSQL: Robert Hodges: Webinar: The Coolest Scale-Out Projects on the Planet

Wed, 02 Jul 2008 11:36:22 +0000

My company Continuent sponsors Continuent.org, an open source site that contains some of the coolest scale-out projects around. You may have heard of Sequoia, which implements middleware clustering of any database that has a JDBC driver. However, Sequoia is really just the beginning.

We have several other projects that offer interesting scale-out technologies. Myosotis implements fast SQL proxying, Hedera provides wrappers for group communications, and Bristlecone has tools for performance testing of scale-out architectures. This summer we will add projects for database neutral master/slave replication as well as cluster management. In short, there's a lot to look at.

If you would like a closer look at Continuent.org, I'm doing a Webex webinar to talk about the overall technology stack and project roadmaps. It's scheduled for Thursday June 26th at 10am EDT. You can sign up here to see what's going on.

Don't worry if you miss the presentation--I'll post slides here and will be doing a series of blog entries on each of the projects in the coming weeks.

Planet MySQL: Independent MONyog review by Sheeri Cabral of The Pythian Group

Wed, 02 Jul 2008 10:59:58 +0000

Sheeri Cabral, an active MySQL blogger and two-time winner of MySQL Community Contributor awards has written a great review on MONyog.

Read the full review here

Our favorite highlights:

As an overall review ? MONyog is the best out-of-the-box GUI monitoring tool for MySQL that I have seen. It ?just works.? As promised, getting up and running quickly is easy, and having a centralized location for monitoring is very useful. The graphs are beautiful and the statistics that are graphed are useful time-savers.

The biggest difference between MySQL?s Enterprise Monitor and MONyog is that MONyog is agentless. At Pythian, we have many clients with differing security requirements. Requiring a daemon process to be running is not something we currently do, and it might be a hard sell for some clients. Even if every client was amenable to it, making sure the daemon is up and running on each MySQL server is tedious. Note that agentless operation works for every feature, including log analysis as well as operating system statistics.

Planet MySQL: Generating on-the-fly SQL scripts for batch mode

Wed, 02 Jul 2008 08:07:34 +0000

When writing scripts for running MySQL in batch mode, it is often desired to be able to change the SQL queries on-the-fly so that we can use the script to solve more generic problems.

For example, I need to use my script for different storage engines and table names and I want to be able to do it at runtime instead of manually changing my script every time.

Here is how I do it: I have a template file which is my script but with the variables having generic names. In this case, I have considered plugin_name as the storage engine name , and table_name as the table name.

Consider the following script:

# automation-script-template
# Mixing shell commands and SQL queries in batch mode- demo script

# Replace variables: plugin_name, table_name with appropriate names- done automatically by the calling shell script.

use test;

INSTALL PLUGIN plugin_name SONAME 'ha_plugin_name.so'; #Ignore statement

system mkdir traces/plugin_name

system cp /tmp/mysqld.trace traces/plugin_name/init.trace # Shell commands prefixed with a 'system'

create table table_name(num integer) engine=plugin_name;

system cp /tmp/mysqld.trace traces/plugin_name/after_create.trace # Shell commands prefixed with a 'system'

system diff traces/plugin_name/init.trace traces/plugin_name/after_create.trace # Shell commands prefixed with a 'system'

In the above script, plugin_name and table_name are the two parameters which I want to be able to change. Now, the strategy I follow is that using a shell script (below) by passing the relevant values as command line arguments to it, I 'generate' a new batch-sctipt file everytime using the above template file:

#!/usr/bin/bash
#launch

#sed- replace VARIABLES with arguments.
#plugin_name, table_name with appropriate names

#replace table_name and plugin_name
sed -e 's/table_name/'${1}'/g' -e 's/plugin_name/'${2}'/g' automation-script-template > collect-traces


# Invoked MySQL in batch mode
mysql -u root < collect-traces > log;

As you can see, I use 'sed' to replace all the occurences of the parameters with the given command line arguments and generate a case-specific script on-the-fly using the automation-script-template. A sample invocation would be:

$ ./launch dummy4 example

Which produces the following 'collect-traces' file (case-specific file):

# Mixing shell commands and SQL queries in batch mode- demo script

# Replace variables: example, dummy4 with appropriate names- done automatically by the calling shell script.

use test;

INSTALL PLUGIN example SONAME 'ha_example.so'; #Ignore statement

system mkdir traces/example

system cp /tmp/mysqld.trace traces/example/init.trace # Shell commands prefixed with a 'system'

create table dummy4(num integer) engine=example;

system cp /tmp/mysqld.trace traces/example/after_create.trace # Shell commands prefixed with a 'system'

system diff traces/example/init.trace traces/example/after_create.trace # Shell commands prefixed with a 'system'

As is apparent from the 'launch' script, this script is fed to the MySQL client in batch mode. Thus, you can successfully write more generic SQL queries using the method above.

You may also want to see this post: Mixing SQL and shell commands in MySQL

Planet PostgreSQL: Leo Hsu and Regina Obe: How to Inherit, Unherit and Merge Inherit

Wed, 02 Jul 2008 07:37:00 +0000

A lot of this information is nicely tucked away in the PostgreSQL docs in http://www.postgresql.org/docs/8.3/interactive/ddl-inherit.html, but since the docs are so huge and rich, one may tend to miss these things.

While there are numerous interesting use cases for the PostgreSQL inheritance structure, one of the key reasons people use it is for table partitioning strategies.

How do you make a stand-alone table a child of another table?

The first question that comes to mind is why would you ever need a table to adopt another table. There are 2 reasons that come to mind.

When you are loading huge amounts of data especially of a read only nature - its often convenient to not have that table be visible to your applications until you are done with the loading process. So you may want to make it a child after the loading.
Your tables seemed fairly unrelated when you started out and then one day you realized you really were talking about apples and apples and need to report on them together at a higher level. One situation like this to give a somewhat real-world perspective - lets say you developed a timesheet app for an organization and each department insisted on having their own version of the app and each along with the basic fields needed to track some additional ones of their own. Then higher forces came in and said I need to know what everyone is doing, but I don't need to see all that other crap they keep track of.. Two options come to mind - create a bunch of views that union stuff together or institute a round-up-the-children-and-adopt-them program.

This fits into one of the categories of things that PostgreSQL lets you do that PgAdmin III doesn't have a graphical way to let you do it. If you try to inherit in PgAdmin III from a table that already exists, that option is just greyed out. So you have to resort to DDL SQL statements. Luckily its fairly trivial. Well this really only works for PostgreSQL 8.2+. I don't think PostgreSQL 8.1 and below supported INHERIT/NO INHERIT in the ALTER TABLE statement.

Continue reading "How to Inherit, Unherit and Merge Inherit"

Planet MySQL: 451 CAOS Links - 2008.07.01

Wed, 02 Jul 2008 04:59:23 +0000

Progress acquires IONA. Red Hat announces quarterly earnings. Optaros raises new round of funding. (and more)

Progress Software Corporation to Acquire IONA Technologies, Progess Software / IONA Technologies (Press Release)

Red Hat Reports First Quarter Results, Red Hat (Press Release)

Optaros Closes $12 Million in Series C Financing Co-Led by .406 Ventures & Globespan Capital Partners, Optaros (Press Release)

Sourcefire, Inc. Responds to Barracuda Networks Letter, Sourcefire (Press Release)

Linux Phone Standards (LiPS) Forum Joins Forces with LiMo, LiPS Forum / LiMo Foundation (Press Release)

Microsoft Takes Additional Steps in Implementing Interoperability Principles, Microsoft (Press Release)

Sun Microsystems Releases New GlassFish and MySQL Offering To Provide Customers With Unlimited Deployments of Enterprise Class Application Server and Database Software, Sun Microsystems (Press Release)

Oracle Updates Entire Family of Oracle Berkeley DB Embeddable Databases, Oracle (Press Release)

Openmoko Signs Five Distributors for Freerunner Open Source Mobile Phone, Openmoko (Press Release)

MuleSource Releases Mule Galaxy Enterprise, MuleSource (Press Release)

NETGEAR Launches Open Source Wireless-G Router Enabling Linux Developers and Enthusiasts to Create Firmware for Specialized Applications, NETGEAR (Press Release)

Zenoss Expands IT Management Solution for Managed Service Providers, Zenoss (Press Release)

Actuate First to Provide Support for Eclipse BIRT 2.3, Actuate (Press Release)

Latest Open-Xchange Public Beta Gives Customers a Free Preview to Business-Class Groupware, Open-Xchange (Press Release)

Open Source Community Wins at Ingres Code Sprint, Ingres (Press Release)

Coming Battle Over Open Source Phones, Forbes.com, Elizabeth Woyke (Article)

Linux experiences ‘prolific’ growth, says Linux Foundation’s Zemlin, SearchEnterpriseLinux.com, Pam Derringer (Article)

Nvidia Reiterates Position on Closed Source Driver, OSnews, Thom Holwerda (Article)

Red Hat CEO: Oracle/BEA Deal Is Helping Us, Internet News, Sean Michael Kerner (Article)

The State of Open Mobile OSes, TechCrunch, John Biggs (Blog)

When is an open-source project ready?, Practical Technology, Stephen j. Vaughan-Nichols (Blog)

Nokia?s Open Source Response: The Symbian Q&A, RedMonk - tecosystems, Stephen O’Grady (Blog)

Planet MySQL: MySQL Challenge: Limit Rows Accessed - Continued

Wed, 02 Jul 2008 04:36:00 +0000

Yesterday I blogged about the MySQL Challenge to limit rows accessed. Sheeri had some great comments about the subject, but didn't know either why the first 4 rows were not counted in my original solution:

mysql> explain select actor_id from actor where actor_id % 5 = 0 and actor_id < 26;

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: actor
type: range
possible_keys: PRIMARY
key: PRIMARY
key_len: 2
ref: NULL
rows: 20
Extra: Using where; Using index

mysql>

I've been thinking about this some more, and wondered what would happen if I tried this query:

mysql> select actor_id from actor where actor_id % 5 = 0 and actor_id < 26 and actor_id > 4;
+----------+
| actor_id |
+----------+
|        5 | 
|       10 | 
|       15 | 
|       20 | 
|       25 | 
+----------+
5 rows in set (0.00 sec)

This has the exact same results as my original solution, but check out what EXPLAIN has to say:

mysql> explain select actor_id from actor where actor_id % 5 = 0 and actor_id < 26 and actor_id > 4\G

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: actor
         type: range
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 2
          ref: NULL
         rows: 16
        Extra: Using where; Using index
1 row in set (0.00 sec)

I would expect EXPLAIN to report 20 rows, not 16. I know EXPLAIN takes a best estimate as to how many rows maybe accessed for a given query, but I would think both EXPLAINations would return 20 rows. This is a simple query... Could this be a bug?

More digging is in order!

Planet MySQL: Should we proclaim MySQL Community Edition Dead ?

Wed, 02 Jul 2008 02:41:44 +0000

We were chatting with Jeremy Cole today and he brought to my attention last version of MySQL Community Eddition (5.0.51) was released in November 2007 - over 7 months ago. MySQL 5.0.51a and MySQL 5.0.51b security fixes were released but these can't be considered proper releases.

If we look at the old Kaj's Announcement we can see there suppose to be 2 yearly binary releases (which are overdue) and 4 predictable yearly source releases, which we have not seen either.

Though, Honestly, I do not care that much any more. I have adapted my ways not to go to dev.mysql.com for MySQL downloads any more but go to mirror.provenscaling.com, where Jeremy Cole promptly publish MySQL Enterprise sources and binaries. We take sources and add spice them up with other community extensions which can't ever get included in official MySQL tree and use that version instead.

It is worth to mention at the same time there are some improvements when it comes to development community MySQL now moved sources to launchpad - now it makes it easier for you as contributor to create your branch or grab and test someone elses. If you're just looking for stable, recent, binary MySQL Community release, there is little MySQL offers you.

Entry posted by peter | 19 comments

Add to: | | | |

Planet MySQL: The best MySQL book ever

Tue, 01 Jul 2008 20:04:00 +0000

My copy of High Performance MySQL arrived today. Pardon the hyperbole but I reviewed it and I just saw another episode of The Bachelor with the most dramatic rose ceremony ever.

As good as this book is, it will soon need a third edition. There are several storage engines under development (PBXT, q4m, Maria, Falcon) and InnoDB continues to innovate (compression in the 5.1 plugin). Some of the things that we know (MyISAM and InnoDB don't scale on SMP) will no longer be true. And there will be many new things to figure out (just how do you tune Falcon).

Can Domas help write the next edition?