PROVE IT !!If you know it then PROVE IT !! Skill Proficiency Test

Changing c3p0 parameters in Cloudera manager

This post was ready long ago but I had many doubts and thoughts before I decided to publish it anyway.

We had a problem with Cloudera manager and we could not find any ready-made solution on the web or in Cloudera’s official documentation. So we ended up doing some reverse engineering and changing Cloudera manager’s code. It solved our problem and we felt victorious. Then I wrote this post detailing the process we did.

Some time later we started working closely with a Cloudera support engineer. After hearing about what we did he showed us an internal document showing that this capability is already built into Cloudera manager and we did not have to go through all of this, just add few lines to a configuration file. However, I still think that explaining what we did and the techniques we used is still valuable. So here is the full story of how we changed the parameters the hard way, if you’re only interested in changing c3p0 parameters then you can jump directly to the last paragraph.

Lately, we faced a problem with Cloudera manager. Every once in a while it got stuck and the log showed many messages like this:
sqlExceptionHelper.jaca ERROR: an attempt by a client to checkout a conbection has timed out

Restarting Cloudera manager fixed it for a while but we wanted to find the root problem. Some internet research led us to this post that suggested that c3p0 is the culprit. A deeper look at Cloudera documentation revealed that Cloudera manager indeed uses c3p0, we’re getting closer.

c3p0 is a java library that enables jdbc connection pooling. I guess you know what a connection pool is and if not then you can read about it here. So Cloudera manager uses c3p0 to connect to its database.

We noticed that every time the CM was stuck, c3p0 had 50 open connections, and that led us to think that the default upper connections limit for the pool is 50. You can see it in charts using the query
select c3p0_num_connections, c3p0_num_busy_connections

The database itself (we use Oracle) was far from exhausting its connections. So we figured we were hitting the limit and we should increase it. We looked for a configuration file or a CM configuration that changes this parameter but did nit find one, so we had to fix it directly in the code. This is how we did it (This is the procedure I used, other tools and procedures can get the same result).

Disclaimer: Changing the code of a system can cause unexpected behavior and in most cases the software vendor will not support a modified system. Do it only if you are willing to take the risks. I will not be responsible to any damage caused by such changes.

All Cloudera manager jar files are stored in
/usr/share/cmf/common_jars so we started searching there.

First of all, copy the whole common_jars directory to your local computer and make sure this computer has IntelliJ idea installed. You need the paid version but a trial version is ok too (do not use the community edition).

We spent quite a lot of time looking for the specific class where the parameters are set, but I will save you the long story and just direct you to the right file and class.

In the common_jars directory, locate files called “common-xxx.jar” where xxx is your CDH version. For example, in my case it’s common-5.16.1.jar. From now on I will refer to common-5.16.1.jar and you should replace it with your own version.

Run Idea and create a new project of type Kotlin java:

After creating the new project, press F4  to bring up the project structure dialog:

View full size image

Choose “libraries” on the left pane, then click the plus sign to add jars. Then select the common_jars directory you copied to your local computer and press ok.

Now you should be able to see all the jars under “external libraries”:

Scroll down and find common-5.16.1.jar and open it. Inside the file, ,go to com.cloudera.enterprise and find a class named CommonMain. Double click it and you will see its source code on the right pane:

View full size image

Copy the code to your clipboard and scroll up to the project. Under src directory, create a directory cloudera.com.enterprise and in it create a new java file called CommonMain. Paste the code into this file.

Look for this function which sets the connection pool properties:

We will add a function that reads the configuration from a file located at /opt/cloudera/c3p0/c3p0.properties and change the existing function to use the values from the property file. Change the code above to this:

You should also add those imports:

Now create a file /opt/cloudera/c3p0/c3p0.properties with this content:

If IntelliJ Idea does not report any errors, then we are ready to compile our code. Before you do that, make sure that the version of  Java interpreter you are compiling your code with is not higher than the one your Cloudera manager runs on, or you may encounter compatibility issues (such as “Unsupported major.minor version 52.0” errors).

If you have compilation errors you will have to fix them until you have a successful compilation. Then go to the project directory (usually under c:usersIdeaProjectsout) and find the CommonMain.class files. There should be Three files: CommonMain.class, CommonMain$1.class and CommonMain$2.class. you should use all of them.

Now open the common-5.16.1.jar file on your computer (just open, do not extract). A jr file is compressed using zip algorithm, so any unzip program will do. I used 7zip for this matter. In the opened archive, navigate to com.cloudera.enterprise and find commonMain.class files. Just drag and drop the files you compiled onto the old files and let them overwrite the old files, then close the jar archive.

Now backup the existing file. Login to the machine that runs Cloudera manager and run 
cp /usr/share/cmd/common_Jars/common5.16.1.jar /usr/share/cmd/common_Jars/common5.16.1.jar.orig

Now shut down cloudera manager and use winscp or any other program to copy the modified common-5.16.1.jar file from your computer to the CM server at /usr/share/cmf/common_jars. Make sure you transfer the file in binary mode.

Start Clousera manager and make sure everything runs without problems. Now you can get back to charts and run the query 
select c3p0_num_connections, c3p0_num_busy_connections  again. It may take some time but you can see that the number of connections now exceeds 50. Here is a chart of my c3p0 connections over time.

You can see from the chart that the old limit was at 50 and the busy connections line touched it from time to time. After the change (On December 8) the limit has increased to 150 and Cloudera manager often takes advantage of the extra connections.

We did not encounter any halts and errors coming from c3p0 since we changed it. It’s a good idea to save the original common-5.16.1.jar file in a safe place in case you will need to undo the changes.

Here our heroic journey ends. As I said, there is a better, easier and safer way. For some reason Cloudera does not publish it (at least a thorough google search could not find anything about it).

Go to /etc/cloudera-scm-server and edit the file db.properties.

Normally you will find there the parameters used to connect to the metastore. However, you can also add c3p0 parameters there:

Then restart Cloudera manager and the change will have the same effect on c3p0 as the long procedure. Still, in case you have to change java code for some reason you can find some help in this post.

Let’s block ads! (Why?)