Using Buildbot with NSS on Windows systems 
Wednesday, January 9, 2013, 10:34 PM
Posted by Administrator
Here is how I had success to use Buildbot with NSS on Windows systems.

Because buildbot on Windows isn't able to kill the full set of processes belonging to an active build/test job, each time the connection between buildbot-master and buildbot-slaves (the slave on the Windows client) breaks, the slave system ends up being in a mess, where parts of the old job are still active. In that state, a new started job will misbehave, because old and new job fight against the files on disk.

I tried many approaches, including attempts to kill the jobs by name. For various reasons, most of them didn't work right for me. And by work right, I expect that the system automatically recovers from the failure, and fully restarts the job - without requiring manual cleanup by an admin, without requiring manual restart of jobs.

I heard that some consumers modify buildbot to their needs, but that is not acceptable to me. I need to focus on other things, I don't have time to learn the internals of buildbot. For that reason, approaches like "use a custom logfile parser" are not an option for me.

Here is a solution that seems to work for me, which involves automatically rebooting the Windows slave after each connection failure to the buildmaster. We don't reboot if jobs finish without exceptions (a build failure is fine, it isn't an exception).

It requires a Windows slave with buildbot-slave installed.
The buildbot-master must be configured to allow only one build for the Windows slave, BuildSlave(max_builds=1, ...).
You must setup the slave to automatically login and start the buildbot-slave after a reboot (e.g. using the StartUp program group).
If you use an encrypted tunnel for the connection between buildslave and buildmaster, that tunnel must be automatically started after reboot, too.

Let's say you have a script that you use on Linux and other Unix-like platforms, which you run from buildbot master as a "buildstep".

On Windows, we'll use different buildsteps, and one of them will call the script. (Note that the NSS build system uses the Mozilla-build tools on Windows, including MSYS, which provide a Unix-like environment, including a bash shell).

Let's say your non-Windows buildsteps are:
* get source code

On Windows, we'll use:
* maybe-reboot.bat
* get source code
* start-b-and-t.bat

The .bat files have the following contents:

==== maybe-reboot.bat ====
IF EXIST ..\buildbot-is-building (
del ..\buildbot-is-building
shutdown /r /t 0

timeout /t 120

=== start-b-and-t.bat ====
echo running > ..\buildbot-is-building

rem This is an Example.
rem Do whatever you must do to start your build.

"%MOZILLABUILD%\msys\bin\bash" -c "hg/tinder/buildbot/ %*"

if %errorlevel% neq 0 (
) else (

del ..\buildbot-is-building

exit /b %EXITCODE%

This assumes that the user account used for building and testing has permission to reboot the system, executed using Window's "shutdown" tool.

If all goes well, what happens is:
* maybe-reboot will do nothing
* start-b-and-t will create a file to signal that it's active,
and will delete the file after the job has completed.

In case there is an exception, and start-b-and-t is still active, what happens is:
* buildslave will notice the exception and attempt to kill the active job. For advanced build/tests this will partially fail.
* buildmaster and buildslave regain the connection, and buildmaster asks to start the build.
* maybe-reboot will detect that a job is still active, and will prepare to reboot
* it will clean up by removing the status file, to ensure we will proceed to start-b-and-t after the reboot
* it will ask the system to reboot immediately
* because it will take some time before the reboot has killed all active processes, we must make sure that this job won't make things worse by running the job again, in parallel
* the timeout command will wait for 2 minutes, that should be enough for the reboot to happen, including termination of maybe-reboot.bat
* buildmaster will notice an exception, and will remember once again to retry that interrupted job
* as soon as the slave has rebooted and automatically started the buildslave, the job will be restarted in a clean environment.

You might ask, why are we using the status file in the parent directory of the buildbot
slave directory? ("..\")
Because your slave might be configured to build multiple different configurations,
that file must be in a place that is shared by all those build configurations.
The parent directory of an individual configuration is buildbot's main slave directory,
that makes it a good place for that file.

[edit]: Updated start-b-and-t.bat to return the inner script's exit status to buildbot.
view entry ( 1227 views )   |  permalink   |  $star_image$star_image$star_image$star_image$star_image ( 3 / 980 )

No error messages in Thunderbird for SSL/TLS protocol failures 
Wednesday, December 5, 2012, 06:01 PM
Posted by Administrator
Starting With Mozilla Thunderbird 10 and newer (and up to at least Thunderbird 17), SSL/TLS protocol failures are no longer user discoverable.

In Mozilla Thunderbird 9 and earlier, if the SSL/TLS protocol handshake with a server resulted in a fatal failure, an error prompt was shown that notified the user about the failure. With Thunderbird 10 and newer, no feedback will be shown, and the user cannot know whether the failure is on the client side or on the failure side, and what exactly causes the problem. With Thunderbird 17, you can at least find the error message hidden in the "error console" that can be opened from the menu (hidden between all the other frequent messages that are dumped to that place).

One example scenario where you'd run into this silent failing is a typical enterprise configuration, where the server requires the client to authenticate with a client certificate. If the client certificate is not accepted by the server, or no such cert hat been installed yet, then Thunderbird will fail silently. (There are additional SSL/TLS protocol failure scenarios where you'll get the same silent failing.)

The reason for this regression is that error reporting was removed, instead of working on a smarter solution for Bug 682329 . All proposals that would have kept the error reporting in Thunderbird working and that could have been implemented by me with a reasonable amount of work were rejected.

In case you'd like to test the regression yourself, you can follow the steps below.

Use Thunderbird to configure an additional email account. When asked by Thunderbird 17 choose existing account.

- Username: test
- email:
- password: test (it's wrong, but it doesn't matter)
- check "remember password"
- click continue
- click "manual config"

- Select IMAP
- hostname (NO leading dot as proposed, in other words, NOT
- port 993
- auth: normal password

SMTP configuration doesn't matter, you won't be able to send email through this configuration, but in order to allow you to complete the configuration:
- hostname:
- port 465
- auth: normal password

click ok

With thunderbird 17, the test fails, click "advanced config", and in the next dialog click ok.

You'll now see the standard Thunderbird window, and you should have the test account configured. Click on the inbox. Your statusbar (lower part of the window) might show "connected to ssltls" but that's all you'll ever get with Thunderbird 10 and newer, even if you click the "get mail" toolbar button. You might eventually, much later, see a "connection timed out" notification popup, but that's incorrect, and it's not helpful in diagnosing the cause of the problem.

If you do the above with Thunderbird 9 and earlier, you'll get
"An error occurred during a connection to, SSL peer was unable to negotiate an acceptable set of security parameters. ssl_error_handshake_failure_alert"

With Thunderbird 17, this message can be found in the error console.

view entry ( 1041 views )   |  permalink   |  $star_image$star_image$star_image$star_image$star_image ( 3 / 150 )

Mutually Endorsing CA Infrastructure, proposal v2 
Saturday, February 25, 2012, 12:47 AM
Posted by Administrator
We must solve the problem that any CA is a single point of failure for the Web's trust, as used by the SSL/TLS protocol.

In October 2011 I had published a rough description of a system that I named MECAI, Mutually Endorsing CA Infrastructure.

It took some time to think in more detail about the various aspects, and I'm thankful for the discussions I had with other security experts, who pointed out missing details, problems that still need to be solved, etc.

Now I'm able to announce that I've published (yesterday) a more detailed writeup (version 2), you can find it here:

I'm looking forward to your feedback. It might be best to post your feedback to the IETF's mailing list therightkey.

view entry ( 1144 views )   |  permalink   |  related link   |  $star_image$star_image$star_image$star_image$star_image ( 3 / 979 )

NSS and keys 
Thursday, February 9, 2012, 01:09 PM
Posted by Administrator
Using NSS command line tools can be tricky.

I recently wrote this little HOWTO that describes how you can use the NSS command line utilities to create a CSR (Certificate Signing Request), later import the certificate you receive, and make a backup of key and certificate to a PKCS#12 (p12) file. ... st-etc.txt

Also recently, I was asked for help. An Add-On author had created a key pair using NSS tools, for the purpose of the Add-On updating mechanisms (which involves the McCoy tool). Later it became necessary to extract the private key in order to use it with other applications.

Unfortunately, dealing with raw key pairs can be tricky in NSS, the tools are more powerful when dealing with keys that are bound to certificates.

So, we had an NSS database containing a private key, an external copy of the public key as base64, but no direct way of exporting the private key. The solution we used was tricky, so I'm documenting it here.

Using a separate empty NSS database, we used certutil to create a new CSR, of the same key type and size, and stored the binary encoding of the CSR in file some-other.csr

We also created the binary encoding of the public key that we had externally available (found in an update.rdf file).

Then we used the dumpasn1 tool to create a binary dump of some-other.csr - this told us at which byte position inside the public key starts.

Then we used a binary file editor (dd) to create a remixed CSR...
- start with the initial part of some-other.csr until the public key begins
- append the public key from the production key
- append the remainder of some-other.csr, the part after the public key

At this point we had a CSR for the production key, but an invalid one. It's invalid, because the digital signature inside the CSR (a proof of possesion of the private key) didn't match the contained public key.

Next we created a hacked version of NSS' certutil, we disabled the call to CERT_VerifySignedDataWithPublicKeyInfo - which disabled the check for the signature.

This way, we were able to use a local CA certificate (one that I use for testing purposes) to create a certificate for this certificate request. This certificate could be imported (and a nickname assigned to it) into the original production database that had only keys, but no cert.

Now it was possible to use NSS utility pk12util to export the certificate, including its keys, to a PKCS#12 files. As this is a standard container, you can use tools provided by other crypto toolkits to extract the keys and use it elsewhere.

I'm documenting the procedure in the hope it can be helpful for other's who run into the same dilemma.

But the more important message of this story, please avoid this kind of hassle. You shouldn't deal with raw keys, but rather use certificates.

view entry ( 7441 views )   |  permalink   |  $star_image$star_image$star_image$star_image$star_image ( 3 / 987 )

Fosdem 2012: Web Security, and how to stop the next DigiNotar 
Thursday, February 9, 2012, 01:03 PM
Posted by Administrator
At the 2012 conference in Brussels I had the opportunity to talk about Web Security, the various proposals we've seen regarding trust of key ownership (including my own MECAI proposal), and my opinion what we should be done next.
The text can be found at:

view entry ( 1244 views )   |  permalink   |  $star_image$star_image$star_image$star_image$star_image ( 3 / 826 )

<<First <Back | 1 | 2 | 3 | 4 | 5 | Next> Last>>