TDD, Unit Testing and Mock

TDD, Unit Testing and Mock

Hey!

So the last couple of weeks I have been working on fixing the tests that were failing on the glance-replicator patch. While trying to fix these tests, I learned quite a bit about TDD, unit tests and mock.

The Importance of Unit Testing

Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: requirements are turned into very specific test cases, then the software is improved to pass the new tests, only.

When there are so many people working on such a huge amount of code, writing up new functionalities, features, behaviours etc  can get pretty difficult to manage. As you edit/update/enhance code you can make a change which could break the code which was earlier functional and this break might even go undetected causing lots of damage later on to the development team and the stakeholders involved. If such a break occurs it might even get difficult to debug where the error has crept in and which piece of code is no longer working correctly.

Unit testing saves us here. It is a process in which the smallest testable parts of an application, called units, are individually and independently scrutinized for proper operation. Developers write tests for every class they produce. They are extremely helpful from a developer’s point of view. The tests are intended to test every aspect of the class that could conceivably not work.

OpenStack which is mainly written in Python employs the library that exists for unit testing in Python (the documentation for which can be found here) to good use. Every OpenStack project has its own set of unit tests contained within the project, which can be run using tox in most projects and/or run_tests.sh by developers.

Since we’re testing your code as we introduce functionality, we’re going to begin developing a suite of test cases that can be run each time you work on your logic. When a failure happens, we know that we have something to address. Of course, this comes at the expense of investing time to write a suite of tests early in development, then running them again and again, investing in machinery to automate running tests before a new merge (Good ol’ Jenkins) but as the project grows we can simply run the tests that you’ve developed to ensure that existing functionality isn’t broken when new functionality is introduced.

I came across the description of a good unit test here which I think was pretty informative. We can follow the principles mentioned in the link to the last alphabet and be able to write extremely good quality tests.

Why Mock?

Now we know why testing is important and how to write a good testing suite but it is not always that simple. Sometimes the code we write directly interacts with what we can call “dirty” services. These are the services that are crucial to our application, but whose interactions have intended but undesired side-effects—that is, undesired in the context of an autonomous test run. These issues could be the overhead of a system call, filesystem accesses, network accesses, external API requests or in my case http requests.

The Python unittest library which facilitates unit testing includes a subpackage named unittest.mock—or if you declare it as a dependency, simply mock—which provides extremely powerful and useful means by which to mock and stub out these undesired side-effects.

Mocking basically is the replacement of one or more function calls or objects with mock calls or objects.

A mock function call returns a predefined value immediately, without doing any work. A mock object’s attributes and methods are similarly defined entirely in the test, without creating the real object or doing any work. However, it needs to be setup by the developer.

I have used the Mock Library to write tests in my glance-replicator patch for the functionalities I wrote while migrating from httplib to the requests library. I found this basic summary of how mock can be used here, which I found very useful when writing my tests.

That’s all for now! Thank you for reading! 😀

Working with Gerrit and Tox

Working with Gerrit and Tox

Hey!!

Today I am going to share some of the small things I’ve picked up during the course of my internship regarding working with Gerrit, running tests locally etc.

Running unit tests locally

One of the small but very useful things I learned about was regarding running unit tests locally. The one thing that should definitely be done is running the tox tests locally and resolving any issues before submitting any change for review. Running the command “tox” in the projects root directory runs all the tests for all the categories of tests as specified for the project like python-2.7, python3.4, pep8, docs etc.

However, running all these tests may take a while. There may be times when you may want to run just a particular category of tests, for which you can just run the “tox -e <test>” command (here <test> refers to the category of test you want to run). Further, there may be a case where you may want to run only one particular unit test, or just a few unit tests. In this case, running even just the py27 tests, for instance, may seem like a bit of a pain. In this case, the following commands can be used to test just a few unit tests at a time:

source .tox/py27/bin/activate
testr list-tests test_name_regex > my-list
python -m testtools.run discover --load-list my-list

In the above commands, test_name_regex refers to the unit test(s) you want to run. Running these commands then only runs the specific unit tests that you want to run instead of running all the unit tests. Alternately, if you want to run just one test, you could simply run the following command:

tox -e py27 -- <test>

In the above command, <test> refers to the unit test you want to run. Although in this case, the entire path to the test needs to be provided (for instance, glance.tests.unit.test_glance_replicator.ImageServiceTestCase.test_rest_errors) which is in contrast with the set of commands I mentioned earlier where just the test name suffices (for instance, test_rest_errors).

Adding dependencies

Another thing I learned about recently is adding dependencies on patches. There may be a case where you may want to submit a very big patch, but instead decide to split it into smaller patches. In this case, it makes sense to make one patch dependent on another. Another case where dependencies may be used is when a patch that you submit may require another patch still under review to be submitted first.

Firstly, what does adding a dependency mean? So, for instance, there is a patch under review, let’s call it patch X. And there is a patch Y that is dependent on X. In terms of code, what this means is that for patch Y, its master branch is actually patch X i.e. all the changes that are made in patch Y are made on the state of the repository with X’s changes made on it. X on the other hand (if it is not dependent on any other patch too) is based on the master branch i.e. the current state of the repository. So once Y has been submitted for review, if any changes are made on the X patch i.e. new patchsets are uploaded for it, Y will have to be rebased. Rebasing a patch just means bringing it up-to-date with the master branch. However, Y will have to be rebased onto X’s new patchset.

However, there are a few fairly simple commands that can be used to add dependencies. First, you need to fetch the change under review and checkout a new branch to work on as follows:

git review -d $CHANGE_NUMBER
git checkout -b $BRANCH_NAME

Next, you can make the changes for the patch to be submitted, and then submit the patch as follows:

git commit -a
git review

So the patch that you have now submitted will be dependent on the patch that you fetched using the first command. This method can be used when creating a new patch and making it dependent on another existing patch. If you have already submitted a patch for a change, and then want to make it dependent on another existing patch, you can simply fetch the parent change, cherry-pick your commit on top of it, and then submit the rebased change for review, as follows:

git review -d $PARENT_CHANGE_NUMBER
git review -x $CHILD_CHANGE_NUMBER
git review

Lastly, if you want to rebase your change since your parent change has been changed since then, you can just run the following command:

git rebase -i $PARENT_CHANGE

That’s all for now. Thanks for reading! 😀

Halfway through Outreachy! :D

Halfway through Outreachy! :D

Hey!

I am mid-way through my Outreachy internship with Openstack now 😀 and have learned quite a few things in this period.

Firstly, one of the most important things that my mentor Nikhil helped me understand was that it is extremely important to interact with the community that you are working with (in my case, this was Glance). It is the people in the community that will review your patches, and it is them that you can go to if facing any problems with your work. One of the ways of interacting with the community is attending the weekly meetings. Another thing that I learned was the importance of reviewing other people’s patches. Reviewing people’s patches is another great way of interacting with the community.

In the last few weeks, I started helping out with some reviews. One of the major challenges I faced when starting out with this was finding patches that I could review. Finding patches that need reviewing, and that I am also experienced enough to be able to review was a bit of a challenge. To start out with, you really need to spend some time going through patches, understand what they are trying to achieve, go through the changes being made, try and see if you can think of a better approach to any of the changes made. Once you’ve reviewed a couple of patches and you get the hang of how this works, the process becomes much easier.

I’ve learned a lot during the period of this internship, and I hope to learn a lot more in the coming weeks!

Thanks a lot for reading! 🙂

 

Glance-Store Drivers

Glance-Store Drivers

Hey!

Last week I mentioned that I had been working on migrating Glance-replicator to using the requests library. This week I have been helping out with improving the help text for Glance configuration options.

I have worked on the configuration options for some of the drivers in Glance Store till now. To begin with, it is important to understand what Glance Store is and what it does. Glance Store performs all the file operations for Glance. It handles the interactions with the local file systems as well as with the external storage back ends. Thus it is the interface between Glance and external data stores. Glance Store provides this interface with external data stores through drivers. It provides several different drivers like that for HTTP, RBD, VMware, S3, Sheepdog, File System, Cinder and Swift.

I worked on improving the help text for the configuration options for the VMware, HTTP, RBD and S3 drivers. While working on some of these driver options, I learned a few things about some of these data stores.

When I first started working on the RBD options, I didn’t really know much about what RBD is. So firstly, what is RBD? RBD stands for RADOS block device. RADOS (Reliable Autonomic Distributed Object Store) distributes objects across the storage cluster and replicates objects for fault tolerance. RBD is one of the storage systems that you can use to store and access your data when using RADOS storage. RBD facilitates storage of block-based data in the storage system. It breaks down block-based data into small chunks which are then stored as objects in RADOS. These objects are stored in virtual block-devices (thus the name RADOS block device) in the storage cluster.

Another driver that I worked with this week was the S3 store driver. Again, I didn’t really know much about the S3 storage system before I started working on its options. So again, the first question is: what is S3? S3 refers to the Amazon Simple Storage Service (Amazon S3). S3 stores the data in the form of objects within buckets. Here, object refers to a file and any metadata related to it. Thus, buckets are containers for objects in the S3 storage system.

Thats it for now. Thanks for reading! 😀

Outreachy Updates! :D

Outreachy Updates! :D

Hi!

It’s been a while since I wrote my last blog post. The major update since then is that I have been accepted as an Outreachy Intern. Yaaay!! 😀

I am doing my internship with OpenStack, specifically I am working on Glance under my mentor Nikhil Komawar. The project that I had proposed as part of my application was “Migrating Glance replicator to requests for HTTPS support”.

Currently, glance replicator uses the httplib library, and thus does not support the proper verification of HTTPS connections. I am working on rewriting the code to switch to using requests library instead which will allow the automatic verification of all HTTPS connections. Also, since an operator may want to sign their own custom certificates, I am also working on adding a command-line option to specify a custom certification bundle to be used when verifying the server certificate.

I have already finished major refactoring of the code to shift to using the requests library. To do this, I first had to make changes in how the connections were being established (as they were using httplib earlier). Also, how the response from the server was handled had to be modified a bit in accordance with the requests library.

I have also added the command line option for specifying a custom certification bundle. It turned out to be a pretty straightforward task since requests has an optional argument that can be specified to use custom certificates when establishing a connection.

The changes I made involved learning some things about the requests library. The first thing that I learned was about the Session object. A session object allows you to reuse certain parameters across requests. So if you’re making a lot of connections to the same host, it will allow you to reuse the same TCP connection. Another concept that was new to me in the requests library was that of Transport Adapters. Transport Adapters basically allow you to define specific interactive methods for a HTTP service i.e. they provide the option to configure the different HTTP services. Users can create their own transport adapters to provide any specific functionality. Once you create the Transport Adapter, you can mount it to a session object and also specify which service(s) is to use that particular adapter. Once the transport adapter has been mounted to the session, any request that is made using that session which has the prefix mentioned for that specific adapter will then use that specific transport adapter.

 
That’s all for now! Thanks for reading! 🙂

Outreachy: The end of the Application Period!

Outreachy: The end of the Application Period!

Hey!

So, the application period is now officially over. And it’s been a really hectic but amazing one month. I have learned so much from the OpenStack community. One of the most important things I ended up learning was how to manage my time well, with trying to contribute as much as possible to OpenStack as well as managing college work.

The last couple of weeks particularly have been a bit challenging, managing working on my application to Outreachy as well as continuing working on bugs. The way I went about trying to juggle so many things at a time was by working on bugs by efficiently keeping in contact with the Glance community members through mail. So I would work on my application with some help from my mentor Nikhil whenever he was available on the IRC. I was motivated by my mentor to work on my own, be proactive and manage my time efficiently, I would continue working on my application by myself and also working on the bugs based on previous communications with the community members. This way I was able to continue working on bugs while also writing my application.

Even though the application period is now over, I will continue to contribute to Glance as much as I possibly can. It has been a great learning experience working on Glance and OpenStack, and I hope to be able to learn more :D.

Thanks for reading! 🙂

OpenStack: My First Contribution!

OpenStack: My First Contribution!

When I read about Openstack on the Outreachy page (Participating Organisations), I found the description quite interesting. On taking a closer look at OpenStack (Openstack Outreachy , Internship Ideas), I decided to start working on it.

My first communication with the OpenStack community was on their IRC channel dedicated to Outreachy mentors, interns and general inquisitors (#openstack-opw on Freenode). Victoria Martinez de la Cruz (IRC nick vkmc on Freenode) who is the program organizer and a mentor helped me get started. She familiarized me with how OpenStack works, how the different components are organized and how the different projects work under OpenStack BigTent model. Once I looked through the different projects (List of Projects), I finally decided to work on OpenStack Glance or as we often refer to it as simply ‘Glance’ (Glance, Glance’s Documentation, Glance Pythonclient). Victoria helped me get in touch with my potential mentor Nikhil Komawar (IRC nick nikhil on Freenode). He familiarized me with Glance and also helped me with setting up Devstack. He also helped me find an easy bug to start with (tagged as a low-hanging-fruit). Low-hanging-fruit bugs are bugs that are easy to fix, ideal for beginners to get familiar with the workflow. I then tried accessing Gerrit by following the instructions mentioned here. After running into some problems with Gerrit, Nikhil helped me solve them. We used paste.openstack.org in our conversation and narrowed down on the issue and thus solved the proxy issues. I started by googling how to access Gerrit when working behind a proxy and came across the solution of using https instead of ssh. I then used this link to access Gerrit over HTTPS. I learnt something new about gerrit’s https usage instead of ssh in the process. I finally managed to successfully merge my first patch! 😀 (the link to it) and learnt a lot about the review process, technical communication and having a critical perspective to writing code.

It’s now been almost a month since I started contributing to OpenStack, and I am extremely happy to have been able to contribute to this community. Whether or not I get selected for the Outreachy internship, I will definitely keep working and keep contributing to OpenStack. It has been an amazing experience being a part of this community and so far and I am really glad to have chosen OpenStack.

Hope you found this useful and thank you for reading! I would love to read your comments on the post. 🙂