API Testing

1 API Testing

Application programmable Interfaces (APIs) are collections of software functions or procedures that can be used by other applications to fulfill their functionality. APIs provide an interface to the software component. These form the critical elements for the developing the applications and are used in varied applications from graph drawing packages, to speech engines, to web-based airline reservation systems, to computer security components.

Each API is supposed to behave the way it is coded, i.e. it is functionality specific. These APIs may offer different results for different type of the input provided. The errors or the exceptions returned may also vary. However once integrated within a product, the common functionality covers a very minimal code path of the API and the functionality testing / integration testing may cover only those paths. By considering each API as a black box, a generalized approach of testing can be applied. But, there may be some paths which are not tested and lead to bugs in the application. Applications can be viewed and treated as APIs from a testing perspective.

There are some distinctive attributes that make testing of APIs slightly different from testing other common software interfaces like GUI testing.

ü Testing APIs requires a thorough knowledge of its inner workings - Some APIs may interact with the OS kernel, other APIs, with other software to offer their functionality. Thus an understanding of the inner workings of the interface would help in analyzing the call sequences and detecting the failures caused.

ü Adequate programming skills - API tests are generally in the form of sequences of calls, namely, programs. Each tester must possess expertise in the programming language(s) that are targeted by the API. This would help the tester to review and scrutinize the interface under test when the source code is available.

ü Lack of Domain knowledge – Since the testers may not be well trained in using the API, a lot of time might be spent in exploring the interfaces and their usage. This problem can be solved to an extent by involving the testers from the initial stage of development. This would help the testers to have some understanding on the interface and avoid exploring while testing.

ü No documentation – Experience has shown that it is hard to create precise and readable documentation. The APIs developed will hardly have any proper documentation available. Without the documentation, it is difficult for the test designer to understand the purpose of calls, the parameter types and possible valid/invalid values, their return values, the calls it makes to other functions, and usage scenarios. Hence having proper documentation would help test designer design the tests faster.

ü Access to source code – The availability of the source code would help tester to understand and analyze the implementation mechanism used; and can identify the loops or vulnerabilities that may cause errors. Thus if the source code is not available then the tester does not have a chance to find anomalies that may exist in the code.

ü Time constraints – Thorough testing of APIs is time consuming, requires a learning overhead and resources to develop tools and design tests. Keeping up with deadlines and ship dates may become a nightmare.

Testing of API calls can be done in isolation or in Sequence to vary the order in which the functionality is exercised and to make the API produce useful results from these tests. Designing tests is essentially designing sequences of API calls that have a potential of satisfying the test objectives. This in turn boils down to designing each call with specific parameters and to building a mechanism for handling and evaluating return values.

Thus designing of the test cases can depend on some of the general questions like

ü Which value should a parameter take?

ü What values together make sense?

ü What combination of parameters will make APIs work in a desired manner?

ü What combination will cause a failure, a bad return value, or an anomaly in the operating environment?

ü Which sequences are the best candidates for selection? etc.

Some interesting problems for testers being

1. Ensuring that the test harness varies parameters of the API calls in ways that verify functionality and expose failures. This includes assigning common parameter values as well as exploring boundary conditions.

2. Generating interesting parameter value combinations for calls with two or more parameters.

3. Determining the content under which an API call is made. This might include setting external environment conditions (files, peripheral devices, and so forth) and also internal stored data that affect the API.

4. Sequencing API calls to vary the order in which the functionality is exercised and to make the API produce useful results from successive calls.

By analyzing the problems listed above, a strategy needs to be formulated for testing the API. The API to be tested would require some environment for it to work. Hence it is required that all the conditions and prerequisites understood by the tester. The next step would be to identify and study its points of entry. The GUIs would have items like menus, buttons, check boxes, and combo lists that would trigger the event or action to be taken. Similarly, for APIs, the input parameters, the events that trigger the API would act as the point of entry. Subsequently, a chief task is to analyze the points of entry as well as significant output items. The input parameters should be tested with the valid and invalid values using strategies like the boundary value analysis and equivalence partitioning. The fourth step is to understand the purpose of the routines, the contexts in which they are to be used. Once all this parameter selections and combinations are designed, different call sequences need to be explored.

The steps can be summarized as following

1. Identify the initial conditions required for testing.

2. Identify the parameters – Choosing the values of individual parameters.

3. Identify the combination of parameters – pick out the possible and applicable parameter combinations with multiple parameters.

4. Identify the order to make the calls – deciding the order in which to make the calls to force the API to exhibit its functionality.

5. Observe the output.

1.Identify the initial condition:

The testing of an API would depend largely on the environment in which it is to be tested. Hence initial condition plays a very vital role in understanding and verifying the behavior of the API under test. The initial conditions for testing APIs can be classified as

ü Mandatory pre-setters.

ü Behavioral pre-setters.

Mandatory Pre-setters

The execution of an API would require some minimal state, environment. These type of initial conditions are classified under the mandatory initialization (Mandatory pre-setters) for the API. For example, a non-static member function API requires an object to be created before it could be called. This is an essential activity required for invoking the API.

Behavioral pre-setters

To test the specific behavior of the API, some additional environmental state is required. These types of initial conditions are called the behavioral pre-setters category of Initial condition. These are optional conditions required by the API and need to be set before invoking the API under test thus influencing its behavior. Since these influence the behavior of the API under test, they are considered as additional inputs other than the parameters

Thus to test any API, the environment required should also be clearly understood and set up. Without these criteria, API under test might not function as required and leave the tester’s job undone.

2.Input/Parameter Selection: The list of valid input parameters need to be identified to verify that the interface actually performs the tasks that it was designed for. While there is no method that ensures this behavior will be tested completely, using inputs that return quantifiable and verifiable results is the next best thing. The different possible input values (valid and invalid) need to be identified and selected for testing. The techniques like the boundary values analysis and equivalence-partitioning need to be used while trying to consider the input parameter values. The boundary values or the limits that would lead to errors or exceptions need to be identified. It would also be helpful if the data structures and other components that use these data structures apart from the API are analyzed. The data structure can be loaded by using the other components and the API can be tested while the other component is accessing these data structures. Verify that all other dependent components functionality are not affected while the API accesses and manipulates the data structures

The availability of the source code to the testers would help in analyzing the various inputs values that could be possible for testing the API. It would also help in understanding the various paths which could be tested. Therefore, not only are testers required to understand the calls, but also all the constants and data types used by the interface.

3. Identify the combination of parameters: Parameter combinations are extremely important for exercising stored data and computation. In API calls, two independently valid values might cause a fault when used together which might not have occurred with the other combinational values. Therefore, a routine called with two parameters requires selection of values for one based on the value chosen for the other. Often the response of a routine to certain data combinations is incorrectly programmed due to the underlying complex logic.

The API needs to be tested taking into consideration the combination of different parameter. The number of possible combinations of parameters for each call is typically large. For a given set of parameters, if only the boundary values have been selected, the number of combinations, while relatively diminished, may still be prohibitively large. For example, consider an API which takes three parameters as input. The various combinations of different values for the input values and their combinations needs to be identified.

Parameter combination is further complicated by the function overloading capabilities of many modern programming languages. It is important to isolate the differences between such functions and take into account that their use is context driven. The APIs can also be tested to check that there are no memory leaks after they are called. This can be verified by continuously calling the API and observing the memory utilization.

4.Call Sequencing: When combinations of possible arguments to each individual call are unmanageable, the number of possible call sequences is infinite. Parameter selection and combination issues further complicate the problem call-sequencing problem. Faults caused by improper call sequences tend to give rise to some of the most dangerous problems in software. Most security vulnerabilities are caused by the execution of some such seemingly improbable sequences.

5.Observe the output: The outcome of an execution of an API depends upon the behavior of that API, the test condition and the environment. The outcome of an API can be at different ways i.e., some could generally return certain data or status but for some of the API's, it might not return or shall be just waiting for a period of time, triggering another event, modifying certain resource and so on.

The tester should be aware of the output that needs to be expected for the API under test. The outputs returned for various input values like valid/invalid, boundary values etc needs to be observed and analyzed to validate if they are as per the functionality. All the error codes returned and exceptions returned for all the input combinations should be evaluated.

API Testing Tools: There are many testing tools available. Depending on the level of testing required, different tools could be used. Some of the API testing tools available are mentioned here.

JVerify: This is from Man Machine Systems.

JVerify is a Java class/API testing tool that supports a unique invasive testing model. The invasive model allows access to the internals (private elements) of any Java object from within a test script. The ability to invade class internals facilitates more effective testing at class level, since controllability and observability are enhanced. This can be very valuable when a class has not been designed for testability.

JavaSpec: JavaSpec is a SunTest's API testing tool. It can be used to test Java applications and libraries through their API. JavaSpec guides the users through the entire test creation process and lets them focus on the most critical aspects of testing. Once the user has entered the test data and assertions, JavaSpec automatically generates self-checking tests, HTML test documentation, and detailed test reports.

Here is an example of how to automate the API testing.

Assumptions: -

1. Test engineer is supposed to test some API.

2. The API’s are available in form of library (.lib).

3. Test engineer has the API document.

There are mainly two things to test in API testing: -

1. Black box testing of the API’s

2. Interaction / integration testing of the API’s.

By black box testing of the API mean that we have to test the API for outputs. In simple words when we give a known input (parameters to the API) then we also know the ideal output. So we have to check for the actual out put against the idle output.

For this we can write a simple c program that will do the following: -

a) Take the parameters from a text file (this file will contain many of such input parameters).

b) Call the API with these parameters.

c) Match the actual and idle output and also check the parameters for good values that are passed with reference (pointers).

d) Log the result.

Secondly we have test the integration of the API’s.

For example there are two API’s say

Handle h = handle createcontext (void);

When the handle to the device is to be closed then the corresponding function

Bool bishandledeleted = bool deletecontext (handle &h);

Here we have to call the two API’s and check if they are handled by the created createcontext () and are deleted by the deletecontext ().

This will ensure that these two API’s are working fine.

For this we can write a simple c program that will do the following: -

a) Call the two API’s in the same order.

b) Pass the output parameter of the first as the input of the second

c) Check for the output parameter of the second API

d) Log the result.

The example is over simplified but this works because we are using this kind of test tool for extensive regression testing of our API library.