воскресенье, 27 мая 2012 г.

Junit and java 7.

Recently our project was moving from java 6 to java 7, and we have got many kinds of problems due to this process.
One of this problem was - tests execution order in junit 4.

In general, the execution order of tests should not be determinated.
Each test should be run independently from the others.
You can read it from junit documentation. But when you having thousands of tests and they fails when running in random order - this is a very big problem and you will spend a lot of time on refactoring all tests, make them independent. I would like to share with you my temporary solution of this problem.

Junit uses reflection for getting and executing test methods.
Junit get list of all tests using "Method[] getDeclaredMethods()" from java.lang.Class.
You can read from javadoc of this method or from junit docs that:
"The elements in the array returned are not sorted and are not in any particular order.", but in previous jvm implemetation methods list was ordered as they were in source code, and someone was writting tests which depends on each others. It was wrong, and now we have this problem and need some quick solution. Now they are in random order, and this is why this problem appeared.


We will sort all tests before junit will execute them. We will change some junit classes, and then put our new classes before junit jar file in our classpath.
Junit 4 has two different basic runners clases: JUnit3Builder and JUnit4Builder from packege org.junit.internal.builders. JUnit3Builder uses for old junit tests (without annotations and others).
And our new runners:


package org.junit.internal.builders;

import custom.junit.runners.OrderedJUnit3ClassRunner;
import org.junit.internal.runners.JUnit38ClassRunner;
import org.junit.runner.Runner;
import org.junit.runners.model.RunnerBuilder;

public class JUnit3Builder extends RunnerBuilder {

    public Runner runnerForClass(Class testClass) throws Throwable {
        if (isPre4Test(testClass)) {
//            return new JUnit38ClassRunner(testClass);
            return new OrderedJUnit3ClassRunner(testClass);
        } else {
            return null;
        }
    }

    boolean isPre4Test(Class testClass) {
        return junit.framework.TestCase.class.isAssignableFrom(testClass);
    }
}

package org.junit.internal.builders;

import custom.junit.runners.OrderedJUnit4ClassRunner;
import org.junit.runner.Runner;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.RunnerBuilder;

/**
 * This must come first on the classpath before JUnit 4's jar so it
 * is instantiated instead of the default JUnit 4 builder.
 */

public class JUnit4Builder extends RunnerBuilder {
    @Override
    public Runner runnerForClass(Class<?> testClass) throws Throwable {

        // Using Class Runner with sorting of test methods:
        // sorting in order as they are in java code.
        return new OrderedJUnit4ClassRunner(testClass);

        // JUnit Original Class Runner
        // return new BlockJUnit4ClassRunner(testClass);
    }
}


For junit 3 we will use our custom implementation of TestSuite class. When somewhere in junit core will be added new test in test suite it will be sorted in our OrderedTestSuite.


package custom.junit.runners;

import custom.junit.framework.OrderedTestSuite;
import junit.framework.Test;
import junit.framework.TestCase;
import org.apache.log4j.Logger;
import org.junit.internal.runners.JUnit38ClassRunner;

public class OrderedJUnit3ClassRunner extends JUnit38ClassRunner {

    private static final Logger logger = Logger.getLogger(OrderedJUnit3ClassRunner.class.getName());

    public OrderedJUnit3ClassRunner(Class<?> aClass) {
        this(new OrderedTestSuite(aClass.asSubclass(TestCase.class)));
    }

    public OrderedJUnit3ClassRunner(Test test) {
        super(test);
        logger.info("Using custom JUNIT CLASS RUNNER: " + this.getClass().getCanonicalName());
    }
} 

package custom.junit.framework;

import custom.junit.runners.MethodComparator;
import junit.framework.Test;
import junit.framework.TestCase;
import junit.framework.TestSuite;
import org.apache.log4j.Logger;

import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class OrderedTestSuite extends TestSuite {

    private static final Logger logger = Logger.getLogger(OrderedTestSuite.class.getName());

    public OrderedTestSuite(final Class<?> theClass) {
        addTestsFromTestCase(theClass);
    }

    /**
     * Adds the tests from the given class to the suite
     */

    @Override
    public void addTestSuite(Class<? extends TestCase> testClass) {
        addTest(new OrderedTestSuite(testClass));
    }

    private void addTestsFromTestCase(final Class<?> theClass) {
        this.setName(theClass.getName());
        try {
            getTestConstructor(theClass); // Avoid generating multiple error messages
        } catch (NoSuchMethodException e) {
            addTest(warning("Class " + theClass.getName() + " has no public constructor TestCase(String name) or TestCase()"));
            return;
        }

        if (!Modifier.isPublic(theClass.getModifiers())) {
            addTest(warning("Class " + theClass.getName() + " is not public"));
            return;
        }

        Class<?> superClass = theClass;
        List<String> names = new ArrayList<String>();
        while (Test.class.isAssignableFrom(superClass)) {
            Method[] methods = superClass.getDeclaredMethods();

            // Sorting methods.
            final List<Method> methodList = new ArrayList<Method>(Arrays.asList(methods));
            try {
                Collections.sort(methodList, MethodComparator.getMethodComparatorForJUnit3());
                methods = methodList.toArray(new Method[methodList.size()]);
            } catch (Throwable throwable) {
                logger.fatal("addTestsFromTestCase(): Error while sorting test cases! Using default order (random).", throwable);
            }

            for (Method each : methods) {
                addTestMethod(each, names, theClass);
            }
            superClass = superClass.getSuperclass();
        }
        if (this.testCount() == 0)
            addTest(warning("No tests found in " + theClass.getName()));
    }

    private void addTestMethod(Method m, List<String> names, Class<?> theClass) {
        String name = m.getName();
        if (names.contains(name))
            return;
        if (!isPublicTestMethod(m)) {
            if (isTestMethod(m))
                addTest(warning("Test method isn't public: " + m.getName() + "(" + theClass.getCanonicalName() + ")"));
            return;
        }
        names.add(name);
        addTest(createTest(theClass, name));
    }

    private boolean isPublicTestMethod(Method m) {
        return isTestMethod(m) && Modifier.isPublic(m.getModifiers());
    }

    private boolean isTestMethod(Method m) {
        return m.getParameterTypes().length == 0 && m.getName().startsWith("test") && m.getReturnType().equals(Void.TYPE);
    }
}

For junit 4 we can override just one special method - "protected List<FrameworkMethod> computeTestMethods()".


package custom.junit.runners;

import org.apache.log4j.Logger;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class OrderedJUnit4ClassRunner extends BlockJUnit4ClassRunner {

    private static final Logger logger = Logger.getLogger(OrderedJUnit4ClassRunner.class.getName());

    public OrderedJUnit4ClassRunner(Class aClass) throws InitializationError {
        super(aClass);
        logger.info("Using custom JUNIT CLASS RUNNER: " + this.getClass().getCanonicalName());
    }

    @Override
    protected List<FrameworkMethod> computeTestMethods() {
        final List<FrameworkMethod> list = super.computeTestMethods();
        try {
            final List<FrameworkMethod> copy = new ArrayList<FrameworkMethod>(list);
            Collections.sort(copy, MethodComparator.getFrameworkMethodComparatorForJUnit4());
            return copy;
        } catch (Throwable throwable) {
            logger.fatal("computeTestMethods(): Error while sorting test cases! Using default order (random).", throwable);
            return list;
        }
    }
}

And finally a comparator class which will sort our junit tests as we want. Of cource you can make your own comparator and sort all tests in different order. My comparator use tests order just like it was in your source code.


package custom.junit.runners;

import org.apache.log4j.Logger;
import org.junit.Ignore;
import org.junit.runners.model.FrameworkMethod;

import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.LineNumberReader;
import java.lang.reflect.Method;
import java.util.Comparator;

public class MethodComparator<T> implements Comparator<T> {

    private static final Logger logger = Logger.getLogger(MethodComparator.class.getName());

    private static final char[] METHOD_SEPARATORS = {17};

    private MethodComparator() {
    }

    public static MethodComparator<FrameworkMethod> getFrameworkMethodComparatorForJUnit4() {
        return new MethodComparator<FrameworkMethod>();
    }

    public static MethodComparator<Method> getMethodComparatorForJUnit3() {
        return new MethodComparator<Method>();
    }

    @Override
    public int compare(T o1, T o2) {
        final MethodPosition methodPosition1 = this.getIndexOfMethodPosition(o1);
        final MethodPosition methodPosition2 = this.getIndexOfMethodPosition(o2);
        return methodPosition1.compareTo(methodPosition2);
    }

    private MethodPosition getIndexOfMethodPosition(final Object method) {
        if (method instanceof FrameworkMethod) {
            return this.getIndexOfMethodPosition((FrameworkMethod) method);
        } else if (method instanceof Method) {
            return this.getIndexOfMethodPosition((Method) method);
        } else {
            logger.error("getIndexOfMethodPosition(): Given object is not a method! Object class is "
                    + method.getClass().getCanonicalName());
            return new NullMethodPosition();
        }
    }

    private MethodPosition getIndexOfMethodPosition(final FrameworkMethod frameworkMethod) {
        return getIndexOfMethodPosition(frameworkMethod.getMethod());
    }

    private MethodPosition getIndexOfMethodPosition(final Method method) {
        final Class aClass = method.getDeclaringClass();
        if (method.getAnnotation(Ignore.class) == null) {
            return getIndexOfMethodPosition(aClass, method.getName());
        } else {
            logger.debug("getIndexOfMethodPosition(): Method is annotated as Ignored: " + method.getName()
                    + " in class: " + aClass.getCanonicalName());
            return new NullMethodPosition();
        }
    }

    private MethodPosition getIndexOfMethodPosition(final Class aClass, final String methodName) {
        MethodPosition methodPosition;
        for (final char methodSeparator : METHOD_SEPARATORS) {
            methodPosition = getIndexOfMethodPosition(aClass, methodName, methodSeparator);
            if (methodPosition instanceof NullMethodPosition) {
                logger.debug("getIndexOfMethodPosition(): Trying to use another method separator for method: " + methodName);
            } else {
                return methodPosition;
            }
        }
        return new NullMethodPosition();
    }

    private MethodPosition getIndexOfMethodPosition(final Class aClass, final String methodName, final char methodSeparator) {
        final InputStream inputStream = aClass.getResourceAsStream(aClass.getSimpleName() + ".class");
        final LineNumberReader lineNumberReader = new LineNumberReader(new InputStreamReader(inputStream));
        final String methodNameWithSeparator = methodName + methodSeparator;
        try {
            try {
                String line;
                while ((line = lineNumberReader.readLine()) != null) {
                    if (line.contains(methodNameWithSeparator)) {
                        return new MethodPosition(lineNumberReader.getLineNumber(), line.indexOf(methodNameWithSeparator));
                    }
                }
            } finally {
                lineNumberReader.close();
            }
        } catch (IOException e) {
            logger.error("getIndexOfMethodPosition(): Error while reading byte code of class " + aClass.getCanonicalName(), e);
            return new NullMethodPosition();
        }
        logger.warn("getIndexOfMethodPosition(): Can't find method " + methodName + " in byte code of class " + aClass.getCanonicalName());
        return new NullMethodPosition();
    }

    private static class MethodPosition implements Comparable<MethodPosition> {
        private final Integer lineNumber;
        private final Integer indexInLine;

        public MethodPosition(int lineNumber, int indexInLine) {
            this.lineNumber = lineNumber;
            this.indexInLine = indexInLine;
        }

        @Override
        public int compareTo(MethodPosition o) {

            // If line numbers are equal, then compare by indexes in this line.
            if (this.lineNumber.equals(o.lineNumber)) {
                return this.indexInLine.compareTo(o.indexInLine);
            } else {
                return this.lineNumber.compareTo(o.lineNumber);
            }
        }
    }

    private static class NullMethodPosition extends MethodPosition {
        public NullMethodPosition() {
            super(-1-1);
        }
    }
}

You can use some some advanced libs, like ASM, for code decompilation, but I just use method separators from java byte code.


Hope this post will help you.
Any comments are appreciated.

5 комментариев:

  1. Hi,

    Nice work.!

    1.But i couldn't understand the methodSeparator logic.., are you appending 1 or 7 with your test methods.?

    2.Then, the test method names should be really UNIQUE ..!! You should not use the same name (test method name) anywhere else in the class.? even for any variable name.

    Help me with these two..

    Nishok

    ОтветитьУдалить
    Ответы
    1. Hello, thx)
      1. In java byte code just after method name goes byte 1 or 7, so I'm searching this:
      "final String methodNameWithSeparator = methodName + methodSeparator;".
      2. You are right, I can't say exactly how it is in byte code, but there really can be that variable name follows also by 1 or 7 byte, and then it could broke my algorithm. You could use some lib for decompilation, for example ASM, and rewrite with it method "getIndexOfMethodPosition".
      My code is working with all our tests, so there are no collisions between test method names and other code. But if you really (for some reason) have tests with non unique names, then you should use decompilator, or may be explore byte code specifiaction and write your own "getIndexOfMethodPosition" using some reqular expression for searching start positions of every test method.

      Удалить
  2. If there are dependencies between tests then you can easily enforce order by grouping dependent tests and placing them in a single test method. That is IMHO the proper fix.

    ОтветитьУдалить
    Ответы
    1. It's impossible if you have big product with over than 5000+ tests.
      Also when that "composite" test will fail, it will be too hard to determine what happend.

      Удалить
  3. Thank you very much for going through the trouble of writing all this and then posting it here for us. I confirm that it worked perfectly on my JUnit4 test suites on first attempt.

    ОтветитьУдалить