I want to round BigDecimal decimal part according to the following rules:

1710.10 becomes 1710.10
1710.11 becomes 1710.15
1710.15 becomes 1710.15
1710.16 becomes 1710.20

I tried this way new BigDecimal("1710.11").setScale(2, RoundingMode.HALF_UP)), expecting to get 1710.15 but I get 1710.11. I also tried with Apache Math Utils and Decimal Format but no way to achieve this.

Round decimals up to multiples of 5 with BigDecimal
Your misunderstanding
First of all setting the scale on a BigDecimal only returns a BigDecimal with a different value if your current BigDecimal has a higher scale than you specified.
For example
new BigDecimal("1710.113").setScale(2, RoundingMode.HALF_UP)

returns a new BigDecimal equal to new BigDecimal("1710.11")
However, if you already have a scale of 2 or lower in this example, the new BigDecimal stays the same (read: equal).
HALF_UP simply means that a BigDecimal ending on a 5 will result in the higher value, e.g.
new BigDecimal("1710.115").setScale(2, RoundingMode.HALF_UP)

returns a result equal to new BigDecimal("1710.12")
Now to your question
Since your way of "rounding" doesn't actually change the scale, I doubt there is an already existing function you could use. However, doing it by hand is actually quite simple:
public class QuickMathz {

    public static void main(String[] args) {
        System.out.println(roundToNext5(new BigDecimal("1710.10"), 2));
        System.out.println(roundToNext5(new BigDecimal("1710.11"), 2));
        System.out.println(roundToNext5(new BigDecimal("1710.15"), 2));
        System.out.println(roundToNext5(new BigDecimal("1710.16"), 2));
        System.out.println(roundToNext5(new BigDecimal("1710.1135"), 2));
        System.out.println(roundToNext5(new BigDecimal("1710.1635"), 2));
        System.out.println(roundToNext5(new BigDecimal("1710.1675"), 2));
        System.out.println();
        System.out.println(roundToNext5(new BigDecimal("1710.10"), 3));
        System.out.println(roundToNext5(new BigDecimal("1710.11"), 3));
        System.out.println(roundToNext5(new BigDecimal("1710.15"), 3));
        System.out.println(roundToNext5(new BigDecimal("1710.16"), 3));
        System.out.println(roundToNext5(new BigDecimal("1710.1135"), 3));
        System.out.println(roundToNext5(new BigDecimal("1710.1635"), 3));
        System.out.println(roundToNext5(new BigDecimal("1710.1675"), 3));
    }

    public static BigDecimal roundToNext5(BigDecimal bigDecimal, int scale) {
        // Get the last digit we need to decide if we have to round to 0, 5 or 10
        int lastDigit = bigDecimal
                .movePointRight(scale)
                .remainder(BigDecimal.TEN).intValue();

        // Setting the Scale to scale - 1 to remove one more digit than we need
        // and then increase the scale to what we want
        BigDecimal result = bigDecimal
                .setScale(scale - 1, RoundingMode.DOWN)
                .setScale(scale, RoundingMode.UNNECESSARY);

        if (lastDigit == 0) {
            // Last digit is a 0 upscaling adds a 0
            return result;
        } else if (lastDigit <= 5) {
            // rounding up to 5
            return result.add(new BigDecimal("5").movePointLeft(scale));
        } else {
            // rounding up to 10
            return result.add(new BigDecimal("1").movePointLeft(scale - 1));
        }
    }
}

This class yields the output of
1710.10
1710.15
1710.15
1710.20
1710.15
1710.20
1710.20

1710.100
1710.110
1710.150
1710.160
1710.115
1710.165
1710.170

(It's neither optimized nor checked for negative values, so don't use in critical environments)
