An image is made up of a number of pixels arranged in a 2-dimensional grid; rows and columns. The resolution of the image is a product of the number of rows and columns, with the rows representing the height of the image and columns representing the width of the image. A 1080x720 image has a total of 777,600 pixels, and each pixel comprises of a number of colors referred to as channels. A digital image would typically consist of 3 channels; red, green, and blue. Greyscale images consists of just a single channel; black or white. Each pixel is represented by 8-bits (~a byte) with values ranging from 0-255. Whenever effects such as blurs are applied to images, under the hood, it's a set of mathematical operations applied directly on pixels to achieve the desired results.

There are various techniques used to blur images, some of which are;

• Mean filters
• Gaussian blurs
• Bilateral filters

A popular approach to applying filters is the Kernel Convolution, where a set of numbers (kernel) is passed accross the pixels of an image. A kernel is a small matrix used for blurring, and other image effects. The size of the kernel must be odd, and the greater the value of the kernel, the greater the blurring effect. The kernel is applied to every pixel in the image, with the currently selected pixel at the center of the kernel. Each value in the kernel is multiplied with the respective pixel values within the matrix range highlighted by the kernel and summed up together. The central pixel value is eventually replaced with the result of these operations.

Native approach to blurring images

To implement the mean filter in android, we can obtain the pixels by converting the image to bitmap, applying the kernel to every pixel. We'll be applying the blurs to this sample image:

Here's my implementation for this:

private suspend fun meanFilter(bitmap: Bitmap) {
        val kernel = floatArrayOf(1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f, 1 / 9f)
        return withContext(Dispatchers.IO) {
            val diameter = 3 // 3x3 matrix
            val cols = bitmap.width
            val rows = bitmap.height
            val step = (diameter - 1) / 2

            val imagePixels = IntArray(rows * cols)
            val newImagePixels = IntArray(rows * cols)

            bitmap.getPixels(imagePixels, 0, cols, 0, 0, cols, rows) // here we are assigning all pixel values to the IntArray so we can process the individual pixels via the kernel

            for (i in 0 until cols) {
                for (j in 0 until rows) {
                    val startCoord = Point(j - step, i - step)
                    val endCoord = Point(j + step, i + step)

                    val matrix = ArrayList<Int>()
                    for (_col in startCoord.y..endCoord.y) {
                        for (_row in startCoord.x..endCoord.x) {
                            if (_row < 0 || _row >= rows || _col < 0 || _col >= cols) {
                                //ignore pixels out of matrix bounds
                                matrix.add(0)
                            } else {
                                val pixel = imagePixels[_row * cols + _col]
                                matrix.add(pixel)
                            }
                        }
                    }

                    val sum = matrix.mapIndexed { index, value ->
                        val multiplier = kernel[index]

                        val alpha = value shr 24 and 0xFF
                        val red = ((value ushr 16 and 0xFF) * multiplier).toInt()
                        val green = ((value ushr 8 and 0xFF) * multiplier).toInt()
                        val blue = ((value and 0xFF) * multiplier).toInt()
                        ((alpha) shl 24) or ((red) shl 16) or ((green) shl 8) or (blue)
                    }.sum()
                    newImagePixels[j * cols + i] = sum
                }
            }
            Bitmap.createBitmap(newImagePixels, cols, rows, Bitmap.Config.ARGB_8888)
        }
    }

You'll notice we did some bitwise shift operations on the pixel values before applying the kernel on them, that is because every pixel is made up of certain 8-bit color values, and these values need to be isolated before being processed, then combined again to form a pixel value.

Blurring with OpenCV

Now that we have an idea on how blur filters work natively, a useful image processing library that does a lot of these special type of operations faster is OpenCV. Here, images can be blurred more accurately using just a few lines of code compared to what we initially wrote. You should have OpenCV setup with a new or existing android project to get started with implementing them in your application.

Trying out the same blurring filter above with OpenCV;

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {

            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)
            val size = 24.0

            val destination = Mat()
            Imgproc.blur(imageSrc, destination, Size(size, size))

            val copy = Bitmap.createBitmap(
                destination.width(),
                destination.height(),
                Bitmap.Config.ARGB_8888
            )
            Utils.matToBitmap(destination, copy)
            copy
        }
    }

Here's the output:

Pretty straightfoward eh? Great! Well, what if we have to only blur only a particular segment of the image, instead of the whole image. We'll use the submat() function to select our region of interest, and only blur those parts we need. As an example, we'll try to blur an image starting from 30% away from the origin at the top and left side of the image, having a width/height that is 30% of the image size.

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {

            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)
            val size = 24.0

            val rows = imageSrc.rows()
            val cols = imageSrc.cols()

            val rowStart = (0.3 * rows).toInt()
            val rowEnd = (0.6 * rows).toInt()
            val colStart = (0.3 * cols).toInt()
            val colEnd = (0.6 * cols).toInt()

            val subRegion = imageSrc.submat(rowStart, rowEnd, colStart, colEnd)
            Imgproc.blur(subRegion, subRegion, Size(size, size))

            val copy = Bitmap.createBitmap(
                imageSrc.width(),
                imageSrc.height(),
                Bitmap.Config.ARGB_8888
            )
            Utils.matToBitmap(imageSrc, copy)
            copy
        }
    }

Image output:

This is a bit different from the previous implementation, and the explanation is quite simple. the submat function accepts a number of arguments that defines rect points we need as our region of interest. Even though this function attempts to select just a segment from the original image and returns it, it still holds a direct reference to the original image so anything we do with the subregion reflects onn the selected region, and also inadvertently affects the original image. Here's an explanation for it

Now, we have a bit of control over the kind of blurring operations we need for our images, but not so much control as we would want. With the previous example, we can only blur out certain segments in rectangular shapes only. What if we need to make the blur masks to be based off various forms such as circles, rectangles, or even freehand shapes. How do we achieve this using OpenCV?

Advanced blurring in OpenCV

For this example, we'll try reimplementing the rectangular blurring using a more dynamic approach before trying out something different. OpenCV consists of a number of methods that allows for creating masks in various shapes we want ranging from rectangles, triangles, and other polygons.

For rectangles, we'll create a rectangular mask in black and white and perform bitwise operations on this mask so that our blurring operations can occur within these masks only. To create a rectangular mask in OpenCV, we'll make use of the Imgproc.rectangle() function.

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {
            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)

            val rows = imageSrc.rows()
            val cols = imageSrc.cols()

            val startY = (0.3*rows).toInt()
            val endY = (0.7*rows).toInt()
            val startX = (0.3*cols).toInt()
            val endX = (0.7*cols).toInt()

            val innerMask = Mat.zeros(imageSrc.size(), imageSrc.type())
            innerMask.setTo(Scalar.all(0.0))

            //thickness set to -1 for inner fill
            Imgproc.rectangle(
                innerMask,
                Rect(startX,startY,endX,endY),
                Scalar.all(255.0),
                -1
            )

            val outerMask = Mat()
            Core.bitwise_not(innerMask, outerMask)

            val blurredImage = Mat()
            Imgproc.blur(imageSrc, blurredImage, Size(64.0, 64.0))

            //original image minus cutout
            val originalCut = Mat()
            Core.bitwise_and(imageSrc, outerMask, originalCut)

            //blurred image minus cutout
            val blurredCut = Mat()
            Core.bitwise_and(blurredImage, innerMask, blurredCut)

            val merge = Mat()
            Core.bitwise_or(originalCut, blurredCut, merge)

            val copy = Bitmap.createBitmap(merge.width(), merge.height(), Bitmap.Config.ARGB_8888)
            Utils.matToBitmap(merge, copy)
            copy
        }
    }

And here's the result:

Here's an explanation of what is going on.

val innerMask = Mat.zeros(imageSrc.size(), imageSrc.type())
innerMask.setTo(Scalar.all(0.0))

Here we are creating an empty image, with pixel values set to 0, meaning this is a black image

Imgproc.rectangle(innerMask, Rect(startX,startY,endX,endY), Scalar.all(255.0), -1)

The above code means we are drawing a rectangle within the black image we created. The start and end points are provided within the rect object. This region is filled with a white mask via Scalar.all(255.0) having a thickness of -1. According to the docs, a negative thickness value enables the shape to be filled rather than having a stroke for the drawn shape.

At this point, we have a black image containing a white mask. We'll need this mask to perform some bitmanipulation and embed our blurred segment within this white mask, and the rest of the original image within the black region.

Here's the result for the inner mask:

Now, unto the next piece of code

val outerMask = Mat()
Core.bitwise_not(innerMask, outerMask)

From the black and white image, we're trying to invert the segments such that the black region becomes white, and the white region becomes black. It follows the basic principle of binary operators using the NOT gate. Such that when we have a value like 1110011 and a not gate is applied here, the ones (1) becomes zeros (0) and vice versa. Luckily, OpenCV has a function that allows for performing these forms of operations. In the above piece of code, we used Core.bitwise(srcImage, destinationImage) to achieve this.

At this point, the value of outer mask would appear like this:

For the outer mask, we'll attempt to copy the the orignal image pixel values to this variable using another bitwise operator, and. This also follows the same principle as bitwise operations when combining two binary numbers together, such that there can only be a truthy value when all values involved are non-zero, else we have a false value. This way the colored parts of the original image remain the same, while the black regions remain zero as required.

val originalCut = Mat()
Core.bitwise_and(imageSrc, outerMask, originalCut)

here's the result:

We'll be doing the opposite for our blurred image as well to obtain a blurred mask:

val blurredCut = Mat()
Core.bitwise_and(blurredImage, innerMask, blurredCut)

Result for this:

Finally, we combine the original cut and the blurred cut to achieve a single image that represents a merge between the two image. The black pixels would be added with the blurred pixels to yield a blurred pixel, thereby replacing all black pixels between both images. A bitwise operation that gurantees this form of idea is the or operator, whereby we have a truthy value as long as one of the values involved in the artihmetic operation is non-zero. Here's how it is implemented.

val merge = Mat()
Core.bitwise_or(originalCut, blurredCut, merge)

Now merge represents our true blurred region results we're trying to achieve as we did using submat. Here's the result you should be getting:

If we want to achieve a circular blur using this same approach, all we have to do is replace the Imgproc.rectangle() with Imgproc.circle() and pass in the required parameters required to draw the circle. Such as this:

val circleMask = Mat()
Imgproc.circle(circleMask, org.opencv.core.Point(rowEnd/2.0, rowEnd/2.0), rowEnd/2, Scalar(1.0))

One more thing...

Looking at the previous image, you'll notice there's a sharp edge between the blurred image bounds and the rest of the original image. What if we wanted to avoid this, such that the blurred region appears to blend alongside the original image as shown in the figure below?

We'll be doing something called an alpha blend, which is a process of overlaying a foreground image with transparency over a background image. At every pixel of the image, we need to combine the foreground image, with the background image using an alpha mask. The foreground image would be the blurred image, the background image is the original image, and the alpha mask would be represented by the blurred black and white mask. The formula for doing this operation is gotten from here:

Here's our implementation in kotlin:

    private fun alphaBlend(imageSrc:Mat, blurredImage: Mat, blurredMask: Mat): Mat{
        val destination = Mat(imageSrc.size(), imageSrc.type())

        for (col in 0 until destination.cols()){
            for (row in 0 until destination.rows()){
                val pixel = imageSrc.get(row, col)
                val blurredPixel = blurredImage.get(row, col)
                val blurVal = blurredMask.get(row, col)[0] / 255

                val newPixelValue = pixel.mapIndexed { index, value ->
                    (blurVal * blurredPixel[index]) + ((1.0f - blurVal) * value)
                }.toDoubleArray()

                destination.put(row, col, *newPixelValue)
            }
        }
        return destination
    }

We're getting every pixel from the blurred mask, dividing it by 255 such that we get a value ranging between 0 and 1. We then multiply this value with every blurred pixel, and subtract the same value with from 1.0 then multiply it with the original image. We finally obtain the new pixel value and assign it to our new image variable. The transparency would be 0 on the original and blurry image, thereby yielding the same pixel value, however this value varies as it approaches the edges of the blurry image, thus leading to the transparent blurry effect along the edges of the blurred mask.

Now we'll update our blurImage() function to match this blend function.

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {
            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)

            val innerMask = Mat.zeros(imageSrc.size(), imageSrc.type())

            //thickness set to -1 for inner fill
            Imgproc.circle(
                innerMask,
                Point(imageSrc.width() / 2.0, imageSrc.height() / 1.5),
                600,
                Scalar.all(255.0),
                -1
            )

            val blurredImage = Mat(imageSrc.size(), imageSrc.type())
            Imgproc.blur(imageSrc, blurredImage, Size(64.0, 64.0))

            val blurredMask = Mat(innerMask.size(), innerMask.type())
            Imgproc.blur(innerMask, blurredMask, Size(64.0, 64.0))

            val merge = alphaBlend(imageSrc, blurredImage, blurredMask)

            val copy = Bitmap.createBitmap(merge.width(), merge.height(), Bitmap.Config.ARGB_8888)
            Utils.matToBitmap(merge, copy)
            copy
        }
    }

Now this is the final result:

A new problem emerges

One thing you'll notice about this is that our blur process has become really slow, taking up to over 15 seconds to achieve the blur result. And this is because of two reasons;

• We are looping through every pixel in the 2-dimensional grid, meaning for a 1080x1080 image, we're actually iterating through the image 1,166,400 times which is quite a lot!
• We are accessing the pixels directly from the image using the get and set methods, which are fairly expensive operations.

A way out of this mess

After doing a bunch of research on various ways to optimze this process, I was finally able to figure out a way out of this thanks to this post which provided a couple of insights regarding OpenCV.

From this post, I figured the best way to optimize the entire process was to convert the pixels to primitives like IntArray/FloatArray and perform the operations on this object, the reassign the processed pixels back to the image object. The get/set operations would be faster than way, and could also allow for single interation of every pixel since the image is now represented as a one-dimensional array.

First, I would like to highlight some approaches that didn't work for me at first. I assigned the pixels to ByteArray, and performed the normal alpha blending operations on the pixels but for some reason, I wasn't able to achieve the desired result. Here's what I did:

    private fun alphaBlend(imageSrc:Mat, blurredImage: Mat, blurredMask: Mat): Mat{

        val total = imageSrc.width() * imageSrc.height()
        val channels = imageSrc.channels()
        val size = total * channels

        val array = ByteArray(size)
        val array1 = ByteArray(size)
        val array2 = ByteArray(size)

        val array3 = ByteArray(size)

        val destination = Mat(imageSrc.size(), imageSrc.type())

        imageSrc.get(0,0, array)
        blurredImage.get(0,0, array1)
        blurredMask.get(0,0, array2)

        for (index in 0 until size){
            val pixel = array[index]
            val blurredPixel = array1[index]
            val blurVal = (array2[index]) / 255.0f

            val newValue = ((blurVal * blurredPixel) + ((1.0f - blurVal) * pixel)).toInt().toByte()
            array3[index] = newValue
        }
        destination.put(0,0, array3)
        return destination
    }

Then I spent days figuring out why this didn't work, and decided to opt in for other valid primitives. Instead of using ByteArray, I opted for FloatArray since the alpha mask value ought to be a decimal value ranging from 0.0 to 1.0. After doing this, I encountered some errors thrown like this:

After a couple of hours spent on debugging, I realized that the data type of the images, ought to match the array primitives they're being assigned to due to certain conditions that have been implemented under the hood by OpenCV. What's even more surprising is that these conditions have been stated here in this post that helped me out with optimization, but overlooked at the time:

So what I did was simple, before assigning the pixels to the FloatArray primitive, I converted the images to the required data types for float arrays, like this:

imageSrc.convertTo(imageSrc, CV_32F)
blurredImage.convertTo(blurredImage, CV_32F)
blurredMask.convertTo(blurredMask, CV_32F)

Well the above approach solved the error, only that this form of conversion gave rise to another error as shown below:

Well as stated in the error message, before using Utils.MatToBitmap(), the mat object needs to be a datatype that is either CV_8UC1, CV_8UC3, or CV_8UC4. So the solution is to convert the destination mat object to either of these by;

destination.convertTo(destination, CV_8UC3)

Here's the full code implementation for the optimized image blur using alpha blending:

    private fun alphaBlend(imageSrc:Mat, blurredImage: Mat, blurredMask: Mat): Mat{

        val total = imageSrc.width() * imageSrc.height()
        val channels = imageSrc.channels()
        val size = total * channels

        val array = FloatArray(size)
        val array1 = FloatArray(size)
        val array2 = FloatArray(size)

        val array3 = FloatArray(size)
        imageSrc.convertTo(imageSrc, CV_32F)
        blurredImage.convertTo(blurredImage, CV_32F)
        blurredMask.convertTo(blurredMask, CV_32F)

        val destination = Mat(imageSrc.size(), imageSrc.type())

        imageSrc.get(0,0, array)
        blurredImage.get(0,0, array1)
        blurredMask.get(0,0, array2)

        for (index in 0 until size){
            val pixel = array[index]
            val blurredPixel = array1[index]
            val blurVal = (array2[index]) / 255.0f

            val newValue = ((blurVal * blurredPixel) + ((1.0f - blurVal) * pixel))
            array3[index] = newValue
        }
        destination.put(0,0, array3)
        destination.convertTo(destination, CV_8UC3)
        return destination
    }

    private suspend fun blurImage(bitmap: Bitmap):Bitmap {
        return withContext(Dispatchers.IO) {
            val imageSrc = Mat()

            Utils.bitmapToMat(bitmap, imageSrc)

            val innerMask = Mat.zeros(imageSrc.size(), imageSrc.type())

            //thickness set to -1 for inner fill
            Imgproc.circle(
                innerMask,
                Point(imageSrc.width() / 2.0, imageSrc.height() / 1.5),
                600,
                Scalar.all(255.0),
                -1
            )

            val blurredImage = Mat(imageSrc.size(), imageSrc.type())
            Imgproc.blur(imageSrc, blurredImage, Size(64.0, 64.0))

            val blurredMask = Mat(innerMask.size(), innerMask.type())
            Imgproc.blur(innerMask, blurredMask, Size(64.0, 64.0))

            val merge = alphaBlend(imageSrc, blurredImage, blurredMask)

            val copy = Bitmap.createBitmap(merge.width(), merge.height(), Bitmap.Config.ARGB_8888)
            Utils.matToBitmap(merge, copy)
            copy
        }
    }

This approach works well, is better optimized, and performs the blur operations WAY faster than the previous implementation since we are performing the bit manipulations using kotlin array primitives on a single iteration, rather than using the direct access method.

Here's a link to the gist. I'm open to different ideas, so if you think there is a more efficient/less lengthy way to achieve this, please let me know!😊