Live Image editor with JavaScript, Canvas API and Tesseract.js(OCR)

Posted on:August 27, 2023 at 06:01 AM

Edited image of nebula and mantis from Guardians Of the galaxy

One of my favorite charting libraries, chart.js is built around the canvas,

The goat of ML and Deep learning tensorflow.js, can use webgl for computation, which is faster than both the CPU and WebAssembly.

Guess what, the canvas API provides a webgl context:

const canvas = document.createElement("canvas");
const webgl = canvas.getContext("webgl");

All this is usually reserved for lower level languages, it’s undeniable the browser is powerful, made even more by web API’s,

granting us lower level acess and control,

We can animate, do game graphics/engine, physics simulation, data visualization, photo-manipulation, real-time video processing etc etc

all with the canvas.

This article will explore one of those possibilities, pixel by pixel image manipulation.

Table of contents

Open Table of contents

The Canvas API: an overview

A book on the power of the canvas and potential applications can span 1000’s of pages,

We will cover only the necessary basics,

create a new project folder, with the following structure:

 imgs
 src\
   app.js
 index.html

Copy and Paste the below HTML starter:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Live Img editor</title>

    <style>
        *, *::before, *::after {
            box-sizing: border-box;
        }

        body{
            margin: 0;
            padding: 0;
            height: 100vh;

        }

        .graphics{
            width: 100%;
            height: 100%;
            display: flex;
            align-items: center;
            justify-content: center;
            gap: 5em;
            padding: .9em;
        }
    </style>
</head>
<body>

   <div class="graphics">
     <div style="display: grid;">
         <h2>Edit</h2>
         <canvas id="canvas" width="600" height="400"></canvas>
    </div>
         <div style="display: grid;">
            <h2>Original</h2>
            <img src="./imgs/1.jpg" width="400" height="255" id="i"/>
         </div>
     </div>

   </div>


   <script type="module" src="./src/app.js"></script>
</body>
</html>

Clone the repo to access the image files, which are optional you can use your own:

  git clone https://github.com/SfundoMhlungu/graphics-live-img-editor.git

Navigate to app.js, it is always advisable to listen for the content loaded event before any operation:

// app.js

document.addEventListener("DOMContentLoaded", () => {
  /**
   * @type {HTMLCanvasElement}
   */
  const canvas = document.getElementById("canvas");

  /**
   * @type {CanvasRenderingContext2D}
   */
  const ctx = canvas.getContext("2d");
});

We are getting a reference to the canvas, and from the canvas the 2d context,

giving us an API for drawing 2d graphics onto the canvas,

for example let’s draw a simple square in the middle:

ctx.fillStyle = "green";

ctx.fillRect(
  canvas.clientWidth / 2 - 100,
  canvas.clientHeight / 2 - 100,
  200,
  200
);

The canvas API is imperative, we have to provide every instruction, as compared to declarative programming

We are first telling the context, the color to use from now on, meaning anything we drawn under ctx.fillStyle = "green" will be filled or styled with green,

for example to draw a yellow square after the green, we have to explicitly specify

ctx.fillStyle = "green";

ctx.fillRect(
  canvas.clientWidth / 2 - 100,
  canvas.clientHeight / 2 - 100,
  200,
  200
);

ctx.fillStyle = "yellow";
ctx.fillRect(
  canvas.clientWidth / 2 - 100,
  canvas.clientHeight / 2 - 100,
  200,
  200
);

The canvas provides a coordinate system with the top left being (0, 0), going to the right on the x axis is +n, to left -n, n being a number of pixels.

The y axis is flipped, +n goes down and -n goes up.

Graphics Coordinate System and Units example

for example the fillRect, takes coordinates(x, y), width(w) and height(h)

ctx.fillRect(x, y, w, h);

We can draw all sorts of complex shapes, using the canvas, consult this article for more basics.

It will be an injustice, to not show the potential of the canvas with a complex example,

One of my favorite books I read as beginner, was Dan’s The Nature of Code book, originally written in Java,

JavaScript p5 version

I translated most of it directly to the canvas API, and I will be posting articles related to it, in some form in the near future,

graphics programming is fun.

The example below is comprised of content spanning over 148 pages of the nature of code,

to make it easier I created a simple npm package for education purposes,

in the same folder initialize a package.json file:

  npm init

and install the package:

npm i natureofcode-canvas-ed

It only has two classes: Vector and Mover, the first for vector math and the latter a physics object.

create a new script under src particles.js and comment out app.js and the image for now:

<!-- <img src="./imgs/1.jpg" width="400" height="255" id="i"/> -->

<script type="module" src="./src/particles.js"></script>
<!-- <script type="module" src="./src/app.js"></script> -->

copy and paste the following in particles.js:

import { Vector, Mover } from "natureofcode-canvas-ed";

/**
 * @type {HTMLCanvasElement}
 */
const canvas = document.getElementById("canvas");

/**
 * @type {CanvasRenderingContext2D}
 */
const ctx = canvas.getContext("2d");

canvas.width = window.innerWidth;
canvas.height = window.innerHeight;

canvas.width = window.innerWidth;
canvas.height = window.innerHeight;

let particle = new Mover(5, 100, 20, 200, [1, 1, 1], canvas);
// blowing to positive x direction
const wind = new Vector(0.01, 0);
const gravity = new Vector(0, 1);

let particles = [];

function update() {
  // clear the canvas before we draw
  ctx.clearRect(0, 0, canvas.width, canvas.height);

  for (let i = 0; i < 5; i++) {
    particles.push(
      new Mover(1, 200, 20, 100 * i, [(155 * i) ^ 2, 155, 155 * i], canvas)
    );
  }

  particles.forEach(p => {
    // p.applyForce(gravity)
    p.applyForce(wind);
    p.checkEdges();
    p.update();
    p.display();
  });

  for (let i = particles.length - 1; i > 0; i--) {
    if (particles[i].finished()) {
      // console.log("removed")
      particles.splice(i, 1);
    }
  }

  particle.applyForce(gravity);
  particle.applyForce(wind);
  particle.checkEdges();
  particle.update();
  particle.display(false);
  requestAnimationFrame(update);
}

update();

Use parcel to bundle and run:

 npx parcel index.html

The code below is creating a physics object,

bound by physical laws(nature) e.g wind, gravity etc,

The mover object can respond/simulate to these laws

params:

Mover(mass, x, y, lifespan, color, canvas)

let particle = new Mover(5, 100, 20, 200, [1, 1, 1], canvas);
constructor(m, x, y, (life = 200), (color = [155, 155, 155]), canvas);

the update function, applys gravity and wind to the mover object on every tick:

particle.applyForce(gravity);
particle.applyForce(wind);

check for bounds and commit calculations:

particle.checkEdges();
particle.update();
particle.display(false);

You can play with forces, for example making the wind stronger:

const wind = new Vector(0.1, 0);

Change the direction etc;

const wind = new Vector(0.2, 0.15);

physics canvas gif

The canvas can do more and beyond, on top of being fun, we will certainly explore all of this in upcoming articles.

Now let’s move on to the issue at hand,

let’s go back to app.js

<img src="./imgs/1.jpg" width="400" height="255" id="i" />

<!-- <script type="module" src="./src/particles.js"></script>-->
<script type="module" src="./src/app.js"></script>

Image manipulation

Comment out the lines creating squares, we need the canvas and the context for image manipulation

First we need to turn our image to pixel data, we can load the image into the canvas from the image tag,

but for this tutorial the <img> will serve as a prop or preview only, we will create a new Image in JS with:

// app.js

const img = new Image();

// handle events

img.src = "";

This article will be followed by a chrome extension version where we take this Image editor and combine it with Optical Character Recognition(OCR), to extract text in images.

You can find the OCR part of the tutorial on dev.to, It’s a only a 4 minute read,

In the extension the editor will allow users to enhance the image, for better text extraction by the OCR engine.

Here is the entire code to load the image, before we implement the function to paint into the canvas:

const img = new Image();
img.onload = function () {
  // paint into the canvas
  drawImageOnCanvas(img, canvas, ctx);
};
img.src = "./imgs/1.jpg";

the drawImageOnCanvas will put the image into the canvas, turning it into pixel data

create a new file in the src directory imageEdit.js and export the following function:

/**
 *
 * @param {HTMLImageElement} img
 * @param {HTMLCanvasElement} canvas
 *  @param {CanvasRenderingContext2D} ctx
 */
export function drawImageOnCanvas(img, canvas, ctx) {
  const maxSide = Math.max(img.width, img.height);
  const scale = 400 / maxSide;
  const canvasWidth = img.width * scale;
  const canvasHeight = img.height * scale;
  canvas.width = canvasWidth;
  canvas.height = canvasHeight;
  ctx.clearRect(0, 0, canvasWidth, canvasHeight);
  ctx.drawImage(img, 0, 0, canvasWidth, canvasHeight);
}

The image size may not be the same size as the canvas, an image can be for example 3000 * 2400 pixels, we cannot fit that onto a screen,

the first few lines handle that, by scaling the image down, to change the scale of the image tweak the 400 value, in the scale variable const scale = 400 / maxSide;

const maxSide = Math.max(img.width, img.height);
const scale = 400 / maxSide;
const canvasWidth = img.width * scale;
const canvasHeight = img.height * scale;

The following function is responsible for turning the image to pixel data, which we need for manipulation:

ctx.drawImage(img, 0, 0, canvasWidth, canvasHeight);

Taking as params the image, x, y and size.

let’s look at the pixel data

Pixel data

To get the pixel data from the canvas we use getImageData:

let pixelData = ctx.getImageData(0, 0, canvasWidth, canvasHeight);

which returns an object, with a property data, a one dimensional array containing the pixel data:

pixelData.data;

Now a pixel is made up of four values rgba(red, green, blue and alpha), meaning when we process the array we stride by 4,

in simple terms the data only makes sense if we process the values, four at a time:

Image data Array Drawing

which forms the repetition of rbga, we will talk about data representation in the upcoming ML in JavaScript series.

for example to print the first pixel

const r = pixelData.data[0];
const g = pixelData.data[1];
const b = pixelData.data[2];
const a = pixelData.data[3];

The second pixel we move by four.

Let’s create a grayscale function:

/**
 *
 * @param {CanvasRenderingContext2D} ctx
 */
export function grayScale(ctx, canvasWidth, canvasHeight) {
  let toEditImage = ctx.getImageData(0, 0, canvasWidth, canvasHeight);

  for (let i = 0; i < toEditImage.data.length; i += 4) {
    const avg =
      (toEditImage.data[i] +
        toEditImage.data[i + 1] +
        toEditImage.data[i + 2]) /
      3;

    toEditImage.data[i] =
      toEditImage.data[i + 1] =
      toEditImage.data[i + 2] =
        avg;

    toEditImage.data[i + 3] = toEditImage.data[i + 3];
  }

  ctx.putImageData(toEditImage, 0, 0);
}

To get the grayscale value of a pixel, we get the average of the pixel’s color space,

To know how much to darken it:

// getting the average
const avg =
  (toEditImage.data[i] + toEditImage.data[i + 1] + toEditImage.data[i + 2]) / 3;

after we get the average we darken rgb while preserving the alpha value:

toEditImage.data[i] = toEditImage.data[i + 1] = toEditImage.data[i + 2] = avg;
toEditImage.data[i + 3] = toEditImage.data[i + 3];

we do this for all the pixels in the image, from top to bottom, when the process is complete we draw the image back to the canvas

ctx.putImageData(toEditImage, 0, 0);

Let’s test it out by importing the grayscale function into app.js

we will take a programmatic approach for this article, only focusing on functionality

we will worry about the UI for the extension part

import { drawImageOnCanvas, grayScale } from "./ImageEdit.js";

const img = new Image();
img.onload = function () {
  drawImageOnCanvas(img, canvas, ctx);
  grayScale(ctx, canvas.width, canvas.height);
};
img.src = "./imgs/1.jpg";

The image should appear darker, another thing we can do is isolate color channels by turning off the unwanted

by turning, we set their values to 0, navigate to imageEdit.js:

export function colorChanel(ctx, canvas) {
  let editedImage = ctx.getImageData(0, 0, canvas.width, canvas.height);

  for (let i = 0; i < editedImage.data.length; i += 4) {
    // editedImage.data[i] = 0
    // editedImage.data[i + 1] = 0
    // turning off blue
    editedImage.data[i + 2] = 0;
  }
  ctx.putImageData(editedImage, 0, 0);
}

the above turns off blue, you can uncomment out any, you will see the difference,

How about putting a tint color on all the pixels:

/**
 *
 * @param {CanvasRenderingContext2D} ctx
 */
export function tint(ctx, tintColor, canvasWidth, canvasHeight) {
  const editedImage = ctx.getImageData(0, 0, canvasWidth, canvasHeight);

  //  tintColor = [255, 0, 0]; // Red tint color

  for (let i = 0; i < editedImage.data.length; i += 4) {
    editedImage.data[i] = editedImage.data[i] + tintColor[0];
    editedImage.data[i + 1] = editedImage.data[i + 1] + tintColor[1];
    editedImage.data[i + 2] = editedImage.data[i + 2] + tintColor[2];
    editedImage.data[i + 3] = editedImage.data[i + 3]; // Preserve alpha value
  }

  ctx.putImageData(editedImage, 0, 0);
}

We can keep going there’s a lot we can do with a pixel space: rotate, scale, compress etc,

This example is enough for our purpose, to test this import the tint and color channel functions in app js

import {
  drawImageOnCanvas,
  grayScale,
  tint,
  colorChanel,
} from "./ImageEdit.js";

Testing tint:

img.onload = function () {
  drawImageOnCanvas(img, canvas, ctx);
  // grayScale(ctx, canvas.width, canvas.height)
  tint(ctx, [32, 38, 51], canvas.width, canvas.height);
};

Isolating color channels:

img.onload = function () {
  drawImageOnCanvas(img, canvas, ctx);
  // grayScale(ctx, canvas.width, canvas.height)
  // tint(ctx, [32, 38, 51], canvas.width, canvas.height)
  colorChanel(ctx, canvas);
};

You can modify colorChanel to accept a switch case, on what channels to isolate,

After all this is done we need a way to save our modified image into an actual image(png/jpg etc)

Saving the Image

The canvas provides a convenient method for that:

toDataURL("image/jpeg", 1.0);

The first param is the type and second the quality, 1.0 being the highest

let’s replace the preview image with edited one:

const img = new Image();
img.onload = function () {
  // resizeImgtoCanvas(canvas, ctx, img)
  drawImageOnCanvas(img, canvas, ctx);
  // grayScale(ctx, canvas.width, canvas.height)
  colorChanel(ctx, canvas);
  // tint(ctx, [32, 38, 51], canvas.width, canvas.height)
  setTimeout(() => {
    const i = document.getElementById("i");

    i.src = canvas.toDataURL("image/jpeg", 1.0);
  }, 2000);
};
img.src = "./imgs/1.jpg";

After 2 seconds the edited image should be the preview:

We can add more functionality like scale, rotate etc, more complex but easy to google, to avoid a long blog

We will pause here, until we pick this up for the OCR chrome extension.

In this article we looked at image manipulation using the canvas

Which allows us to turn an image to pixel data, in preparation for the combined image editor and OCR chrome extension.

Thank you for reading, if you want be to updated on new articles you can follow me on twitter,

I will strive to be active there.

Other than that, I hope to see you next time, for more code adventures