Documentation

Here you will find all the reference information on ImgBot. Anything you need to know about how it works or how to contribute can be found below.

ImgBot <3 contributions.

All of the ImgBot documentation is checked into GitHub. If there is any issue or something that is missing please feel free to open an issue and/or pull request to get it fixed up.

What is ImgBot?

Images are one of the key parts of almost any website or application, but they also play a really big role in performance degradation. The more images you have, the slower your load times can become.

There are a lot of tactics for updating your images to be more performant like cropping and resizing, but running a lossless compression is something you should always do to optimize the delivery without sacrificing size or quality.

ImgBot does just that and takes the heavy lifting out for you. With just a couple of clicks you can get ImgBot running in your project's repo on GitHub and sending you pull requests with your images optimized.

As you continue making further optimizations to the design and delivery of your application, ImgBot will be there optimizing your images for you the whole time.

Compression

ImgBot uses ImageMagick to optimize images through the LosslessCompress() routine from the The .NET wrapper Magick.NET. This means that while the file size is going down, the quality and dimensions remain intact.

The file size is measured before and after compression and the results are reported in the commit message body.

The images are compressed in place after cloning the repo.

Once installed into any repo, ImgBot will run the LosslessCompress() routine on an ongoing basis and open PRs to keep your images optimized.

Configuration

ImgBot supports optional configuration through a .imgbotconfig json file. This is not a required step to using ImgBot and is only for more advanced scenarios.

This file should be placed in the root of the repository and set to your liking. See this past issue for details about the location for this file.

Here is an example .imgbotconfig setup that shows some of the options.

{
    "schedule": "daily", // daily|weekly|monthly
    "ignoredFiles": [
        "*.jpg",                   // ignore by extension
        "image1.png",              // ignore by filename
        "public/special_images/*", // ignore by folderpath
    ],
    "aggressiveCompression": "true" // true|false
}

If there are any configuration settings you would like to see supported, please feel free to open an issue in the repo or shoot an email over to help@imgbot.net

Schedule

You can set a schedule for your ImgBot PRs through the .imgbotconfig file.

  • This configuration is optional and is only required if you want less frequent pull requests
  • Accepts the following syntax: daily|weekly|monthly
  • Limits the PRs from ImgBot to once a day, once a week, or once a month respectively
  • The default behavior is to receive ImgBot PRs as images require optimization

This will effectively set the maximum frequency of pull requests submitted from ImgBot to your repo. This is ideal for projects that receive many image updates and don't want to see optimizations until they can all be batched up together. Most projects will benefit the most form the default setting of receiving image optimizations from ImgBot as new images are added to the project.

If you do want to set a schedule for ImgBot pull requests the current supported values are 'daily', 'weekly', and 'monthly'.

.imgbotconfig

{
    "schedule": "weekly"
}

Ignoring images

Some users may want to keep some images from being optimized. There can be a variety of reasons to want to keep an image in it's original state that differ from project to project.

If this is something you need in your project, ImgBot offers an ignore option.

  • This configuration is optional and is only required if there are specific images or folders of images you do not want touched
  • Accepts the syntax for searchPattern on Directory.EnumerateFiles()
  • Limits the images optimized by ImgBot by essentially ignoring them
  • When ignoring by filename no path is necessary
  • When ignoring by folder name full path from root is necessary

.imgbotconfig

Ignoring by extension

"ignoredFiles": [
    "*.jpeg"
]

Ignoring all images in a specific folder

"ignoredFiles": [
    "public/special_images/*"
]

Ignoring individual image files

"ignoredFiles": [
    "special-image1.png",
    "other-image1.png"
]

Aggressive compression

You can maximize ImgBot's compression through the .imgbotconfig file.

  • This configuration is optional and is only required if you require a more aggressive compression of images
  • Accepts the syntax: true|false
  • Uses a more aggressive compression, rather than the lossless compression that is used by default

.imgbotconfig

Using full compression capabilities

{
    "aggressiveCompression": true
}

Using lossless compression (default)

{
    "aggressiveCompression": false
}

Authorization

After you install ImgBot into your repo, this is registered with GitHub and you are issued an InstallationID. ImgBot itself has an ApplicationID and a private key that is kept secret. With a combination of the InstallationID, the ApplicationID, and the private key, ImgBot can request an Installation Token from GitHub.

This Installation Token is valid for 10 minutes at a time and provides the permissions agreed to at the time of installation. If the permissions required by ImgBot change you will be notified and have to accept before they can take effect.

ImgBot specifically uses this Installation Token to clone the repo, create a branch, push a commit to the branch, and open pull requests. These actions are performed on behalf of the installation with a username of 'x-access-token' and a password set to the value of the Installation Token.

The author and committer for these actions are set to ImgBotApp<ImgBotHelp@gmail.com>.

Furthermore, the commits pushed by ImgBot are digitally signed with a secure and registered PGP private key and password and verified against a public key registered with GitHub. This verification provides an extra level of security to ensure that commits that come into your repo are in fact coming from the authentic ImgBot. And you get the nice green verified badge in the PR.

If at anytime you want to remove ImgBot's ability to interact with your repo, you simply uninstall the application from your settings and all further action's will be immediately blocked with no access.

Contributing to ImgBot

All the code for ImgBot is available on GitHub. We will gladly accept contributions for the service, the website, and the documentation. If you are unsure what to work on, but still want to contribute you can look for an existing issue in the repo.

The following is where you can find out how to get set up to run locally as well as detailed information on exactly how ImgBot works.

ImgBot Service

The core of ImgBot runs on a serverless stack called Azure Functions. The Function apps are running the image compression, pushing the commits, and opening the Pull Requests. Once you get the tools you need to work with Azure functions you can run the apps locally.

You can either get the tools integrated with Visual Studio and use F5 or you can get the CLI standalone and use func run ImgBot.Function. If you are using Visual Studio for Mac there is built-in support for Azure functions.

We also have support for running with VS Code. You will still need to get the CLI as mentioned above and the C# extension for VS Code in order to compile and get intellisense. Each function has a task you can execute that will clean + build + run the process. To start one open the prompt with cmd/ctrl + shift + p and select Run task. From there you will see all the tasks checked into .vscode/tasks.json. Choose a function to run such as Run CompressImagesFunction and it will build and start up. To attach to this process choose the Debug a function configuration from the debug tab to see the running processes. Type func into the picker to see your running function and select it. It's a two-step process, the debugger and the function process. When you kill the debugger, the process will still be running. You can kill the function host by bringing up the prompt again with cmd/ctrl + shift + p and select Kill the active terminal instance.

Azure Functions operate on top of storage. To run the function locally you will need to bring your own storage account and add a local.settings.json in the root with AzureWebJobsStorage filled out and FUNCTIONS_WORKER_RUNTIME set to dotnet.

You can see the schema of this file in the doc

Alternatively, running func init from the function directory will stamp out everything for you, or if you have a deployed function you can fetch the storage it is using.

func azure functionapp fetch-app-settings <functionName>

Now that you are running the service locally, within the root of the repo you will see the following directories:

  • CompressImagesFunction - The function that does the work of cloning, compressing, and pushing commits
  • Install - The shared library to provide Installation Tokens to the functions
  • OpenPrFunction - The function that opens Pull Requests
  • RouterFunction - The orchestration layer
  • WebHook - The function that is triggered from GitHub activity

The following file locations may be helpful if you are looking for specific functionality:

  • CommitMessage.cs - generation of commit message compression report
  • CompressImages.cs - clones the repo, compresses the images, pushes the commit
  • CommitSignature.cs - uses a pgp private key and password to sign a commit message
  • ImageQuery.cs - searches and extracts all the images out of the repo to be optimized
  • InstallationToken.cs - uses a pem file and the GitHub API to get access to repos
  • LocalPath.cs - generates the location to clone from
  • PullRequest.cs - opens the pull request and sets the title and description
  • Schedule.cs - logic to limit the frequency of ImgBot PRs
  • WebHookFunction.cs - reads the GitHub hook messages and kicks off tasks

Triggers

ImgBot uses QueueTriggers to kick off workflows.

The triggers in place today are routermessage, openprmessage, compressimagesmessage.

ImgBot also uses an HttpTrigger as the entry point for the service.

Compression workflow

ImgBot uses LibGit2Sharp to perform all the git operations.

The following is the high-level workflow for the CompressImagesFunction:

  1. Clone the repo
  2. Create the 'imgbot' branch
  3. Run the optimize routine
  4. Commit the changes
  5. Push the 'imgbot' branch to the remote

The clone directory is read from environment variable %TMP% or falls back to /private/tmp/ if this environment variable is not set.

Once the branch is pushed, ImgBot uses Octokit to create the pull request in GitHub.

Installation tokens

ImgBot uses BouncyCastle to help generate an installation token.

This requires a combination of a Private Key, an App Id, and an InstallationId to generate a token to be used on behalf of the installation.

The benefit of using installation tokens is that the user is in full control of the permissions at all times. We never have to store any tokens and they expire after about 10 minutes.

The installation token serves as a password to clone private repos, push to remotes, and open pull requests.

The username to accompany the installation token password is x-access-token.

For security reasons we cannot provide contributors with a pem file as this is a secret that delegates permissions in GitHub. You can run every part of the function except parts where authentication is required without this secret. If you are working on a part of the function that requires this secret then you can generate one for yourself to test with. Register a GitHub app for development purpose and install this app into the repo you are using to test with. Set the AppId in the code and you should be good to go.

If there is a part of this process that isn't clear or you have any questions at all feel free to open an issue in this repo and we'll work it out :)

Schedules

This Schedule class is responsible for throttling optimization routines. ImgBot is triggered when there is a new image added to a repo and by default will submit a PR as soon as it can.

Some users prefer to defer the pull requests and do the optimization in bigger batches. This is implemented by offering three options 'daily', 'weekly', and 'monthly'.

ImgBot will check the commit log to find the last time we committed an optimization. If it has been long enough since ImgBot has last committed in this branch then we will try to optimize the images again. Otherwise we will skip running optimizations for this run.

Commit messages

ImgBot uses a standard commit title and generates a report of the image optimizations to be used in the commit message body.

The input is a dictionary where the key is the filename and the value is a pair of numbers that represent file size before and after compression.

This dictionary is transformed into an optimization report in the form of a commit message.

Image query

ImgBot locates all the images that are to be sent through the optimization routine with file directory access against a local clone.

The known image extensions are used to find all the images recursively and the ignored files from imgbotconfig are parsed.

Local Paths

For each execution, ImgBot generates the folder for the git operations to take place in.

Today this is done by combining the name of the repo with a random number.

Webhooks

The 2 main events we deal with through webhooks are installation events and push events. Each time a repo has ImgBot installed, GitHub fires a hook to WebHookFunction.cs and we start the installation workflow.

Each time a repo that already has ImgBot installed gets pushed to, GitHub fires a hook to WebHookFunction.cs and, if the commit contains an image update, we start the compression worflow.

ImgBot website

The frontend that drives https://imgbot.net/ is a generated static web app built with Grunt and a little bit of JavaScript. This static site is generated to be completely stand alone and hosted on a CDN for caching worldwide. The grid system for the ImgBot site is bootstrap 4. The purpose of this website is to run the landing page and docs for ImgBot.

You will find the package.json file for the website in the Web/ directory of the repo. From here the input files live in the src/ directory and the generated site is output to the dist/ directory and git-ignored.

To kick off the generation

npm run gen

To start a lightweight dev server at http://localhost:8888

npm run serve

To compile the site on save

npm run watch

Within the Web/ directory you will see the following key files

  • src/index.html - the landing page markup
  • src/docs/layout.jst - the docs page template
  • src/layout.jst - the layout for the landing pages
  • src/css/site.less - the landing page stylesheet
  • gruntfile.js - the task configuration for generating and serving the site

ImgBot Docs

The docs are published from checked in markdown files to the ImgBot website to view in a browser. Alternatively, the docs can be browsed and edited within GitHub.

When the docs are compiled to HTML they use the layout and metadata found in the Web/src/docs directory. The metadata.json file here will define the order and title of each doc. For example:

{
    "slug": "contributing-website",
    "title": "Contribute to ImgBot website"
}

The slug matches the name of the markdown file and also drives the URL segment to browse this doc.

This metadata file is read within the Web/tasks/compile-docs.js file. This is a custom grunt task that arranges all the documentation.

The template that renders the documentation is Web/src/docs/layout.jst. This is a template file that renders the documentation navigation and content as HTML.

Tests

ImgBot uses VSTest with NSubstitue to manage unit testing of the logic within the ImgBot codebase. The tests live in the Test directory. Please do your best to add tests as new features or logic are added so we can keep Imgbot running smoothly. :)