ImgBot Install now

Documentation

Here you will find all the reference information on ImgBot. Anything you need to know about how it works or how to contribute can be found below.

ImgBot <3 contributions.

All of the ImgBot documentation is checked into GitHub. If there is any issue or something that is missing please feel free to open an issue and/or pull request to get it fixed up.

What is ImgBot?

Images are one of the key parts of almost any website or application, but they also play a really big role in performance degradation. The more images you have, the slower your load times can become.

There are a lot of tactics for updating your images to be more performant like cropping and resizing, but running a lossless compression is something you should always do to optimize the delivery without sacrificing size or quality.

ImgBot does just that and takes the heavy lifting out for you. With just a couple of clicks you can get ImgBot running in your project's repo on GitHub and sending you pull requests with your images optimized.

As you continue making further optimizations to the design and delivery of your application, ImgBot will be there optimizing your images for you the whole time.

Compression

ImgBot uses ImageMagick to optimize images through the LosslessCompress() routine from the The .NET wrapper Magick.NET. This means that while the file size is going down, the quality and dimensions remain intact.

The file size is measured before and after compression and the results are reported in the commit message body.

The images are compressed in place after cloning the repo.

Once installed into any repo, ImgBot will run the LosslessCompress() routine on an ongoing basis and open PRs to keep your images optimized.

Configuration

ImgBot supports optional configuration through a .imgbotconfig json file. This is not a required step to using ImgBot and is only for more advanced scenarios.

This file should be placed in the root of the repository and set to your liking. See this past issue for details about the location for this file.

Here is an example .imgbotconfig setup that shows some of the options.

{
    "schedule": "daily" // daily|weekly|monthly
    "ignoredFiles": [
        "*.jpg",                   // ignore by extension
        "image1.png",              // ignore by filename
        "public/special_images/*", // ignore by folderpath
    ]
}

If there are any configuration settings you would like to see supported, please feel free to open an issue in the repo or shoot an email over to ImgBotHelp@gmail.com

Schedule

You can set a schedule for your ImgBot PRs through the .imgbotconfig file.

  • This configuration is optional and is only required if you want less frequent pull requests
  • Accepts the following syntax: daily|weekly|monthly
  • Limits the PRs from ImgBot to once a day, once a week, or once a month respectively
  • The default behavior is to receive ImgBot PRs as images require optimization

This will effectively set the maximum frequency of pull requests submitted from ImgBot to your repo. This is ideal for projects that receive many image updates and don't want to see optimizations until they can all be batched up together. Most projects will benefit the most form the default setting of receiving image optimizations from ImgBot as new images are added to the project.

If you do want to set a schedule for ImgBot pull requests the current supported values are 'daily', 'weekly', and 'monthly'.

.imgbotconfig

{
    "schedule": "weekly"
}

Ignoring images

Some users may want to keep some images from being optimized. There can be a variety of reasons to want to keep an image in it's original state that differ from project to project.

If this is something you need in your project, ImgBot offers an ignore option.

  • This configuration is optional and is only required if there are specific images or folders of images you do not want touched
  • Accepts the syntax for searchPattern on Directory.EnumerateFiles()
  • Limits the images optimized by ImgBot by essentially ignoring them
  • When ignoring by filename no path is necessary
  • When ignoring by folder name full path from root is necessary

.imgbotconfig

Ignoring by extension

"ignoredFiles": [
    "*.jpeg"
]

Ignoring all images in a specific folder

"ignoredFiles": [
    "public/special_images/*"
]

Ignoring individual image files

"ignoredFiles": [
    "special-image1.png",
    "other-image1.png"
]

Authorization

After you install ImgBot into your repo, this is registered with GitHub and you are issued an InstallationID. ImgBot itself has an ApplicationID and a private key that is kept secret. With a combination of the InstallationID, the ApplicationID, and the private key, ImgBot can request an Installation Token from GitHub.

This Installation Token is valid for 10 minutes at a time and provides the permissions agreed to at the time of installation. If the permissions required by ImgBot change you will be notified and have to accept before they can take effect.

ImgBot specifically uses this Installation Token to clone the repo, create a branch, push a commit to the branch, and open pull requests. These actions are performed on behalf of the installation with a username of 'x-access-token' and a password set to the value of the Installation Token.

The author and committer for these acctions are set to ImgBotApp<ImgBotHelp@gmail.com>.

Furthermore, the commits pushed by ImgBot are digitally signed with a secure and registered PGP private key and password and verified against a public key registered with GitHub. This verification provides an extra level of security to ensure that commits that come into your repo are in fact coming from the authentic ImgBot. And you get the nice green verified badge in the PR.

If at anytime you want to remove ImgBot's ability to interact with your repo, you simply uninstall the application from your settings and all further action's will be immediately blocked with no access.

Contributing to ImgBot

All the code for ImgBot is available on GitHub. We will gladly accept contributions for the service, the website, and the documentation. If you are unsure what to work on, but still want to contribute you can look for an existing issue in the repo.

The following is where you can find out how to get set up to run locally as well as detailed information on exactly how ImgBot works.

ImgBot Service

The core of ImgBot runs on a serverless stack called Azure Functions. The Function apps are running the image compression, pushing the commits, and opening the Pull Requests. Once you get the tools you need to work with Azure functions you can run the apps locally.

You can either get the tools integrated with Visual Studio and use F5 or you can get the CLI standalone and use func run ImgBot.Function. If you are using Visual Studio for Mac there is built-in support for Azure functions.

Azure Functions operate on top of storage. To run the function locally you will need to bring your own storage account and add a local.settings.json in the root with AzureWebJobsStorage/AzureWebJobsDashboard filled out.

You can see the schema of this file in the doc

Alternatively, running func init from the function directory will stamp out everything for you, or if you have a deployed function you can fetch the storage it is using.

func azure functionapp fetch-app-settings <functionName>

Now that you are running the service locally, within the root of the repo you will see the following directories:

  • CompressImagesFunction - The function that does the work of cloning, compressing, and pushing commits
  • Install - The shared library to provide Installation Tokens to the functions
  • OpenPrFunction - The function that opens Pull Requests
  • RouterFunction - The orchestration layer

The following file locations may be helpful if you are looking for specific functionality:

  • CommitMessage.cs - generation of commit message compression report
  • CompressImages.cs - clones the repo, compresses the images, pushes the commit
  • CommitSignature.cs - uses a pgp private key and password to sign a commit message
  • ImageQuery.cs - searches and extracts all the images out of the repo to be optimized
  • InstallationToken.cs - uses a pem file and the GitHub API to get access to repos
  • LocalPath.cs - generates the location to clone from
  • PullRequest.cs - opens the pull request and sets the title and description
  • Schedule.cs - logic to limit the frequency of ImgBot PRs

Triggers

ImgBot uses QueueTriggers to kick off workflows.

The triggers in place today are routermessage, openprmessage, compressimagesmessage.

Triggers are the entry points for the service.

Compression workflow

ImgBot uses LibGit2Sharp to perform all the git operations.

The following is the high-level workflow for the CompressImagesFunction:

  1. Clone the repo
  2. Create the 'imgbot' branch
  3. Run the optimize routine
  4. Commit the changes
  5. Push the 'imgbot' branch to the remote

The clone directory is read from environment variable %TMP% or falls back to /private/tmp/ if this environment variable is not set.

Once the branch is pushed, ImgBot uses Octokit to create the pull request in GitHub.

Installation tokens

ImgBot uses BouncyCastle to help generate an installation token.

This requires a combination of a Private Key, an App Id, and an InstallationId to generate a token to be used on behalf of the installation.

The benefit of using installation tokens is that the user is in full control of the permissions at all times. We never have to store any tokens and they expire after about 10 minutes.

The installation token serves as a password to clone private repos, push to remotes, and open pull requests.

The username to accompany the installation token password is x-access-token.

For security reasons we cannot provide contributors with a pem file as this is a secret that delegates permissions in GitHub. You can run every part of the function except parts where authentication is required without this secret. If you are working on a part of the function that requires this secret then you can generate one for yourself to test with. Register a GitHub app for development purpose and install this app into the repo you are using to test with. Set the AppId in the code and you should be good to go.

If there is a part of this process that isn't clear or you have any questions at all feel free to open an issue in this repo and we'll work it out :)

Schedules

This Schedule class is responsible for throttling optimization routines. ImgBot is triggered when there is a new image added to a repo and by default will submit a PR as soon as it can.

Some users prefer to defer the pull requests and do the optimization in bigger batches. This is implemented by offering three options 'daily', 'weekly', and 'monthly'.

ImgBot will check the commit log to find the last time we committed an optimization. If it has been long enough since ImgBot has last committed in this branch then we will try to optimize the images again. Otherwise we will skip running optimizations for this run.

Commit messages

ImgBot uses a standard commit title and generates a report of the image optimizations to be used in the commit message body.

The input is a dictionary where the key is the filename and the value is a pair of numbers that represent file size before and after compression.

This dictionary is transformed into an optimization report in the form of a commit message.

Image query

ImgBot locates all the images that are to be sent through the optimization routine with file directory access against a local clone.

The known image extensions are used to find all the images recursively and the ignored files from imgbotconfig are parsed.

Local Paths

For each execution, ImgBot generates the folder for the git operations to take place in.

Today this is done by combining the name of the repo with a random number.

ImgBot website

The frontend that drives https://imgbot.net/ is a lightweight web app running on ASP.NET Core. This framework runs on all operating systems :)

The purpose of this website is to run the landing page and docs for ImgBot as well as an endpoint for the webhooks from GitHub for installed repos.

The website uses bootstrap for a grid. To copy the lib files out of node_modules we use grunt (one time only).

npm run copy-libs

Within the Web directory you will see the following key files

  • Views/Home/Index.cshtml - the landing page markup
  • Views/Home/Docs.cshtml - the docs page markup
  • Views/Shared/_Layout.cshtml - the layout for the landing page
  • wwwroot/css/site.less - the landing page stylesheet
  • Controllers/HookController.cs - the route for the webhooks
  • Controllers/HomeController.cs - the routes for the landing page and the docs

The stylesheet is compiled using a grunt task mapped in the package.json.

npm run compile-less

The stylesheet can be compiled on save through grunt as well.

npm run watch

The 2 main events we deal with through webhooks are installation events and push events. Each time a repo has ImgBot installed, GitHub fires a hook to HookController.cs and we start the installation workflow.

Each time a repo that already has ImgBot installed gets pushed to, GitHub fires a hook to HookController.cs and, if the commit contains an image update, we start the compression worflow.

You need to make a connection to a real storage account to run the website. To do this you can add the appsettings.json locally with the following format. Note: this should be the same storage account as the service is using for Azure Functions to enable message passing from the website to the service.

{
  "Logging": {
    "IncludeScopes": false,
    "LogLevel": {
      "Default": "Warning"
    }
  },
    "Storage": {
        "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=ACCOUNT_NAME;AccountKey=ACCOUNT_KEY;EndpointSuffix=core.windows.net"
    }
}

ImgBot Docs

The docs are published from checked in markdown files to the ImgBot website to view in a browser. Alternatively, the docs can be browsed and edited within GitHub.

To work on the docs within the context of the website locally ImgBot uses a gruntjs task to compile the markdown into HTML. To compile the docs locally run the following npm commands.

npm install
npm run compile-docs

To compile the markdown automatically on save

npm run watch

Check out the gruntfile.js and package.json to see how it is configured.

When the docs are compiled to HTML they are copied into the Web/Docs directory. In this directory there is a metadata.json file that will define the order and title of the doc. For example:

{
    "slug": "contributing-website",
    "title": "Contribute to ImgBot website"
}

The slug matches the name of the markdown file and also drives the URL segment to browse this doc.

This metadata file is read within the HomeController.cs file. The controller arranges all the documentation in memory in preparation for rendering.

The template that renders the documentation is Docs.cshtml. This is a razor file that renders the documentation navigation and content as HTML.

Tests

ImgBot uses VSTest with NSubstitue to manage unit testing of the logic within the ImgBot codebase. The tests live in the Test directory. Please do your best to add tests as new features or logic are added so we can keep Imgbot running smoothly. :)