ImgBot Install now

Documentation

Here you will find all the reference information on ImgBot. Anything you need to know about how it works or how to contribute can be found below.

ImgBot <3 contributions.

All of the ImgBot documentation is checked into Github. If there is any issue or something that is missing please feel free to open an issue and/or pull request to get it fixed up.

What is ImgBot?

Images are one of the key parts of almost any website or application, but they also play a really big role in performance degradation. The more images you have, the slower your load times can become.

There are a lot of tactics for updating your images to be more performant like cropping and resizing, but running a lossless compression is something you should always do to optimize the delivery without sacrificing size or quality.

ImgBot does just that and takes the heavy lifting out for you. With just a couple of clicks you can get ImgBot running in your project's repo on GitHub and sending you pull requests with your images optimized.

As you continue making further optimizations to the design and delivery of your application, ImgBot will be there optimizing your images for you the whole time.

Compression

ImgBot uses ImageMagick to optimize images through the LosslessCompress() routine from the The .NET wrapper Magick.NET. This means that while the file size is going down, the quality and dimensions remain intact.

The file size is measured before and after compression and the results are reported in the commit message body.

The images are compressed in place after cloning the repo.

Once installed into any repo, ImgBot will run the LosslessCompress() routine on an ongoing basis and open PRs to keep your images optimized.

Configuration

ImgBot supports optional configuration through a .imgbotconfig json file. This is not a required step to using ImgBot and is only for more advanced scenarios.

This file should be placed in the root of the repository and set to your liking. There is an open issue for supporting alternate locations for this file besides the root of the repo.

Here is an example .imgbotconfig setup that shows some of the options.

{
    "schedule": "daily" // daily|weekly|monthly
    "ignoredFiles": [
        "*.jpg",                   // ignore by extension
        "image1.png",              // ignore by filename
        "public/special_images/*", // ignore by folderpath
    ]
}

If there are any configuration settings you would like to see supported, please feel free to open an issue here in the repo or shoot an email over to ImgBotHelp@gmail.com

Schedule

You can set a schedule for your ImgBot PRs through the .imgbotconfig file.

  • This configuration is optional and is only required if you want less frequent pull requests
  • Accepts the following syntax: daily|weekly|monthly
  • Limits the PRs from ImgBot to once a day, once a week, or once a month respectively
  • The default behavior is to receive ImgBot PRs as images require optimization

This will effectively set the maximum frequency of pull requests submitted from ImgBot to your repo. This is ideal for projects that receive many image updates and don't want to see optimizations until they can all be batched up together. Most projects will benefit the most form the default setting of receiving image optimizations from ImgBot as new images are added to the project.

If you do want to set a schedule for ImgBot pull requests the current supported values are 'daily', 'weekly', and 'monthly'.

.imgbotconfig

{
    "schedule": "weekly"
}

Ignoring images

Some users may want to keep some images from being optimized. There can be a variety of reasons to want to keep an image in it's original state that differ from project to project.

If this is something you need in your project, ImgBot offers an ignore option.

  • This configuration is optional and is only required if there are specific images or folders of images you do not want touched
  • Accepts the syntax for searchPattern on Directory.EnumerateFiles()
  • Limits the images optimized by ImgBot by essentially ignoring them
  • When ignoring by filename no path is necessary
  • When ignoring by folder name full path from root is necessary

.imgbotconfig

Ignoring by extension

"ignoredFiles": [
    "*.jpeg"
]

Ignoring all images in a specific folder

"ignoredFiles": [
    "public/special_images/*"
]

Ignoring individual image files

"ignoredFiles": [
    "special-image1.png",
    "other-image1.png"
]

Authorization

After you install ImgBot into your repo, this is registered with GitHub and you are issued an InstallationID. ImgBot itself has an ApplicationID and a private key that is kept secret. With a combination of the InstallationID, the ApplicationID, and the private key, ImgBot can request an Installation Token from GitHub.

This Installation Token is valid for 10 minutes at a time and provides the permissions agreed to at the time of installation. If the permissions required by ImgBot change you will be notified and have to accept before they can take effect.

ImgBot specifically uses this Installation Token to clone the repo, create a branch, push a commit to the branch, and open pull requests. These actions are performed on behalf of the installation with a username of 'x-access-token' and a password set to the value of the Installation Token.

The author and committer for these acctions are set to ImgBotApp<ImgBotHelp@gmail.com>.

Furthermore, the commits pushed by ImgBot are digitally signed with a secure and registered PGP private key and password and verified against a public key registered with GitHub. This verification provides an extra level of security to ensure that commits that come into your repo are in fact coming from the authentic ImgBot. And you get the nice green verified badge in the PR.

If at anytime you want to remove ImgBot's ability to interact with your repo, you simply uninstall the application from your settings and all further action's will be immediately blocked with no access.

Contributing to ImgBot

All the code for ImgBot is available on GitHub. We will gladly accept contributions for the service, the website, and the documentation. This is where you can find out how to get set up to run locally as well as detailed information on exactly how ImgBot works.

ImgBot Service

The core of ImgBot runs on a serverless stack called Azure Functions. The Function app is running the image compression, pushing the commits, and opening the Pull Requests. Once you get the tools you need to work with Azure functions you can run the app locally from the ImgBot.Function directory.

You can either get the tools integrated with Visual Studio and use F5 or you can get the CLI standalone and use func run ImgBot.Function.

The CLI is built with JavaScript so all operating systems are welcome :)

Azure Functions operate on top of storage. To run the function locally you will need to bring your own storage account and add a local.settings.json in the root with AzureWebJobsStorage/AzureWebJobsDashboard filled out.

You can see the schema of this file in the doc

Alternatively, running func init from the function directory will stamp out everything for you, or if you have a deployed function you can fetch the storage it is using.

func azure functionapp fetch-app-settings <functionName>

Now that you are running the service locally, within the ImgBot.Function directory you will see the following key classes

  • Functions.cs - the triggers for starting the workflows
  • CompressImages.cs - clones the repo, compresses the images, opens the PR
  • InstallationToken.cs - uses a pem file and the GitHub API to get access to repos
  • Schedule.cs - logic to limit the frequency of ImgBot PRs
  • CommitMessage.cs - generation of commit message compression report
  • ImageQuery.cs - Searches and extracts all the images out of the repo to be optimized
  • LocalPath.cs - Generates the location to clone from

Functions.cs

ImgBot uses QueueTriggers to kick off workflows.

The two triggers in place today are imageupdatemessage and installationmessage. Image update is the event that performs ongoing image optimizations for repos that already have ImgBot installed. Installation is the initial event for freshly installed repos.

This class is the entry point for the service. It is responsible for logging the installation in a storage table and kicking off the compression routine.

CompressImages.cs

ImgBot uses LibGit2Sharp to perform all the git operations.

This class works through the following steps:

  1. Clones the repo
  2. Creates the 'imgbot' branch
  3. Runs the optimize routine
  4. Commits the changes
  5. Pushes the 'imgbot' branch to the remote

The clone directory is read from environment variable %TMP%.

Once the branch is pushed, ImgBot uses Octokit to create the pull request in GitHub.

InstallationToken.cs

ImgBot uses BouncyCastle to help generate an installation token.

This class uses a combination of a Private Key, an App Id, and an InstallationId to generate a token to be used on behalf of the installation.

The benefit of using installation tokens is that the user is in full control of the permissions at all times. We never have to store any tokens and they expire after 10 minutes.

The installation token serves as a password to clone private repos, push to remotes, and open pull requests.

The username to accompany the installation token password is x-access-token.

For security reasons we cannot provide you with a pem file as this is a secret that delegates permissions in GitHub. You can run every part of the function except parts where authentication is required without this secret. If you are working on a part of the function that requires this secret then you can generate one for yourself to test with. Register a GitHub app for development purpose and install this app into the repo you are using to test with. Set the AppId and you should be good to go.

If there is a part of this process that isn't clear or you have any questions at all feel free to open an issue in this repo and we'll work it out :)

Schedule.cs

This class is responsible for throttling optimization routines. ImgBot is triggered when there is a new image added to a repo and by default will submit a PR as soon as it can.

Some users prefer to defer the pull requests and do the optimization in bigger batches. This is implemented by offering three options 'daily', 'weekly', and 'monthly'.

ImgBot will check the commit log to find the last time we committed an optimization. If it has been long enough since ImgBot has last committed in this branch then we will try to optimize the images again. Otherwise we will skip running optimizations for this run.

CommitMessage.cs

This class is responsible for generating a report of the image optimizations to be used in the commit message body.

The input is a dictionary where the key is the filename and the value is a pair of numbers that represent file size before and after compression.

This dictionary is transformed into an optimization report in the form of a commit message.

ImageQuery.cs

This class is responsible for locating all the images that are to be sent through the optimization routine.

The known image extensions are used to find all the images recursively and the ignored files from imgbotconfig are parsed.

LocalPath.cs

This class is responsible for generating the folder for the git operations to take place in.

Today this is done by combining the name of the repo with a random number.

ImgBot website

The frontend that drives https://imgbot.net/ is a lightweight web app running on ASP.NET Core. This framework runs on all operating systems :)

The purpose of this website is to run the landing page and docs for ImgBot as well as an endpoint for the webhooks from GitHub for installed repos.

The website uses bootstrap for a grid. To copy the lib files out of node_modules we use grunt (one time only).

npm run copy-libs

Within the ImgBot.Web directory you will see the following key files

  • Views/Home/Index.cshtml - the landing page markup
  • Views/Shared/_Layout.cshtml - the layout for the landing page
  • wwwroot/css/site.less - the landing page stylesheet
  • Controllers/HookController.cs - the route for the webhooks
  • Controllers/HomeController.cs - the routes for the landing page and the docs

The stylesheet is compiled using a grunt task mapped in the package.json.

npm run compile-less

The stylesheet can be compiled on save through grunt as well.

npm run watch

The 2 main events we deal with through webhooks are installation events and push events. Each time a repo has ImgBot installed, GitHub fires a hook to HookController.cs and we start the installation workflow.

Each time a repo that already has ImgBot installed gets pushed to, GitHub fires a hook to HookController.cs and, if the commit contains an image update, we start the compression worflow.

If you need to make a connection to a real storage account you can update the appsettings.json locally to replace the ACCOUNT_NAME and ACCOUNT_KEY placeholders there. This should be the same storage account as the service is using for Azure Functions to enable message passing from the website to the service.

ImgBot Docs

The docs are published from checked in markdown files to the ImgBot website to view in a browser. Alternatively, the docs can be browsed and edited within GitHub.

To work on the docs within the context of the website locally ImgBot uses a gruntjs task to compile the markdown into HTML. To compile the docs locally run the following npm commands.

npm install
npm run compile-docs

To compile the markdown automatically on save

npm run watch

Check out the gruntfile.js and package.json to see how it is configured.

When the docs are compiled to HTML they are copied into the ImgBot.Web/Docs directory. In this directory there is a metadata.json file that will define the order and title of the doc.

{
    "slug": "contributing-website",
    "title": "Contribute to ImgBot website"
}

The slug matches the name of the markdown file and is also drives the URL segment to browse this doc.

This metadata file is read within the HomeController.cs file. The controller arranges all the documentation in memory in preparation for rendering.

The template that renders the documentation is Docs.cshtml. This is a razor file that renders the documentation navigation and content as HTML.

Tests

ImgBot uses VSTest with NSubstitue to manage unit testing of the logic within the ImgBot codebase. The tests live in the ImgBot.Test directory. Please do your best to add tests as new features or logic are added so we can keep Imgbot running smoothly. :)