Continuous Integration with GitHub, SFDX and CircleCI… Easier than you think!

This post is a follow up/companion to the talk I did at Dreamforce 2018. If you didn’t get to see it in person, you can check out the slides here, and I will update this post when the recording becomes available, but for now, read on.

In the salesforce ecosystem, the traditional way of moving code from development environments (sandboxes, etc) to production has been either change sets, or the force.com migration tool (ANT). Neither method is perfect, but until recently they were all we had. Change sets are an easy, but time intensive process. They can be created in the user interface and are well within the reach of most people. The force.com migration tool was arguable more powerful (CLI based tool, able to be scripted, etc) but a lot more difficult to use. Neither tool was particularly well suited to an agile environment that required continuous integration or delivery.

So what do I mean when I say continuous integration? I am referring to both the development practice and the tooling required to facilitate it.

A good explanation of CI, taken from Microsoft’s Azure docs is as follows: “Continuous Integration (CI) is the process of automating the build and testing of code every time a team member commits changes to version control. CI encourages developers to share their code and unit tests by merging their changes into a shared version control repository after every small task completion. Committing code triggers an automated build system to grab the latest code from the shared repository and to build, test, and validate the full master branch (also known as the trunk or main).”

This has traditionally been difficult with salesforce, because we lacked the tooling to do it effectively. In the old world, salesforce development took an ‘org-centric’ view of the world, with your production org serving as the ‘source of truth’ and sandboxes containing work in progress. This ‘org-centric’ model has a number of problems (e.g prod can be changed by anyone, developing dependent features in separate environments, merge conflicts, etc).

Since the advent of Salesforce DX (SFDX) we have been handed the tools to move towards a ‘source-centric’ world, with our source/version control system (e.g Git) becoming the source of truth, and our scratch org’s essentially becoming ‘runtimes’ rather than org’s in their own right. Scratch orgs are ephemeral things that can be created and destroyed at will, only needing to live as long as the development cycle for whatever feature you are working on, or as long as it takes for code to be pushed to them and tested (if used in a CI pipeline)

Because of SFDX, we now get access to all of the power of modern source/version control systems, such as Git. Git providers like GitHub give us powerful user interfaces, the ability to perform code reviews (pull requests) with ease, the ability to track every single change with information like who changed the file, what was changed and when. As code is versioned we also now gain the ability to revert to previous versions of the source making it much easier to recover if something goes wrong. Along side this, we know that our code is stored safely outside of salesforce and we can set up access control to prevent code being overwritten by un-authorised users.

Another huge advantage we gain from source/version control is the ability to create branches of our source code. For example; you may have your ‘production’ ready application in the 'master' branch, with the version you are currently working on in the 'develop' branch (think of your UAT/pre-prod environment). Whenever you work on a new feature, you can make a copy of your 'master' branch in to its own new branch (e.g'feature/new-feature') and do the work there. Once you are happy with it, a pull request can be made to merge it in to develop for testing. Once this has been completed and all of the code in 'develop' is ready for release, this can then be merged in to 'master' for your release.

Source/version control is only half of the equation. It is all well and good to have your code in git, and this itself is valuable, but the real power comes from automation and continuous integration. When we have CI setup, every time we make a commit to our feature branch ('feature/new-feature') we are pulling it from git, pushing it to a scratch org and running all of our tests. This lets us know very quickly if a) our code even deploys and b) if we’ve broken any tests. We also use a scratch org ‘locally’ for running and testing our code on our local machine (of-course the scratch org actually runs in the cloud) in a similar way we would have used a developer sandbox in the past.

Once our code is ready for the next environment (e.g UAT) we can have our CI setup automatically push our code to our UAT sandbox whenever a commit is made to develop, finally, once we are happy and ready to move to our production org, we can then have our CI deploy to production upon a merge into the 'master' branch.

Example of CI Development Process

So lets talk about how we actually achieve this with salesforce. In this example I will be using GitHub as my source/version control system (other alternatives are BitBucket, GitLab) and CircleCI as my CI automation tool (other alternatives are TravisCI, Jenkins). The ‘glue’ that ties all of this together is SFDX.

I’ve created a GitHub repository with everything you need to get started, I would suggest you clone it from here as instructions below reference scripts from it. If you want to find out what the scripts are doing, simply open them in a text editor. These instructions require a *nix like environment (e.g macOS, Linux, Bash on windows) and have been tested on macOS only.

NOTE: The first time this build runs it will deploy whatever is in your force-app/ folder to production. As with anything you do, be sure to try this in a developer edition org, or a sandbox first before using this with your production org!

In this example there is only a simply ‘Hello World’ apex class, test and lightning component. You should remove these before you begin and replace with your own source code. Salesforce provide steps here on how to migrate your existing code to SFDX here, After you’ve cloned the repository to your machine you should follow the instructions here (inside the folder you clone to).

Now that we have our code ready to use with CI, lets get going.

Lets tackle authentication first. To authenticate to our production org and to create scratch orgs we are using JWT to do this.

  1. You first need to create a certificate and key to authenticate with. To do this you can run the script in build/generate-keys.sh
    • Follow the prompts when creating the certificate files
    • Take note of the Base64 output (big long chunk of text), as you will need this to set up CircleCI later

      Output from the key generation script
  2. You will need to create a connected app in your production (and any sandboxes you wish to use CI with)
    • First, from Setup, enter App in the Quick Find box, then select App Manager. Click New Connected App.
    • Give your application a name such as ‘CircleCI’
    • Make sure you check Enable OAuth Settings in the connected app
    • Set the OAuth callback to http://localhost:1717/OauthRedirect
    • Check Use Digital Signatures and add your certificate file (server.crt), this will be in the build/ folder. Once you have done this delete this file
    • Select the required OAuth scopes
    • Make sure that refresh is enabled – otherwise you’ll get this error: user hasn't approved this consumer
    • Ensure that Admin approved users are pre-authorized under Permitted Users is selected
    • Ensure that you allow the System Administrator profile is selected under the Profiles related list
    • Take note of the Consumer Key as you will need it for to setup CircleCI
Connected App Settings

Now that we have authentication setup, we can configure CircleCI. I have provided a basic config.yml file, this is already within the .circleci/ directory along with some shell scripts for circle to use for deployment and validation. Circle has extensive documentation on these config files here and stay tuned for a future post covering this area in more detail.

  1. You now can set up your CircleCI build
    • Ensure you have connected your GitHub account to CircleCI, to do this go https://www.circleci.com, click 'Signup' and then'Signup with GitHub'
      Adding a project in CircleCI

       

    • Once logged in, click on Add Projects choose your GitHub to use repository and click Set Up Project then click Start building there is an example config.yml in this repository already. You can edit this to suit your needs.
    • Cancel the first build, as it will fail without any environment variables set
    • Click the gear icon next to the repository name on the left hand side of the screen
    • In the settings screen, choose Environment Variables you will need to add three variables by clicking Add Variable
      • SFDC_SERVER_KEY is the Base64 output generated in Step 1
      • SFDC_PROD_CLIENTID is the Consumer Key from Step 2
      • SFDC_PROD_USER is the username to use with CircleCI (This should be an Integration user, with the System Administrator profile)
Setting Environment Variables in CircleCI
    • You can now re-run the first build.

Once you have all of this configured and working, you can use the CI build process from here on our, and hopefully never have to worry about a damn change-set again!

Now, every time you push to any branch other than 'master' a scratch org will be created, your code deployed to it and all tests run. If you then merge in to 'master' a production build will be run, validating and deploying your code.

Remember, this is just an introduction. In future posts I will explain in further detail the scripts and config files used in this repository, so you can customise them to suit your exact requirements.

Dreamforce 2017, What an experience! Part 2: Dreamforce itself

The dust is still settling from Dreamforce 2017, having only gotten back to the UK Monday afternoon, but I wanted to share my thoughts while they were still fresh in my mind. This is part two of of this blog, the first is here, about my experience speaking at Dreamforce. This blog is about Dreamforce itself.

Dreamforce 2017!

So Dreamforce is over for another year, and it was just as huge and insane as ever. This is my second Dreamforce, with my first being in 2011. It certainly is a lot bigger than when I last remember! As anyone who has been to Dreamforce knows, it is an overwhelming experience and unlike any other tech conference in existence.

For those that don’t know, Dreamforce is Salesforce’s annual user/partner/customer conference and is held each year in San Francisco, CA. This year is was four days from 6th – 9th November and had speakers like Michelle Obama and performances from Alicia Keys and Lenny Kravitz (see, not your usual tech conference!) plus over 2700 sessions from Salesforce employees, partners and customers (one of which was mine!)

Trailhead was very much at the forefront this year, with an entire ‘Trailhead Area’ in moscone west, decked out with fiberglass rocks, trees, grass and even a waterfall. The road between moscone south and north was closed with ‘Dream Valley’ being created, completely covered in astroturf and home to food stands / cafe’s, lots of seating, a music stage and even a rock climbing wall. There was a trailhead quest to complete, a Dreamforce specific badge and plenty of trailhead swag on offer (including the coveted trailhead hoody).

As I experienced the first time at Dreamforce back in 2011, it is one of these conferences you make all of these plans to see 100 sessions and catch up with everyone you know in the community. In reality you and end up seeing 10% of the sessions you planned to, and see more people than you ever expected to. This may sound like a bad thing, but so much of the value you get from Dreamforce is from the people you meet, the sessions you never thought to attend and the demos you see from Salesforce and from other partners/vendors. A key to enjoying Dreamforce is not worrying too much about what you have planned, and just go with the flow of the week.

Dreamforce is where Salesforce makes its big product announcements for the year and holds Developer, Admin, Trailhead and many more keynote sessions. The theme of the main keynote was ‘We are all trailblazers’ highlighting the economic impact salesforce has had, the fourth industrial revolution and the impact it continues to have on the world, how the 1:1:1 model allows salesforce, and other companies to ‘do well, and do good’. Also highlighted was the stories of ‘trailblazers’ such as Stephanie Herrera, most famous for #SalesforceSaturday.

The focus of the product announcements was on customisation, personalisation, deeper AI integration and IoT, with the announcement of myTrailhead, myLightning, myEinsetin, mySalesforce and myIoT. Of these, I particularly liked myTrailhead, Trailhead is a great learning management system, so rolling out to customers, allowing them to create their own internal trails and track metrics etc is a great move. Hopefully this means the end of super boring and clunky internal training systems.

As usual, here were customer demos, this time from T-Mobile, Adidas and 21st Century Fox to highlight these new product announcements. The T-Mobile and 21st Century Fox presentations had the usual level of salesforce polish and smiling people in trailblazer hoodies, however the Adidas one felt a bit odd to me, especially seeing Marc rocking a full adidas tracksuit and trainers! They centered around how salesforce provides a better understanding of customers.

I attended the developer keynote and was excited about some of the announcements, and a bit disappointed in the lack of others. The major focus of the keynote was platform events, a publish/subscribe architecture allowing you to build event driven applications (similar conceptually to things like MQTT), having used similar tech before I was impressed with this and can’t wait to play with it. Improvements to the Lightning Data Service and new standard lightning components were also announced, bringing it closer to making visualforce completely obsolete. There didn’t really seem to be much in the way of enhancements to the Apex language itself (still no case statement…) which was a bit disappointing.

I managed to attend some sessions as well, including Keir’s session on building offline mobile apps with the Salesforce Mobile SDK, Philipe’s on platform events and Chris Eales’ on helping not-for-profit’s succeed. As always the quality of these sessions was very high and it was great to learn new things from others in the community. When the recorded sessions are release, I will do another post about these in more depth.

One of awesome things about Dreamforce is the opportunities to catch up with people in the community that you’ve not seen, and to meet new people that maybe you only know from twitter / online. I thankfully was able to catch up with many people whom I worked with in Australia and had not seen in a few years. I met some great people from the Good Day Sir podcast community, and I met some new people who were fans of SchemaPuker!

As always, the ‘customer success expo’ was full of Salesforce partners and ISVs showing off their products (and giving out some cool swag). Fidget spinners and socks seemed to be all the rage this year. It is always interesting to see what is available for use with Salesforce, and working at a Salesforce partner its good to have a knowledge of what may be out there to provide solutions to customers needs.

Dreamforce is always a huge week, and it never ever feels like you get to do everything you want to do. While some people think that the whole trailblazer/trailhead/character thing is a bit over the top, the underlying message is solid and its good to part of such a supportive community and to be able to attend events like Dreamforce.

As always, a tonne of talks and keynotes are recorded and will be available online, with some already available here, so even if you didn’t make it to Dreamforce this year, you can get some idea of what it was like.