There have been a number of concerns and questions raised about the current upgrade of the Archer SaaS and Hosted environments to 6.12 (specifically 6.12 P1). This upgrade has not gone as well as intended, and accordingly some explanation and visibility into the challenges of the past couple of weeks is important to share. Archer values the sense of community we share with our customers and our communications on this topic could clearly have been better. We are committed to ensuring that they are going forward.
Background
On October 7th, Archer upgraded the Non Prod environments (SaaS US, SaaS EMEA, SaaS APJ, US Hosted, EMEA Hosted) from Archer 6.11 P2 HF4 to Archer 6.12 P1 as planned. The upgrade was smooth, without apparent issue. Over the subsequent weekend, irregularities in the performance of the environment were observed. Investigations by the Technical Support and SaaS Operations teams identified an issue in job engine logging which was having knock-on effects to other parts of the system. The Continuous Engineering team was engaged, and a replacement hotfix build was deployed to the environments on Monday, October 17th. The intent was to promote this version to Production on the originally scheduled release date. Unfortunately, two further issues arose:
- A second logging issue which had been masked by the first issue.
- A deliberate change in the Reports Listing page functionality was not well-received on initial feedback, and it was agreed to immediately reverse this change.
The various teams involved attempted to remediate these two items in time for the Production upgrade to go ahead as planned. Unfortunately, this was not possible to achieve safely since the intent is to deploy the Hotfix code in NonProd in order to ensure that there is no other masked issue that would impair the Production environment. The decision was made reluctantly to defer the Production upgrade a week, prioritizing the stability of the Production Environment.
The entire Archer team acknowledges the frustration this incident has caused to users, and the further impact these delays have caused for planned migrations of packages. The team is working hard to correct the problems and get the upgrade back on track in order for everyone to benefit from the range of new capabilities, bug fixes and security improvements present in every new release of Archer.
Our Commitment To Do Better
Quality - The job engine logging issues that impacted the upgrade are unequivocally a miss in testing. Once the upgrades are completed, a full retrospective will be conducted to determine how these items were overlooked during testing and, more importantly, how to design future test cycles to prevent this kind of miss. Part of the reason for performing the NonProd upgrade ahead of the Production upgrade is to provide a window to address any missed items like these, but these issues should have been caught internally. Archer Engineering has recently reorganized its internal team structure to place a greater emphasis on test execution and holistic quality, and this retrospective will drive new practices within this updated configuration.
Communications – We know that we can improve the way we communicate about incidents such as this one. As noted above, our goal was to correct the problems and keep to the schedule. However, it was not always clear to those communicating with customers that this was the case, or indeed when this goal had to change and the plans changed. We will take action to improve and streamline the way we communicate in future scenarios.
Package Versioning – The path to an improved experience of SaaS, less frequent incidents and more rapid resolution of issues is to perform more frequent upgrades of the environment that contain smaller changes to the code. But there is an impediment to this: the fact that doing so creates a situation where NonProd packages cannot be deployed to Prod. This functionality is deliberate and intended to protect the integrity of the data within the environment. However, the amplification of the impact of what date what environment moves on is inconvenient. As a result, the Archer team is exploring options to allow the backporting of packages or a compatibility-mode concept to be introduced. These discussions are in their early stages, but expect more to come on this front so that in the future, project plans will not need to be based around plans for upgrades.
Please continue to track the upgrade notification page for updates on timing of the Production upgrade and any further patches which may be introduced.
