# Details on the January 22nd, 2019 outage > On January 22nd, 2019, Asana was unavailable for multiple hours. Here's what caused the outage and how we resolved it. Source: https://asana.com/inside-asana/details-january-22-outage ## Details on the January 22nd, 2019 outage Asana had an outage on January 22, 2019. This blog post is a short postmortem of that outage. On Monday January 21, we had multiple brief events where a database that stores metadata shared across our application became overloaded and unresponsive. We are in the process of moving to a microservices architecture, and some traffic was being sent through the new system. We built in retries for this system, and could see the retry code was running repeatedly. We initially believed that the problem was exacerbated by these retries, as we saw high concurrency for code in a critical section, so we disabled the new architecture. Around this time, traffic was coming down off of its peak, and the site appeared stable. We did additional follow-up work to investigate the events, looking at both data and code. We couldn’t reproduce the behavior when testing the code in question with what we believed was reasonable concurrency. On Tuesday January 22, we had a repeat of these issues, with greater total downtime. When the database became unresponsive, we attempted to fail it over to backup. This process left both unavailable for an extended period. We took steps to shed load, which allowed partial recovery[1], but as soon as we allowed full traffic to return, we saw the failure repeat. Our web app was fully unavailable from 11:46-13:30, partially unavailable from 13:30-15:16, fully unavailable from 15:16-16:22, and partially unavailable until 17:16 (times in UTC). We determined that the cause was an expensive synchronized operation within a critical section in the database (a transaction within a stored procedure). We made application changes which moved some of the shared state onto a separate database instance, sharing the load across two instances. The site recovered after taking this action, and we fully undrained. We spent the rest of Tuesday doing follow-up and this time we were able to reproduce the problematic behavior. The critical section became slightly more expensive (especially under high concurrency) when IDs went from 15 digits to 16 digits, because the stored procedure was doing arithmetic using text. To illustrate: This code was implemented many years ago, re-using an existing field (which was text) as a counter. We fixed the stored procedure by changing it to treat the value as a bigint. Without this bug, the outage would have been brief. We also found that application code intended to prevent contention via a distributed lock was not working, and fixed the use of the lock. This problem was caused by an identity check in a duck-typed language (js): the code passed 1 into a library, and tested that the result that returned was identical (===) to 1. The problem is that the library converted the value 1 to the string “1”, and returned the latter; the identity check failed, and the code determined that the lock was not already owned. Without this bug, the concurrency would not have been high enough to cause downtime. We’ve also added monitoring and additional configuration for emergency responses, and are working on additional follow-up. Looking back, we see that there were similar transient issues with this database. We looked into these and had action items for further investigation, but noted that the events resolved quickly. And looking at the data, there often appear to be multiple (potential) causes. There are two lessons to take from this: First, investigating transient issues is important, as they may indicate underlying problems. When there are transient problems in a system with a negative feedback loop in terms of load[2], this likely indicates that the system is near the point of failure, and random fluctuations in load caused it to exceed the threshold. In that situation, it is important to understand what about the load caused the transient problem. If we had not experienced an inflection point in performance due to the floating point conversion, we might have gotten lucky; but hoping for luck is not a strategy. Second, when doing testing, it’s important to test to failure. If we had done this on Monday (rather than only testing what we believed to be reasonable load), we would have identified the problem earlier. It’s important to determine where failure occurs, and what it looks like, in order to understand the behavior of a system. Similarly, while code inspection might not have detected the string conversion given how the stored procedure was written, if we had tested extreme inputs we would have observed the change in behavior. In the long-term, we’re working on systemic changes to ensure that Asana can maintain high availability as we scale. For example, we’re working to eliminate single points of failure like this database, and we’re introducing more clearly separated failure domains so that outages in general will affect only a small portion of users. Together, these will reduce overall downtime as well as the downtime experienced by any team. Enabling teams to achieve effortless collaboration is our top priority and we deeply appreciate your patience and support as we continue to refine and improve Asana for our customers around the world. [1] When shedding load under such circumstances, it’s our policy to prioritize web traffic from paying customers. As a result, the web app became usable for paying customers about 2 hours before we had fully recovered. [2] A system with a negative feedback loop responds to overload in a way that doesn’t make the problem worse, e.g. by reducing work done or delaying it until the overload condition passes. In a positive feedback loop, overload causes additional work and the system fails quickly once it passes the threshold. - [Microframeworks in the Admin Console](/zh-tw/inside-asana/microframeworks-admin-console) 工程 每個 Asana 部署都有一個系統管理主控台。 IT 系統管理員可以在此處設定公司使用 Asana 的方式，例如調整密碼要求、角色和權限、是否可以從 Dropbox 附加檔案，以及在預設情況下誰可以看見新專案。隨著 Asana 的發展，系統管理主控台累積了多年的自訂邏輯和臨時拼湊的解決方案，導致建立和維護管理控制項的成本越來越高。 以其中一項管理設定為例： ... - [以規格為導向的開發：優點以及我們在三個月後學到的經驗](/zh-tw/inside-asana/spec-driven-development) 工程 #### 主任軟體工程師 三個月後，我們更清楚地瞭解了新增的架構在何時有幫助，以及在何時成為阻礙。我們的一位工程師正在準備資料遷移，並決定使用規格導向開發 (SDD) 來規劃工作。 SDD 的用意是幫助他們及早發現疏漏，使方法更易於審查，並為客服人員提供明確的方向。 最終的計劃非常詳細，從紙面上看來相當合理。 它將工作組織如下：問題 → 研究 → 規格 → 審查 → 實施 → 驗證 ... - [我們在 2 週內完成了 Enzyme 遷移。這本應需要五年的時間](/zh-tw/inside-asana/migrating-off-enzyme-2-weeks) 工程 我們最近使用 AI，在大約一次衝刺中完成了多年的工程工作。 以下是其方法，以及為什麼它改變了我們對可能性的看法。五年問題早在 2022 年，我們就著手將 Asana 的前端測試套件從我們老舊的測試庫 Enzyme 移轉至 React Testing Library (RTL)。 Enzyme 已經失去社群支援，無法與較新版本的 React 正常搭配，並且鼓 ... - [AI 隊友如何建立記憶：將工作轉化為可重複使用的知識](/zh-tw/inside-asana/ai-teammates-turn-work-into-reusable-information) 人工智慧 (AI) 工程 大多數 AI 產品將記憶視為個人功能，記住關於一位使用者或一段對話的事實。 但跨團隊協作的 AI 需要一種截然不同的記憶。 當 AI 系統能夠在先前學習的基礎上再接再厲時，就會變得更加實用。 但在企業軟體中，記憶不僅僅是儲存更多內容的問題。 更困難的問題是，讓記憶在共用工作中發揮作用，同時仍保持其可檢查、可治理且具有權限意識。這就是我們打算透過 AI 隊友 ... - [Details on the January 22 outage](/zh-tw/inside-asana/details-january-22-outage) 工程 Asana had an outage on January 22, 2019. This blog post is a short postmortem of that outage. On Monday January 21, we had multiple brief events where a database that stores meta ... - [工程](/inside-asana/engineering-spotlight)