Merge pull request #296 from nullstyle/out-of-date

Allow horizon operators to define a "Staleness Threshold" that, when passed, disables historical data requests

CHANGELOG.md

@@ -14,6 +14,7 @@ This release contains the initial implementation of the "Abridged History System
 
 - *Elder* ledgers have been introduced:  An elder ledger is the oldest ledger known to a db.  For example, the `core_elder_ledger` attribute on the root endpoint refers to the oldest known ledger stored in the connected stellar-core database.
 - Added the `history-retention-count` command line flag, used to specify the amount of historical data to keep in the history db.  This is expressed as a number of ledgers, for example a value of `362880` would retain roughly 6 weeks of data given an average of 10 seconds per ledger.
+- Added the `history-stale-threshold` command line flag to enable stale history protection.  See the admin guide for more info.
 - Horizon now reports the last ledger ingested to stellar-core using the `setcursor` command.
 - Requests for data that precede the recorded window of history stored by horizon will receive a `410 Gone` http response to allow software to differentiate from other "not found" situations.
 - The new `db reap` command will manually trigger the deletion of unretained historical data
@@ -22,6 +23,7 @@ This release contains the initial implementation of the "Abridged History System
 ### Changed
 
 - BREAKING: When making a streaming request, a normal error response will be returned if an error occurs prior to sending the first event.  Additionally, the initial http response and streaming preamble will not be sent until the first event is available.
+- BREAKING: `horizon_latest_ledger` has renamed to `history_latest_ledger`
 - Horizon no longer needs to begin the ingestion of historical data from ledger sequence 1.  
 - Rows in the `history_accounts` table are no longer identified using the "Total Order ID" that other historical records  use, but are rather using a simple auto-incremented id.
 

docs/reference/admin.md

@@ -115,6 +115,12 @@ We recommend you configure the HISTORY_RETENTION_COUNT in horizon to a value les
 4.  Clear ledger metadata from before the gap by running `stellar-core -c "maintenance?queue=true"`.
 5.  Restart horizon.    
 
+## Managing Stale Historical Data
+
+Horizon ingests ledger data from a connected instance of stellar-core.  In the event that stellar-core stops running (or if horizon stops ingesting data for any other reason), the view provided by horizon will start to lag behind reality.  For simpler applications, this may be fine, but in many cases this lag is unacceptable and the application should not continue operating until the lag is resolved.
+
+To help applications that cannot tolerate lag, horizon provides a configurable "staleness" threshold.  Given that enough lag has accumulated to surpass this threshold (expressed in number of ledgers), horizon will only respond with an error: [`stale_history`](./errors/stale-history.md).  To configure this option, use either the `--history-stale-threshold` command line flag or the `HISTORY_STALE_THRESHOLD` environment variable.  NOTE:  non-historical requests (such as submitting transactions or finding payment paths) will not error out when the staleness threshold is surpassed.
+
 ## Monitoring
 
 To ensure that your instance of horizon is performing correctly we encourage you to monitor it, and provide both logs and metrics to do so.  

docs/reference/errors/stale-history.md

@@ -0,0 +1,30 @@
+---
+title: Stale History
+---
+
+A horizon server may be configured to reject historical requests when the history is known to be further out of date than the configured threshold.  In such cases, this error is returned.  To resolve this error (provided you are the horizon instance's operator) please ensure that the ingestion system is running correctly and importing new ledgers.
+
+## Attributes
+
+As with all errors Horizon returns, `stale_history` follows the [Problem Details for HTTP APIs](https://tools.ietf.org/html/draft-ietf-appsawg-http-problem-00) draft specification guide and thus has the following attributes:
+
+| Attribute | Type   | Description                                                                                                                     |
+| --------- | ----   | ------------------------------------------------------------------------------------------------------------------------------- |
+| Type      | URL    | The identifier for the error.  This is a URL that can be visited in the browser.                                                |
+| Title     | String | A short title describing the error.                                                                                             |
+| Status    | Number | An HTTP status code that maps to the error.                                                                                     |
+| Detail    | String | A more detailed description of the error.                                                                                       |
+| Instance  | String | A token that uniquely identifies this request. Allows server administrators to correlate a client report with server log files  |
+
+## Example
+
+```shell
+$ curl -X GET "https://horizon-testnet.stellar.org/transactions?cursor=1&order=desc"
+{
+  "type": "stale_history",
+  "title": "Historical DB Is Too Stale",
+  "status": 503,
+  "detail": "This horizon instance is configured to reject client requests when it can determine that the history database is lagging too far behind the connected instance of stellar-core.  If you operate this server, please ensure that the ingestion system is properly running.",
+  "instance": "horizon-testnet-001.prd.stellar001.internal.stellar-ops.com/ngUFNhn76T-078058"
+}
+```

src/github.com/stellar/horizon/action.go

@@ -10,6 +10,7 @@ import (
 	"github.com/stellar/horizon/db2/core"
 	"github.com/stellar/horizon/db2/history"
 	"github.com/stellar/horizon/httpx"
+	"github.com/stellar/horizon/ledger"
 	"github.com/stellar/horizon/log"
 	"github.com/stellar/horizon/render/problem"
 	"github.com/stellar/horizon/toid"
@@ -51,7 +52,7 @@ func (action *Action) GetPagingParams() (cursor string, order string, limit uint
 
 	if cursor == "now" {
 		tid := toid.ID{
-			LedgerSequence:   action.App.latestLedgerState.HorizonLatest,
+			LedgerSequence:   ledger.CurrentState().HistoryLatest,
 			TransactionOrder: toid.TransactionMask,
 			OperationOrder:   toid.OperationMask,
 		}
@@ -153,13 +154,30 @@ func (action *Action) ValidateCursorWithinHistory() {
 		return
 	}
 
-	elder := toid.New(action.App.latestLedgerState.HorizonElder, 0, 0)
+	elder := toid.New(ledger.CurrentState().HistoryElder, 0, 0)
 
 	if cursor <= elder.ToInt64() {
 		action.Err = &problem.BeforeHistory
 	}
 }
 
+// EnsureHistoryFreshness halts processing and raises
+func (action *Action) EnsureHistoryFreshness() {
+	if action.Err != nil {
+		return
+	}
+
+	if action.App.IsHistoryStale() {
+		ls := ledger.CurrentState()
+		err := problem.StaleHistory
+		err.Extras = map[string]interface{}{
+			"history_latest_ledger": ls.HistoryLatest,
+			"core_latest_ledger":    ls.CoreLatest,
+		}
+		action.Err = &err
+	}
+}
+
 // BaseURL returns the base url for this requestion, defined as a url containing
 // the Host and Scheme portions of the request uri.
 func (action *Action) BaseURL() *url.URL {

src/github.com/stellar/horizon/actions_effects.go

@@ -33,6 +33,7 @@ type EffectIndexAction struct {
 // JSON is a method for actions.JSON
 func (action *EffectIndexAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 		action.loadRecords,
@@ -47,6 +48,7 @@ func (action *EffectIndexAction) JSON() {
 // SSE is a method for actions.SSE
 func (action *EffectIndexAction) SSE(stream sse.Stream) {
 	action.Setup(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 	)

src/github.com/stellar/horizon/actions_ledger.go

@@ -3,6 +3,7 @@ package horizon
 import (
 	"github.com/stellar/horizon/db2"
 	"github.com/stellar/horizon/db2/history"
+	"github.com/stellar/horizon/ledger"
 	"github.com/stellar/horizon/render/hal"
 	"github.com/stellar/horizon/render/problem"
 	"github.com/stellar/horizon/render/sse"
@@ -26,6 +27,7 @@ type LedgerIndexAction struct {
 // JSON is a method for actions.JSON
 func (action *LedgerIndexAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 		action.loadRecords,
@@ -37,6 +39,7 @@ func (action *LedgerIndexAction) JSON() {
 // SSE is a method for actions.SSE
 func (action *LedgerIndexAction) SSE(stream sse.Stream) {
 	action.Setup(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 	)
@@ -91,6 +94,7 @@ type LedgerShowAction struct {
 // JSON is a method for actions.JSON
 func (action *LedgerShowAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.verifyWithinHistory,
 		action.loadRecord,
@@ -112,7 +116,7 @@ func (action *LedgerShowAction) loadRecord() {
 }
 
 func (action *LedgerShowAction) verifyWithinHistory() {
-	if action.Sequence < action.App.latestLedgerState.HorizonElder {
+	if action.Sequence < ledger.CurrentState().HistoryElder {
 		action.Err = &problem.BeforeHistory
 	}
 }

src/github.com/stellar/horizon/actions_metrics.go

@@ -14,7 +14,6 @@ type MetricsAction struct {
 
 // JSON is a method for actions.JSON
 func (action *MetricsAction) JSON() {
-	action.App.UpdateMetrics(action.Ctx)
 	action.LoadSnapshot()
 	action.Snapshot["_links"] = map[string]interface{}{
 		"self": hal.NewLink("/metrics"),

src/github.com/stellar/horizon/actions_operation.go

@@ -3,6 +3,7 @@ package horizon
 import (
 	"github.com/stellar/horizon/db2"
 	"github.com/stellar/horizon/db2/history"
+	"github.com/stellar/horizon/ledger"
 	"github.com/stellar/horizon/render/hal"
 	"github.com/stellar/horizon/render/problem"
 	"github.com/stellar/horizon/render/sse"
@@ -31,6 +32,7 @@ type OperationIndexAction struct {
 // JSON is a method for actions.JSON
 func (action *OperationIndexAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 		action.loadRecords,
@@ -43,6 +45,7 @@ func (action *OperationIndexAction) JSON() {
 // SSE is a method for actions.SSE
 func (action *OperationIndexAction) SSE(stream sse.Stream) {
 	action.Setup(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 	)
@@ -134,6 +137,7 @@ func (action *OperationShowAction) loadResource() {
 // JSON is a method for actions.JSON
 func (action *OperationShowAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.verifyWithinHistory,
 		action.loadRecord,
@@ -146,8 +150,7 @@ func (action *OperationShowAction) JSON() {
 
 func (action *OperationShowAction) verifyWithinHistory() {
 	parsed := toid.Parse(action.ID)
-
-	if parsed.LedgerSequence < action.App.latestLedgerState.HorizonElder {
+	if parsed.LedgerSequence < ledger.CurrentState().HistoryElder {
 		action.Err = &problem.BeforeHistory
 	}
 }

src/github.com/stellar/horizon/actions_operation_test.go

@@ -86,7 +86,7 @@ func TestOperationActions_Show(t *testing.T) {
 	}
 
 	// doesn't exist
-	w = ht.Get("/operations/10")
+	w = ht.Get("/operations/9589938689")
 	ht.Assert.Equal(404, w.Code)
 
 	// before history

src/github.com/stellar/horizon/actions_payment.go

@@ -23,6 +23,7 @@ type PaymentsIndexAction struct {
 // JSON is a method for actions.JSON
 func (action *PaymentsIndexAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 		action.loadRecords,
@@ -36,6 +37,7 @@ func (action *PaymentsIndexAction) JSON() {
 // SSE is a method for actions.SSE
 func (action *PaymentsIndexAction) SSE(stream sse.Stream) {
 	action.Setup(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 	)

src/github.com/stellar/horizon/actions_root.go

@@ -1,6 +1,7 @@
 package horizon
 
 import (
+	"github.com/stellar/horizon/ledger"
 	"github.com/stellar/horizon/render/hal"
 	"github.com/stellar/horizon/resource"
 )
@@ -18,10 +19,7 @@ func (action *RootAction) JSON() {
 	var res resource.Root
 	res.Populate(
 		action.Ctx,
-		action.App.latestLedgerState.HorizonLatest,
-		action.App.latestLedgerState.HorizonElder,
-		action.App.latestLedgerState.CoreLatest,
-		action.App.latestLedgerState.CoreElder,
+		ledger.CurrentState(),
 		action.App.horizonVersion,
 		action.App.coreVersion,
 		action.App.networkPassphrase,

src/github.com/stellar/horizon/actions_trade.go

@@ -24,6 +24,7 @@ type TradeIndexAction struct {
 // JSON is a method for actions.JSON
 func (action *TradeIndexAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.loadRecords,
 		action.loadPage,

src/github.com/stellar/horizon/actions_transaction.go

@@ -31,6 +31,7 @@ type TransactionIndexAction struct {
 // JSON is a method for actions.JSON
 func (action *TransactionIndexAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 		action.loadRecords,
@@ -44,6 +45,7 @@ func (action *TransactionIndexAction) JSON() {
 // SSE is a method for actions.SSE
 func (action *TransactionIndexAction) SSE(stream sse.Stream) {
 	action.Setup(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.ValidateCursorWithinHistory,
 	)
@@ -121,6 +123,7 @@ func (action *TransactionShowAction) loadResource() {
 // JSON is a method for actions.JSON
 func (action *TransactionShowAction) JSON() {
 	action.Do(
+		action.EnsureHistoryFreshness,
 		action.loadParams,
 		action.loadRecord,
 		action.loadResource,